Every data science program should definitely teach these two books to students:

@jayvanbavel If the knowledge is there as you claim, one expects high predictive performance in predicting success. In the paper you referenced, that's not being specified. Show us evidence of high accuracy in predicting success to back up your claim.

@SamuelMullr Forget about the academic benchmarks! In industry even medium-sized companies have more data than that, think about customers' orders time series etc. I was going to try your model on our customers data but because of this limitation it failed.

@SamuelMullr I see, would it be possible to relax this limitation possibly using batching? Or maybe subsampling to train the model would be good enough assuming this is a pre-trained model that we're just fine tuning?

@macrocephalopod @ryxcommar @andrew_n_carr Most often you use linear regression for inference rather than prediction. And if you have very large number of features that do not fit in memory than most probably you’re doing something wrong in your problem setup.

@cameron_pfiffer @ryxcommar @andrew_n_carr Do you need to use an extraordinarily large dataset for fitting a linear regression? Wouldn’t OLS on a random sample of this huge dataset be more reliable than SGD on the entire dataset?

@anax_perseus @nixtlainc I tried it on demand time series for many SKUs, several of which have intermittent data, for one-step-ahead forecasting (which was all I needed), and LightGBM significantly outperformed TimeGBT.

I’ve heard different opinions about AGI. Some people claim we already have it, while others argue that AGI should possess consciousness and free will. If the latter is true, I’m highly skeptical that we can achieve AGI with just matrix multiplication in a loop!

@_apoorvnandan And how does knowing the answer to this question makes someone more qualified than others in being a good ML eng?!!

@GeeLedger @matloff @lisawehden That doesn’t make sense, considering how much easier it is to hire locals than foreigners due to factors like visa sponsorship and relocation. Besides, it’s not a zero-sum game—more talent contributes to economic growth, which in turn creates demand for even more skilled people.

@MatthewBJane Yes, I meant among researchers in general. People often deny the predictive power of IQ scores because they believe IQ is unchangeable and don't want to accept that they might not be able to become smarter. If you tell them that IQ can be improved, the resistance likely goes away

@tcarpenter216 @BenOgorek @ryxcommar If by 'GenAI can be fabulous with tabular data' you mean using Copilot or ChatGPT for Q&A, of course, we've been using them for a long time. Anyone can do that; it's just prompting. I meant something more advanced that requires knowledge of transformers, RAG structures, etc.

@BenOgorek @tcarpenter216 @ryxcommar In my current role, most of our projects are about working with tabular data including explanatory data analyses, causal inference, predictive modeling etc. Only recently we started working on some GenAI projects as well.

@henrybemis13 @PhDemetri You can only argue with my statement by claiming at least one of them implies absolutely no effects. Please read carefully.