What does it mean by 'trainning a model' on structured data?

Question

Apologies if this is a noob question, I have been trying to find a hello-world kind of example regarding how to train a model on structured data without much success. Also, have been reading about LLMs getting trained on large datasets with 7-70 billion parameters, though I can't find any tutorial on what those parameters could be and what does it mean by training a model with those parameters.

Let's say you have a structured data in RDBMS - OrderItems with attributes like order id, order date, item id, order by, etc.

Is there a simple example on how to train such a structured data with n or more parameters (not sure what parameter means) and how that makes a difference on the final outcome?

Answer 1

First things first, when you see a model being trained on 7-70 or even 140 billion parameters, this refers to the weights and biases of the model. Not sure if you understand this yet, if not I recommend you start with the basics, something like Andrew Ng's Deep Learning specialization

Essentially, the weights and the biases is what is tuned/trained and helps the model learn. Using a simple example, if you have just 1 weight and bias, it would have the equation:

$$ y = w \cdot x + b $$

your goal is to adjust w and b (or in a regular equation of a line, m and b) and get the closest prediction to make this equation true. So you can imagine if the ideal values for w and b are, say, 1 and 2. But they start at a randomly close to 0 number like 0.1 and 0.2. Over time, these numbers of 0.1 and 0.2 will gradually start to increase and head towards those ideal numbers of 1 and 2. This is training.

So when a model has 70 billion parameters, it has 70 billion weights and biases to train in order to reach the ideal value which maps to the very complex function it's trying to solve, because at the end of the day, a neural network is just one really big complex function. Get an input(s), do something to that input(s), give an output(s).

As for your second half of the question, of course resources like this exist. If you go to Kaggle (Popular ML platform), you can find very simple beginner competitions that deals with structured data and is closer to a regression problem, but will help you grasp the basics of neural networks. There are also posted solutions by a bunch of people on all of these competitions. (Some submissions are even tailored for beginner walkthroughs)

The number of parameters in this case, would be the number of separate structured data categories your model would take. In the typical house prediction problem, you could be given the square feet of the house, the year the house was made, how much rooms, how much bathrooms, indoor pool or not, etc. These would all be parameters for your model (and would determine the dimensions of the input layer).

The reason this makes a difference on the final outcome, is because you have more data to work with. If you simply gave a model just the square feet of the house, and asked it to predict the price, you could probably get something reasonable, but it would be nowhere near as accurate compared to a model that had 20 or more unique data points that showed correlation with the price. In this case, it's much easier to learn the pattern and predict the outcome.

Hope this helps!