The Great Regression – Where it All Starts

Yes, statistical! In case you have yet made peace with this fact, machine learning and AI are all about statistical analysis. It doesn’t mean you must be familiar with Inverse Wishart Distributions (although it would help), but you do need to understand what is a statistical model.

Statistical Analysis is the branch of science that takes large amounts of data and then examines it in order to discover patterns and trends that may occur. In addition, Statistical Analysis usually handles the presenting of discovered conclusions in an understandable and often graphical way.

After we've transformed our problem into a mathematical representation, a Statistical Model will then apply a set of assumptions to it. This allows us to (theoretically) calculate the probability of certain events.

Our set of inputs is one sample given from the larger group (or population) of the total possible inputs to the system. When training an AI algorithm (or "fitting a model"), we assume that the sample of data we use represents the entire population of possible inputs.In simpler terms, we use our algorithm on data that is similar to the data it was trained on.

For example, if we built an algorithm that predicts future housing prices with data collected from Chicago’s housing prices in the past 30 years, won't use it to predict future housing prices in Tokyo (obviously…).

We use our given data sample to infer on the probability distribution of the whole population, then we find the most probable value, outputting it as the prediction.

Before we get into actual linear regression, a few words about dependent and independent variables.

Linear Regression is a model of linear relationship between a dependent variable (the height of any blue dot in Figure 1) and independent variable (the distance between any blue dot and the y axis in Figure 1). Linear Regression in its simplest form, will describe the linear mapping from one independent variable to a scalar (quantity) response, as seen in Figure 1.

Fitting the model is done by minimizing a certain loss function, the most common choice being the Least Squares Loss, to find the set of parameters resulting in the best fit to the training data. Eventually, the model will tie the behavior of the response, or result, with the behavior of the input variables:

Or in vector form:

ŷ being the model’s output (the dependent variable), θᵢ are the model’s parameters and x̂ are the input’s components (the independent variable).

One common use for linear regression is finding the linear relationship between the variables for the sake of proving such relationship exists, or quantifying its strength. A second common goal is to predict the response for other sets of inputs.

Going back to our Chicago housing prices predictor, the input vector’s components might be the size of the house, the number of bedrooms and the number of floors. After fitting the model on data gathered across Chicago, the model will be able to predict the cost or value of a house based on those inputs.*

(*Housing prices usually vary over time, meaning a linear model will not be a very good fit. The above model can be used to predict housing prices in a certain year, or a more complex model that accounts for sequential data can be used.)

Now that we understand what a linear regression is, we can complicate things a little. While linear regression can be used to predict values of a certain response, a logistic regression is used to predict probabilities of a binary event, e.g. whether it will be ‘0’ or ‘1’. One example might be using a logistic regression model to build a binary cat classifier, judging whether a cat exists in a given image, based on the dataset that was used to fit (or train) the model.

Logistic regression can be thought of as a single-layer neural network, and its output is given by:

Where σ is the logistic function:

From the above formula, it can be seen that the model’s output will always be ŷ∈(0,1), which makes sense as it is a probability. As a matter of fact, the output of the logistic model will be applying the logistic function on the output of a linear model.

Thanks to the non-linear response of the logistic function, several of those “logistic blocks” can be put together sequentially to construct a neural network, allowing it to learn complex relationships. This is in contrast to the “linear blocks”, which only allow linear relationships, even when several of them are used in a sequence.