People use these terms "input space", "feature space", "sample space", "hypothesis space", "parameter space" in machine learning.
Could anyone explain these terms with a concrete example, such as sklearn MNIST dataset?, which has 1797 Samples, 10 Classes, 8*8 Dimensionality and 17 Features.
Please do NOT talk about in general.
For example, in this particular case, is the feature space a set of 17 elements {0, 1, ..., 16}?
We'll discuss each of the terms.
Input Space
It contains all the possible inputs for a model. Suppose the model takes in a vector, $input = [ x_1 , x_2]$ , where $x_1 , x_2 \in [ 1 , 10 ]$, then we can have $10^{2}$ inputs. This constitutes the "input space". See here.
For the MNIST dataset, the dimensions of the image are 8 * 8 meaning 64 points. Now each point can have a value lying in the interval $[ 0 , 16 ]$, so it can have 16 values. So the input space has a size of $16^{64}$.
Feature Space
The multidimensional space in which are features is defined. Considering the above examples, we can have three samples,
$a_1 = [ 2 , 3 ] \\ a_2 = [ 7 , 4.5 ] \\ a_3 = [ 3.67 , 2 ]$
These vectors could be included in an n-dimensional space ( here n=2 for our case ). Hence, in our case, the 2D space where we can plot our features constitutes our "feature space".
For the MNIST dataset, the input vector has 64 elements which correspond to a 64-dimensional space ( feature space ).
See this answer.
Difference between input space and feature space.
Input spaces include all possible inputs for our model. Feature spaces, on the other hand, include the feature vectors from a given set of data. They may not contain all the possible inputs for a model.
Hypothesis Space
Space which contains all the functions produced by a model. The functions map the inputs to their respective outputs. A model can output various functions ( or rather relationships between the inputs and outputs ) based on its learning. If you have a larger hypothesis space, the model cannot find the "best" one. See this answer.
For the MNIST dataset, as we calculated earlier, the size of the input space is $16^{64}$. Each one of them can have any one of the 10 labels ( classes ). Hence, the size of the hypothesis space is $10^{16^{64}}.$
Parameter Space
For each model in ML, we have some parameters for the model. The space in which we can define these parameters ( or hyperparameters ) is our "parameter space". From Wikipedia's example, we can understand it,
The parameter space would differ for every model.
In a sine wave model ${\displaystyle y(t)=A\cdot \sin(\omega t+\phi > ),}y(t)=A\cdot \sin(\omega t+\phi )$, the parameters are amplitude A > 0, angular frequency ω > 0, and phase φ ∈ S1. Thus the parameter space is ${\displaystyle R^{+}\times R^{+}\times S^{1}}$.
