In natural language processing, why each feature requires an extra dimension?

I am reading Machine Learning by Example. I am trying to understand natural language processing. The book used Scikit-learn's fetch_20newsgroups data as an example.

The book mentioned that the text data in the 20 newsgroups dataset that we downloaded from fetch_20newsgroups data is highly dimensional. I do not understand this statement. It is my understanding that dimension is used to describe axies that an array has. For example,

import numpy as np
np.random.seed(0)
x1 = np.random.randint(10, size=6)
print("x1",x1) # 1 dimensions
np.random.seed(0)
x2 = np.random.randint(10, size=(3,4))
print("x2",x2) # 2 dimensions
np.random.seed(0)
x3 = np.random.randint(10, size=(3,4,5))
print("x3",x3) #3 dimensions


How does no. of axies relates to feature in NLP? Why one feature equals to one dimension? Please explain. Thanks.

from sklearn.datasets import fetch_20newsgroups
groups = fetch_20newsgroups()

• When working with tabular data (two "dimensions" using your understanding of it), we call the number of columns the dimension of the data. This is because you can see each data point as a vector, and we call "how many elements there is in a vector" its dimension in linear algebra. – Mephy Aug 31 '18 at 12:54
• May I assume no. of columns equal to the no. of dimension in an array? In addition to the fact that no. of columns should also equal to no. of features.If so, it is easy. But I am worry that this approach of understanding maybe incomplete. May you clarify? – carch Sep 1 '18 at 8:05

You can think of each feature running along its own axis on a graph. Just because we store all feature e.g. in a single DataFrame – one feature per column – it doesn't mean the data's structure is just 2d (rows and columns). This is not the case only in NLP, but in most contexts involving statistics and modelling.

We can see this with your example data. There are text blocks, which you should match to a category (as far as I can tell).

The initial dataset contains other meta-data, such as a desctiption of the dataset, the names of the target categories and also the location of each sample's file. We don't really care about these for the pure modelling part. So there are only text blocks, called data, and the target categories, called target. Your input is then 1d - the text blocks.

I will show how to put that into a dataframe, being very verbose about dimensions and features:

from sklearn.datasets import fetch_20newsgroups
groups = fetch_20newsgroups()

import pandas as pd                    # needed to use a dataframe

# Get the desired parts from "groups"
desired = ['data', 'target']           # we don't care about the 'filenames' ona so on

# make a new dictionary with only desired key-value pairs
only_data = {k: v for k, v in groups.items() if k in desired}


Now we put this into a dataframe

df = pd.DataFrame.from_dict(only_data)

# Check the shape of the dataframe
df.shape
(11314, 2)


So there are 11314 samples of 1 feature, to 1 target variable. This is therefore 1-dimensional input data (we don't count the target variable).

When we have e.g. 50 features, explaining some target variable, it may be referred to as a 50-dimensional input space. People then may use dimensionality reduction techniques, such as Principal Components Analysis, which will attempt to squeeze the 50 dimensions into a lesser number (you can choose how many to use).

In your data, you will likely pre-process the text samples to create more features. These will just be new columns in the dataframe, whose shape could become e.g. (11314, 40) if you add another 38 features, by doing things like counting words or constructing some word-embeddings.

• Thanks. I get that df.shape is representing no. of columns. which is also no. of features. Under the condition that 'target' is not included in a table, no. of columns on a table is equal to no. of dimensions in a dataset. This statement is in conflict with no. of dimensions in a multi-dimensional array. Considering code in question, x2 is a two dimensional array with size 3 by 4, which makes it a table with 4 columns, 3 rows. Is my understanding wrong by using 4 columns to describe an array with size 3 by 4? Please clarify. I feel like I am getting to the core of my problem. Thanks a lot. – carch Sep 1 '18 at 9:31
• @carch - Notice the difference between dimensions of an array and the semantic meaning of a feature. Imagine each sample is a colour image, with shape = (100, 100, 3. We might consider the image one feature of a larger problem (e.g. maybe there is a temperature measurement too). We have two features: image and temperature. You could break down the image into many more features. Or you could even flatten the pixel array into a single column of length 100 x 100 3 = 30000. Your x2 might also be interpreted like the image (I can't say for sure as the example shows it as random integers). – n1k31t4 Sep 1 '18 at 21:21
• Sorry, I am not getting it. Please judge are following statements correct with shape=(100,100,3) (1) is a 3 dimensional array. (2) shape(height 100, width 100, depth 3) (3) shape (100set of images, 100 rows of values for each image, 3 columns for each rows of values for each image). (4)no. of dimension of an array is not necessary equal to no. of feature. 3 columns in shape does not mean 3 features.(5)no. of feature is dependent on how feature is designed. Many thanks! – carch Sep 2 '18 at 2:35
• (1) True. (2) True. (3) false: one single image has dimensions (100, 100, 3) - 100 rows, 100 columns and 3 deep. (4) True. (5) True. You're welcome :-) – n1k31t4 Sep 2 '18 at 2:39
• Many thanks. I now have a better understanding on this issue. Hopefully this question can also clarify future learner who is confused with dimensions and features. Salute! – carch Sep 2 '18 at 3:38

Not sure how much about NLP you already digested, so shortly from the begining:

Text processing usually starts with tokenization into words (and other segments like numbers and punctuation marks) and each word has an index in dictionary created from whole corpus (set of texts). Each item (word) in the dictionary becomes a separate feature.

When you encode a given text, each word in one-hot encoded, so the text becomes a vector of the length that equals size of the dictionary (classical bag of words approach). That usually goes into thousands (e.g. Yelp review dataset has over milion unique words) so clearly this is a highly dimensional problem.

There is a number of techniques then to reduce this dimentionality, like embeddings of LDA.

Appendix for the question in comment:

When you decide for the number of feature (which is number of words that matter for your taks), usually you exlude stop words (like 'a', 'an', 'the' and others that are very popular in texts but does not carry meaning for e.g. classification). I usually manually select them from frequency dictionary (list of words with counts in corpus, sorted by the count). You can compare your list with that used in NLTK.

And from the other side of the dictionary, you would usually skip words that occur very rare, like less than 3 times (number depends on the corpus).

So you end with your one dictionary which has it's size and this is the number of your features.

• In a bag of words model, text are reduced to words. Each word become a separate feature. May I ask, if I now include no. of occurrence for each word as a feature, would no. of features increases in proportion to no. of words? i.e. By including no. of occurrence as feature. I did not add just one features. I added multiple features with the same size as no. of words that I have. Thanks. – carch Sep 1 '18 at 9:40
• It's typical to include some version of tf-idf as feature, which discounts popularity of a word in whole corpus - so for words that are very popular they count / frequency is divided by the number of documents they appear in (or logarithm, see wiki post en.wikipedia.org/wiki/Tf%E2%80%93idf), – MkL Sep 1 '18 at 16:41