In natural language processing, why each feature requires an extra dimension?

Question

I am reading Machine Learning by Example. I am trying to understand natural language processing. The book used Scikit-learn's fetch_20newsgroups data as an example.

The book mentioned that the text data in the 20 newsgroups dataset that we downloaded from fetch_20newsgroups data is highly dimensional. I do not understand this statement. It is my understanding that dimension is used to describe axies that an array has. For example,

import numpy as np
np.random.seed(0)
x1 = np.random.randint(10, size=6)
print("x1",x1) # 1 dimensions
np.random.seed(0)
x2 = np.random.randint(10, size=(3,4))
print("x2",x2) # 2 dimensions
np.random.seed(0)
x3 = np.random.randint(10, size=(3,4,5))
print("x3",x3) #3 dimensions

How does no. of axies relates to feature in NLP? Why one feature equals to one dimension? Please explain. Thanks.

Below is the code from the book that used to download the data for your reference.

from sklearn.datasets import fetch_20newsgroups
groups = fetch_20newsgroups()

Answer 1

You can think of each feature running along its own axis on a graph. Just because we store all feature e.g. in a single DataFrame – one feature per column – it doesn't mean the data's structure is just 2d (rows and columns). This is not the case only in NLP, but in most contexts involving statistics and modelling.

We can see this with your example data. There are text blocks, which you should match to a category (as far as I can tell).

The initial dataset contains other meta-data, such as a desctiption of the dataset, the names of the target categories and also the location of each sample's file. We don't really care about these for the pure modelling part. So there are only text blocks, called data, and the target categories, called target. Your input is then 1d - the text blocks.

I will show how to put that into a dataframe, being very verbose about dimensions and features:

from sklearn.datasets import fetch_20newsgroups
groups = fetch_20newsgroups()

import pandas as pd                    # needed to use a dataframe

# Get the desired parts from "groups"
desired = ['data', 'target']           # we don't care about the 'filenames' ona so on

# make a new dictionary with only desired key-value pairs
only_data = {k: v for k, v in groups.items() if k in desired}

Now we put this into a dataframe

df = pd.DataFrame.from_dict(only_data)

# Check the shape of the dataframe
df.shape
(11314, 2)

So there are 11314 samples of 1 feature, to 1 target variable. This is therefore 1-dimensional input data (we don't count the target variable).

---

When we have e.g. 50 features, explaining some target variable, it may be referred to as a 50-dimensional input space. People then may use dimensionality reduction techniques, such as Principal Components Analysis, which will attempt to squeeze the 50 dimensions into a lesser number (you can choose how many to use).

In your data, you will likely pre-process the text samples to create more features. These will just be new columns in the dataframe, whose shape could become e.g. (11314, 40) if you add another 38 features, by doing things like counting words or constructing some word-embeddings.

Answer 2

Not sure how much about NLP you already digested, so shortly from the begining:

Text processing usually starts with tokenization into words (and other segments like numbers and punctuation marks) and each word has an index in dictionary created from whole corpus (set of texts). Each item (word) in the dictionary becomes a separate feature.

When you encode a given text, each word in one-hot encoded, so the text becomes a vector of the length that equals size of the dictionary (classical bag of words approach). That usually goes into thousands (e.g. Yelp review dataset has over milion unique words) so clearly this is a highly dimensional problem.

There is a number of techniques then to reduce this dimentionality, like embeddings of LDA.

---

Appendix for the question in comment:

When you decide for the number of feature (which is number of words that matter for your taks), usually you exlude stop words (like 'a', 'an', 'the' and others that are very popular in texts but does not carry meaning for e.g. classification). I usually manually select them from frequency dictionary (list of words with counts in corpus, sorted by the count). You can compare your list with that used in NLTK.

And from the other side of the dictionary, you would usually skip words that occur very rare, like less than 3 times (number depends on the corpus).

So you end with your one dictionary which has it's size and this is the number of your features.