54

Simply put: feature selection: you select a subset of the original feature set; while feature extraction: you build a new set of features from the original feature set. Examples of feature extraction: extraction of contours in images, extraction of digrams from a text, extraction of phonemes from recording of spoken text, etc. Feature extraction ...


43

First of all, dimensionality reduction is used when you have many covariated dimensions and want to reduce problem size by rotating data points into new orthogonal basis and taking only axes with largest variance. With 8 variables (columns) your space is already low-dimensional, reducing number of variables further is unlikely to solve technical issues with ...


29

What is a dimension? To put it simply, if you have a tabular data set with m rows and n columns, then the dimensionality of your data is n: What is a manifold? The simplest example is our planet Earth. For us it looks flat, but it really is a sphere. So it's sort of a 2d manifold embedded in the 3d space. What is the difference? To answer this ...


20

Adding to The answer given by Toros, These(see below bullets) three are quite similar but with a subtle differences-:(concise and easy to remember) feature extraction and feature engineering: transformation of raw data into features suitable for modeling; feature transformation: transformation of data to improve the accuracy of the algorithm; feature ...


18

Dimensionality reduction is typically choosing a basis or mathematical representation within which you can describe most but not all of the variance within your data, thereby retaining the relevant information, while reducing the amount of information necessary to represent it. There are a variety of techniques for doing this including but not limited to PCA,...


17

The dimensions of the low dimensional space have no meaning. Note that the t-SNE loss function is solely based on the distances between points ($y_i$ and $y_j$) and probability distributions over those distances ($p_{ij}$ and $q_{ij}$): $$ \frac{\delta C}{\delta y_i}=4 \sum_j(p_{ij} - q_{ij})(y_i-y_j)(1+||y_i -y_j||^2)^{-1} $$ Thus there is no projection ...


16

One thing to start off with would be k-NN. The idea here is that you have a user/item matrix and for some of the users you have a reported age. The age for a person in the user item matrix might be well determined by something like the mean or median age of some nearest neighbors in the item space. So you have each user expressed as a vector in item space,...


16

I hope that the following resources might get you additional ideas toward solving the problem: 1) Research paper "Efficient K-Nearest Neighbor Join Algorithms for High Dimensional Sparse Data": http://arxiv.org/abs/1011.2807 2) Class project paper "Recommendation System Based on Collaborative Filtering" (Stanford University): http://cs229.stanford.edu/...


14

You must look at this Multicore implementation of t-SNE. I actually tried it and can vouch for its superior performance.


14

There is no specific constraint on the symmetry of an autoencoder. At the beginning, people tended to enforce such symmetry to the maximum: not only the layers were symmetrical, but also the weights of the layers in the encoder and decoder where shared. This is not a requirement, but it allows to use certain loss functions (i.e. RBM score matching) and can ...


12

Have you heard of Uniform Manifold Approximation and Projection (UMAP)? UMAP (Uniform Manifold Approximation and Projection) is a novel manifold learning technique for non-linear dimension reduction. UMAP is constructed from a theoretical framework based in Riemannian geometry and algebraic topology. The result is a practical scalable algorithm ...


11

This is very broad question, which I think it's impossible to cover comprehensively in a single answer. Therefore, I think that it would be more beneficial to provide some pointers to relevant answers and/or resources. This is exactly what I will do by providing the following information and thoughts of mine. First of all, I should mention the excellent and ...


11

First of all your explanation about the methods are right. The point is that Embedding algorithms are not to only visualize but basically reducing the dimentionality to cope with two main problems in Statistical Data Analysis, namely Curse of Dimentionaliy and Low-Sample Size Problem so that they are not supposed to depict physically understood features and ...


11

I take Natural Language Processing as an example because that's the field that I have more experience in so I encourage others to share their insights in other fields like in Computer Vision, Biostatistics, time series, etc. I'm sure in those fields there are similar examples. I agree that sometimes model visualizations can be meaningless but I think the ...


11

One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power. I have also seen people map 1-hot categorical values to lower-dimensional vectors, where each 1-hot vector is re-represented as a draw from a multivariate Gaussian. See e.g. ...


10

Richard Hamming is attributed with the sentence: "The purpose of computing is insight, not numbers." In this 1973 academic paper (see discussion in What is the famous data set that looks totally different but has similar summary stats?), Francis Anscombe argues that "graphs are essential to good statistical analysis." Anscombe's quartet is a long time ...


8

Your question is definitely in place, however I found that any question in the format of "should I do X or Y in deep learning ?" has only one answer. Try them both Deep learning is a very empirical field, and if a non-symmetric auto-encoder works for your domain, then use it (and publish a paper)


8

Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data) Autoencoder is a type of neural network where the inputs and outputs are the same but in the hidden layer the dimensionality is reduced in order to get a more dense representation of the data. ...


7

As in @damienfrancois answer feature selection is about selecting a subset of features. So in NLP it would be selecting a set of specific words (the typical in NLP is that each word represents a feature with value equal to the frequency of the word or some other weight based on TF/IDF or similar). Dimensionality reduction is the introduction of new feature ...


7

I recently did a similar project in Python (predicting opinions using FB like data), and had good results with the following basic process: Read in the training set (n = N) by iterating over comma-delimited like records line-by-line and use a counter to identify the most popular pages For each of the K most popular pages (I used about 5000, but you can play ...


7

Don't bother. First rule of programming- which also applies to data science: get everything working on a small test problem. so take a random sample of your data of say 100,000 rows. try different algorithms etc. once you have got everything working to your satisfaction, you can try larger (and larger) data sets - and see how the test error reduces as you ...


7

Check out FFT-accelerated Interpolation-based t-SNE (paper, code, and Python package). From the abstract: We present Fast Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE), which dramatically accelerates the computation of t-SNE. The most time-consuming step of t-SNE is a convolution that we accelerate by interpolating onto an ...


6

PCA is usually implemented by computing SVD on the covariance matrix. Computing the covariance matrix is an embarrassingly parallel task, so it scales linear with the number of records, and is trivial to distribute on multiple machines! Just do one pass over your data to compute the means. Then a second pass to compute the covariance matrix. This can be ...


6

The dimensionality of a dataset is the number of variables used to represent it. For example, if we were interested in describing people in terms of their height and weight, our "people" dataset would have 2 dimensions. If instead we had a dataset of images, and each image is a million pixels, then the dimensionality of the dataset would be a million. In ...


6

Since, there are no answers in SO, I have asked myself in github page and the issue has been closed by stating the following reply by GaelVaroquaux.. If you only want to parallelise vector operation, then you should use a build of numpy compiled with MKL (don't attempt to do it yourself, it's challenging). There could be approaches to high-level ...


6

Try UMAP. It's significantly faster than t-SNE.


5

You can definitely try to first cluster your data, and then try to see if the cluster information helps your classification task. For example if your data looked like this (in 1D): AA A AA A A BBB B B B BB BB BB AA AA A A AAA then it may be reasonable to run a clustering algorithm on each class, to obtain two different kinds of A, and learn two ...


5

Feature selection is about choosing some of features based on some statistical score but feature extraction is using techniques to extract some second layer information from the data e.g. interesting frequencies of a signal using Fourier transform. Dimensionality reduction is all about transforming data into a low-dimensional space in which data preserves ...


5

Another suggestion is to test the logistic regression. As an added bonus, the weights (coefficients) of the model will give you an idea of which sites are age-distriminant. Sklearn offers the sklearn.linear_model.LogisticRegression package that is designed to handle sparse data as well. As mentionned in the comments, in the present case, with more input ...


5

There is a paper on this subject calle Simple and Effective Dimensionality Reduction for Word Embeddings, Vikas Raunak You can read it here You can also find the implementation here In my opinion it works quite well


Only top voted, non community-wiki answers of a minimum length are eligible