New answers tagged

0

When the dimension is high, all data are approximately at the same distance away from each other. Yes, but this is the worst case scenario of the curse of dimensionality: This happens only if the data is extremely sparse. In real cases it's not really "all data", because some data points are less sparse than others, and it's not really "the ...


0

The column df['text'] was of type 'object', which was a byte type, so the pandas Series contained b"foo", etc. The only change to make was to decode the object using: .str.decode('utf-8') df = tfds.as_dataframe(ds.take(4)) reviews = df['text'].str.decode("utf-8") corpus = reviews.tolist() print(corpus) tokenizer=Tokenizer(num_words=100) ...


0

For TypeErrors it is always good to check the exact type of the variable causing issues, especially when troubleshooting TensorFlow and Keras: print(type(df['text'])) fit_on_texts expects a list of string or similar, but you are providing a dictionary, so you'll want to convert accordingly. Example: from typing import List, Dict foo_dict: Dict[str, str] = {...


1

This answer emphasizes an intuitive understanding since the OP is a beginner. (1) PCA can be used for Feature Selection, in a special case, when the features are already uncorrelated and the 'relevant' features are embedded in a lower-dimensional sub-space. (2) PCA can be used for Feature Extraction, when the features are correlated. Based on variance of the ...


1

PCA reduces dimensionality by generating orthogonal (i.e. uncorrelated) "new" features from the original features. This can be very useful, e.g. when you do linear regression. So principle components "transform" the original features to orthogonal ones. You can see this as some kind of feature engineering. Usually PCA will reduce the ...


3

From my understanding you are working on a regression task in which you have applied MainMaxScaler to your target variable y prior modeling. If so you have two options: As the error message suggests, you can reshape the output with array.reshape(-1, 1) Scikit learn has implemented a class to work with transformations on target: So just try from sklearn....


0

There are a couple of design patterns that are contributing to slow code: Pandas is not designed for large-scale, fast data processing. Your code is using a for loop which can be slow. You are manifesting the sliding window before the program needs it. It might be better to create a view on the data and then manifest the data in memory only when the ...


0

There are several options: Change the training dataset Use some of the test data as training data. This the best option since it better models the problem you are trying to solve. Since it happens over time, take only the most recent data for training. Manually engineer features. If you have knowledge of how the test data feature values are different, ...


1

Overall, scikit-learn is designed to be not inefficient but it's goal is not hyperefficiency. If your goal is hyperefficiency then you should switch to a different machine learning package. The goal of scikit-learn is to provide a high level interface with machine learning and handle the implementation details "under the hood". That is why there is ...


0

Here are couple of other options: Set a threshold and remove all values larger than the threshold RobustScaler which removes the median and scales the data according to the quantile range. QuantileTransformer which transforms the feature to follow a uniform or a normal distribution.


1

It seems very unlikely that centering would hurt, and so I'd suggest just to do it anyway. Theoretically, in a generalized linear model with regularization, no, centering won't change anything. This is because the intercept term can absorb any changes; shifting $x$ by 100 can simply be rewritten: $$ 15 + 0.2*(x-100) = 15 - 0.2\cdot100 + 0.2x = -5 + 0.2x,$$ ...


Top 50 recent answers are included