7

First most of the time there's no "missing text", there's an empty string (0 sentences, 0 words) and this is a valid text value. The distinction is important, because the former usually means that the information was not captured whereas the latter means that the information was intentionally left blank. For example a user not entering a review is ...


3

I summarize your questions and then try to answer under each bullet point: How to remove punctuation marks (e.g. # for hashtags which is used in social media) The first goto is a regular expression that is used in data preprocessing very frequently. But if you are looking for all your punctuation to be removed from the text you can use one of these two ...


3

It looks like that your prediction is clamping at 750. Be mindful of the fact that Tree can't predict a Regression value that is outside the range it has been trained on. So, first of all, please assure that your data doesn't have a trend.


3

I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though. OrdinalEncoder is capable of encoding multiple columns in a dataframe. So, when you instantiate OrdinalEncoder(), you give the categories parameter a list of lists: enc ...


2

Of course, making them smaller will cause loss of information. Since before you had N x M data points for each image to describe the content of the image, but after resizing you will have n x m (Where n and N denote the number of rows in an image, m and M denotes the number of columns in an image. Also, n <= N and/or m <= M). That is, a small number of ...


2

It sounds like you are looking for an unsupervised learning approach (meaning you don't need to manually label your data). Something like k-means clustering could work well. This would allow you to group you comments into k distinct clusters. You could then view counts of comments in those clusters and explore the clusters to determine their meaning. In ...


2

EDA What you are doing there, falls under Exploratory Data Analysis (EDA). A better way to investigate how your features between them and are distributed across classes is a correlogram, sometimes referred to as pairplot. A correlogram helps with investigating relationships between pairs of numerical features, via scatter plots for each pair of features ...


2

Imputing all missing values before the split would mean data leakage - you would use test set information to influence the training set. I had a similar problem recently, I ended up using complete cases of the source data for the test set and then imputing the training data using medians calculated by class. Otherwise, you might split the data and then ...


2

Despite the fact that I'm not sure why it's that, it's seems that's normal. Please see the example from sklearn documentation: https://scikit-learn.org/stable/modules/impute.html#multivariate-feature-imputation import numpy as np from sklearn.experimental import enable_iterative_imputer from sklearn.impute import IterativeImputer imp = IterativeImputer(...


2

The accepted answer will work, but will run df.count() for each column, which is quite taxing for a large number of columns. Calculate it once before the list comprehension and save yourself an enormous amount of time: def drop_null_columns(df): """ This function drops columns containing all null values. :param df: A PySpark ...


2

That to me doesn't seem like messy data at all, it is just a csv file with a ; delimiter. Depending on the region settings excel can use different delimiters when saving data as .csv file, ; being one of them. By default pandas assumes a , as the delimiter, which in this case does not apply. Try reading it in by specifying the correct delimiter using the sep ...


2

If your predictors have nothing to do with the outcome, you should not be able to build a model that works out-of-sample. This is a feature, not a bug, of machine learning. For instance, do you consider what time I set my alarm in the morning to be predictive whether or not you have cereal for breakfast? Features can, however, have just a small relationship ...


1

See also this stackoverflow answer, if you just want the unique values you can use pandas.Series.unique() or pandas.DataFrame.drop_duplicates(). If you need the python set object you can use set(df['colname']).


1

I believe you are looking to work along with the missing values in particular column(X) where column(W,Y,Z) have important values in these rows and can't discard or do imputation, especially for plotting them visually. Yes its possible, considering: When you only plan to plot other columns(W,Y,Z excluding column X) to view them visually When you only plan ...


1

If using OLS for feature selection and a tree-based algorithm for your model, you don't need to worry about standardizing or scaling your data. You want to scale your data when using an algorithm for feature selection or modeling that has a distance-based component. Then you want to have all X variables on a similar scale.


1

I might misunderstand something but it looks to me like you're trying to find a complex method for a simple problem: if there are many strings which occur multiple times in the list, you should deduplicate the list before comparing all the pairs. You could use a set, but since you will need to count how frequent each string is you should probably directly ...


1

A simple two part solution from this site remove any letter sequences longer than two (probably not good for welsh) def reduce_lengthening(text): pattern = re.compile(r"(.)\1{2,}") return pattern.sub(r"\1\1", text) print(reduce_lengthening( "finallllllly" )) Then using pattern.en to check spelling. from pattern.en ...


1

Community analysis implies graph analysis. here is a short list of things you can work on: People often reshares tweets among a certain social group. Minimum-cut method, Girvan–Newman and Modularity maximization are someof the starting algorithms to extract these type of substructures. You can try and find different hierarchies among the groups sharing a ...


1

A slightly hacky way to get there maybe but you can do this to get what you want from the second table; df2['count'] = 1 pivot = df.pivot_table(df, index='userid', columns='productid', values = 'count').reset_index() pivot = pivot.fillna(0) You would then want to merge this to the first dataset like this; finaldf = pd.merge(df1, pivot, left_on='userid', ...


1

If my understanding is right, you have a regression problem, with categorical features with high cardinality and "outliers" (or just big numbers). How have you encoded categories? Target Encoding? There is another option that is not encoding with the mean but with the median that on some cases can perform better. On this notebook , you can see an ...


1

If you have a really small dataset like a few 100 samples it's okay to do preporcessing via hand. But since you have a thousand samples it's better to automate the process. You can use the na_values attribute in pandas to fill in the "???" or "??" value with nan. Then for each column replace the Nan values using some statistical measure (...


1

In a first iteration, use a sentence encoder.You can find pre-trained model on tensorflowhub (https://www.tensorflow.org/hub/tutorials/semantic_similarity_with_tf_hub_universal_encoder), spacy (https://spacy.io/universe/project/spacy-universal-sentence-encoder) or huggingface (https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens) to name a ...


1

The pivot operator can be used to achieve the desired result. As per your example: Define original data: import pandas as pd data = {'Txn_ID': [1, 1, 2, 2, 2, 3], 'Item_Desc': ['Apples', 'Milk ', 'Eggs', 'Flour', 'Salt', 'Eggs'], 'Amt': [0.96, 2.10, 7.00, 2.20, 4.75, 3.50] } Create dataframe df = pd.DataFrame.from_dict(data) Transpose ...


1

Thank you for the above answer! That definitely works. However, I found a more efficient way in terms of computation using np.rolling df['D'] = df['A'].rolling(min_periods=1, window=3).mean() df['B'] = np.where(df['B'].isnull,df['D'],df['B']) np.rolling helps to compute the cumulative sum of previous n values. np.where helps to apply some output based on a ...


1

Strictly theoretically it makes no difference on DNN, I answered it today here and I said: Here is why: We already know mathematically that NN can approximate any function. So lets say that we have Input X. X is highly correlated, than we can apply a decorrelation technique out there. Main Thing is, you get X` that has different numerical representation. ...


1

You probably should conduct a missing values analysis to see what is the percentage of missing per column (figure below, from dataprep package) Decide a threshold according to which you may want to completely drop a column or not (depending on how your analysis or model treats nans as well) For the columns that are not dropped, you should impute the missing ...


1

Aren't you providing the answer? You can split the feature in two, namely, if feature_to_split is the feature you're talking about, you can create feature_to_split_ispresent which will take either 1 or 0 depending on the presence or absence of that specific characteristic, and feature_to_split_value which will take the actual value of that characteristic.


1

When you have high cardinality I suggest you two options (one doesn't exclude the other): Aggregate least frequent categories into one called other Use Neural Network with an embedding layer for high cardinality categories. In case you don't know what an embedding is, it is just a table that map each category into a vector of k features (k your choice)


1

$\hat{p}=2\big(\frac{p}{p_{max}}\big)-1$ so that $\hat{p}\in [-1,1]$ where $p\in[0,p_{max}]$ E.g. with 8-bit channels, $p_{max}=255$, black/white correspond to $-1$ and $1$ respectively, and grayscale is linearly mapped in $[-1,1]$.


1

The short answer is that usually if you are performing data transforms, you fit on the training data and transform training and test data. The long answer is that if you are doing purely supervised machine learning then you want to avoid "leakage". Data leakage is when information from outside the training dataset is used to create the model. ...


Only top voted, non community-wiki answers of a minimum length are eligible