Hot answers tagged

2

The reason the other bars are missing is because of your method of summing the values and the missing values in your dataset. The way you are adding the values together means that if even one value is missing (NA) the total for that column will be missing as well, and as a result, will not be in your final plot. It is better to use pandas built-in methods (...


1

Well, there are a few ways to do the job. Here are some I thought of: Scatterplots with noise: Normally, if you try to use a scatter plot to plot two categorical features, you would just get a few points, each one containing a lot of instances from the data. So, to get a sense of how many there really are in each point, we can add some random noise to each ...


1

Do you only need to differentiate between one or more spaces? If so, maybe you can make use of a regular expression separator as listed in the pandas docs. Something like this: pd.read_csv(... sep='\s+', ...) Also, consider adding the parameter delim_whitespace as such: pd.read_csv(..., delim_whitespace=True, ...)


1

When imputing data, one is looking not to modify the true distribution of your data. So a way to test how good your imputation was is to make a test to contrast the true distribution of every feature that has been imputed vs the true (via KS test for example) distribution of the feature (prior imputing) if you can sate with a level. of confidence that your ...


1

According to my understanding PCA requires that you have the column of equal length, so you either need to shorten the longer columns (basically just skip the incomplete observations) or fill in the gaps in the shorter columns. If you choose the second option, you'll need to learn about the concept of imputation (see the following link for reference ...


1

You could simply drop the index to create and use it as a primary key column. all_data.reset_index(inplace=True) all_data.head()


1

Because on each loop you assign a single value to the entire column: tots_df['orc_4_totals'] = s4 # entire column orc_4_totals = s4! (the same thing for the other column. It equals all ones then, just because your final loop value fills it with 1. You need to insert single values instead, for each row of your target dataframe. However, because I don't ...


1

You may use df.groupby(['BirthDate', 'ZipCode']).size().reset_index().rename(columns={0: 'n'}) and now you have a data frame that you can easily manipulate.


1

See also this stackoverflow answer, if you just want the unique values you can use pandas.Series.unique() or pandas.DataFrame.drop_duplicates(). If you need the python set object you can use set(df['colname']).


1

I think that indeed you may have leakage by using pd.qcut. A solution to avoid that leakage is to do it in a time-series cross-validation fashion. The idea is to derive the quantile values in a training fold, and, with those values, do the cut in its validation fold. This is a little complex and if you want it simpler you can use pd.cut, which will have no ...


1

This may be simpler using sets. Assuming your dataframe is df, first get each column's unique values as a set: import pandas as pd from functools import reduce # df = pd.read_clipboard() cols = df.agg(set) print(cols) This gives a pandas Series of python set objects: CL1 {c, s, d, a, x, f, b} CL2 {c, dc, y, s, d, a, x, dx} CL3 {c, s, y,...


1

Given your input data is saved in a variable df, I count the values which occur in all 4 unique columns as follows: import pandas as pd import numpy as np output = ( df .melt() .drop_duplicates() .groupby("value") .agg(count=("value", "count")) .reset_index() ) output["SIM"] = np.where(output[&...


Only top voted, non community-wiki answers of a minimum length are eligible