New answers tagged

0

You can try using this to help: https://github.com/openvenues/libpostal libpostal looks like it can normalize across various geographic styles with the expand addresses functions.


0

You can do dummy encoding using Pandas in order to get one-hot encoding as shown below: import pandas as pd # Multiple categorical columns categorical_cols = ['a', 'b', 'c', 'd'] pd.get_dummies(data, columns=categorical_cols) If you want to do one-hot encoding using sklearn library, you can get it done as shown below: from sklearn.preprocessing import ...


0

df['item_id'] = df['item_id'].apply(lambda x: x.split('_')[0])


1

LabelEncoder is not made to transform the data but the target (also known as labels) as explained here. If you want to encode the data you should use OrdinalEncoder. If you really need to do it this way : categorical_cols = ['a', 'b', 'c', 'd'] from sklearn.preprocessing import LabelEncoder # instantiate labelencoder object le = LabelEncoder() # apply ...


1

Slight addendum to the above solution: ##First create a mapping between the ID values and the filenames: # Use the "_" to split the filename and take the first items, the ID mapping = {f.split("_")[0]: f for f in movie_filenames} # <-- a dictionary-comprehension ##Now iterate over this mapping, inserting the movie filenames for the correct movie ...


2

I assume you a list of the filenames called movie_images # Could get filenames with: # import os; movie_images = os.listdir("./folder/with/images/") movie_filenames = ["11_lion_king.jpg", "22_avengers.jpg"] First create a mapping between the ID values and the filenames: # Use the "_" to split the filename and take the first items, the ID mapping = {f....


0

df['item_id']=df['item_id'].apply(lambda x: x[:5])


0

So two things: 1) The loc operator can only accept a boolean condition, therefore you need to convert dt to datetime in a separate operation. 2) I think the condition wasn't working because the notnull() function does not apply to np.nan. The right way is to use is the isna() function with the negation ~. I edited the code above to reflect these changes, ...


0

Building on @nemo's answer (above) which will be faster than the accepted solution, this will give the same output that you want: def formatRecords(g): keys = ['value1', 'value2'] result = [] for item in g.values.tolist(): item = dict(zip(keys, item)) result.append(item) return result df_dict = df.groupby('name').apply(...


0

Try using groupby with a custom function and apply. df = pd.DataFrame() df['RunNumber'] = ['Run1','Run1','Run2','Run2'] df['A']=[500,60,20,30] df['B']=[5,5,2,2] df['C']=[5,10,2,6] df['D']=[3,4,5,4] df['E']=[65,56,56,44] def calculate_sum_percent(input_df): output_df = pd.Series() output_df['AB_sum']=input_df['A'].sum() + input_df['B'].sum() ...


2

Those are hex numbers, aka base 16 digits. Hex numbers in Python are represented as strings that start with "0x". Either keep them as hex or convert them to base 10 integers: int("0x000b", 16) #=> 11 int("0xc0a8", 16) #=> 49320


5

Those symbols you are talking about are hex values. They could actually be useful to keep in the dataset, as the tcpdump tool outputs them if the right options are set. Despite that, you can replace occurrences of them using pandas and a simple regular expression. A regular expression that matches hex values is: 0[xX][0-9a-fA-F]+ I downloaded one of the ...


1

As someone who worked on a competitor to Tableau, Data Science skills have largely superseded the need for Bi Software for data munging, complex analysis and ad hoc reports. But BI Software can still be beneficial if you need to deploy your results to lots of people, often with varying rights to view something (e.g. you can only see your performance stats, ...


6

There is the official answer and the realistic answer (from a business perspective): Official Officially the greatest thing your Python skills will bring you is flexibility. If you are going to run some economical model where you want to show a gradient uncertainty or something else crazy, doing that manually in any Data Visualization/Business Intelligence ...


10

Don't worry - your hard-earned Python skills are still important ;) Tableau is not a replacement - it is essentially a means of sharing your insights/findings. It is a wrapper around your normal toolkit (Pandas, Scikit-Learn, Keras, etc.). It can do some basic analysis (just using basic models from sklearn), but the powerful thing is it can deploy your ...


1

Good morning, There is nothing wrong with your data, your main mistake is that you need to pay attention to the plot function used. This link tells you how to plot your data. https://matplotlib.org/tutorials/introductory/pyplot.html However I would change the last line to something like: import matplotlib.pyplot as plt plt.plot(y_rus) plt.show() or ...


3

First: I think you want the product functionality, not zip, since you are checking every df with every ref. In zip, you would check df_a with ref_1 and df_b with ref_2 only. Second: Your can look at the equation $(1+2+3+4)−(5+5+5+5)$ as $(1-5) + (2-5) + ...$ which is simply subtracting data frames and sum over columns. With these two consideration, ...


1

What you want to do is exactly the default behavior of the category type. Convert your month value to the type category declaring all months (it has a somewhat weird interface to create a categorical type) df.month= dd.month.astype(pd.api.types.CategoricalDtype(categories=range(12))) df.month.value_counts() will give you: id code month sally s_A ...


0

I think the following lines should give the output you are looking for: # Create pivot table df = pd.pivot_table(df, values="value", index="country", columns="year") # Calculate cumulative sum and forward fill NaN df = df.cumsum(axis=1).fillna(method="ffill", axis=1) # Reshape data back into long format df = df.reset_index().melt("country") I first create ...


0

len(df1.date.unique()) = 789 len(df1.unit.unique()) = 76 len(df1.company.unique()) = 205 len(df1.city.unique()) = 237 That gives 2,913,350,940 possible combinations, yet you say your dataframe has only 1360 rows. Try creating a column that is built from concatinating the four columns, then doing groupby that.


0

I bet that company and unit are category type? I couldn't explain the underlying reason, but groupby doesn't like the category type. Change your column type to 'object' and it will run in a couple of milliseconds without consuming any memory


0

output = df2[df2['Name'].isin(df1['Name'])


0

Try the below code, it should do the trick for you df1['Age'] = df1['Name'].replace(df2.set_index('Name')['Age'].dropna()) df1['Sex'] = df1['Name'].replace(df2.set_index('Name')['Sex'].dropna()) print(df1)


0

Short answer: NO. The moment you convert the spark dataframe into a pandas dataframe, all of the subsequent operations (pandas, ml etc.) will be run on a single-core as those algorithms and programs are written in native-python and doesn't support multi-core distributions. In a nutshell, someone has to rewrite the whole sklearn to again to be compatible ...


0

Create a user defined function to fill null values of each group >> def func(df): df['B'] = df['B'].fillna(df['B'].mean()) return df >> df.groupby('A').apply(func) // gives required dataframe A B 0 1 20.0 1 2 30.0 2 3 40.0 3 2 30.0 4 3 40.0 5 1 20.0 Or use the below snippet if A column is not required in output df....


1

Well, it always depends, for example, on what model you might be training (i.e. some are robust to multicollinearity). I am pretty sure you are aware, but to have it said as a rule of thumb it is always helpful if you know what you are looking for, rather than hoping naively one function or method would give all the answers. Said that, there are good ...


0

From what I can understand is that you would need a histogram of your receipt no. You can try something like this import pandas as pd data = np.read_csv("your_file_path.csv") data.groupby(["receipt"])receipt.count().sort_values(ascending=False).head(20).plot.bar() This will give you bar plots of most repeating billing numbers (20 most repeating) Change ...


0

It seems like you want to reduce the records in the data if Col1 is a subset of another record in Col1. First you can sort the column alphabetically, and then use the function below. import pandas as pd df = pd.DataFrame() df['Col1'] = ['X','X,Y','Z','Z,W'] def test(x): return df.Col1.apply(lambda y: y in x).sum()>1 mask = df.Col1.apply(lambda x: ...


Top 50 recent answers are included