Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
10
votes
2answers
75k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
6
votes
2answers
9k views

pandas count values for last 7 days from each date

There are two Dataframes. First is like this: ...
9
votes
1answer
5k views

How to binary encode multi-valued categorical variable from Pandas dataframe?

Suppose we have the following dataframe with multiple values for a certain column: categories 0 - ["A", "B"] 1 - ["B", "C", "D"] 2 - ["B", "D"] How can we ...
2
votes
0answers
51 views

Can I conclude my finding with just one linear regression result? [closed]

I'm trying to learn Linear Regression and I'd like someone to look at my notebook to see if I miss out anything at all. How how else I can tune my model better. Here's my notebook. I get the ...
4
votes
2answers
7k views

Pivoting a two-column feature table in Pandas

How can I transform the following DataFrame into one with cities as rows and each cuisine as a column, and 1 or 0 as values (1 if the city has that kind of cuisine)? I think this turns out to be a ...
5
votes
3answers
4k views

Pandas time series optimization problem: add year

I have a pandas DataFrame containing a time series column. The years are shifted in the past, so that I have to add a constant number of years to every element of that column. The best way I found is ...
1
vote
2answers
2k views

Split columns by finding “£” and then converting to minvalue, maxvalue

I have a pandas dataframe with a salary column which contains values like: £36,000 - £40,000 per year plus excellent bene..., £26,658 to £32,547 etc I isolated this column and split it with the ...
1
vote
0answers
833 views

pandas dataframes memory

I have a question about memory usage. I want to do 4 things: ...
3
votes
1answer
771 views

MovieLens data set

I want to analyze MovieLens data set and load on my machine the M1 file. I combine actually two data files (ratings.dat and movies.dat) and sort the table according ...
2
votes
2answers
1k views

how to impute missing values on numpy array created by train_test_split from pandas.DataFrame?

I'm working on the dataset with lots of NA values with sklearn and pandas.DataFrame. I implemented different imputation strategies for different columns of the dataFrame based column names. For ...
1
vote
0answers
441 views

How to fix similarity matrix in Pandas returning all NaNs?

I've been trying to create a similarity matrix in Pandas from with a matrix multiplication operation on a document-term count matrix with 2264 rows and 20475 columns. The calculation completes in ...
7
votes
2answers
5k views

Struggling to integrate sklearn and pandas in simple Kaggle task

I'm trying to use the sklearn_pandas module to extend the work I do in pandas and dip a toe into machine learning but I'm struggling with an error I don't really understand how to fix. I was working ...
3
votes
2answers
1k views

built-in cov in pandas DataFrame results ValueError array is too big [closed]

So, I have a dataset with 39.949 variables and 180 rows. dataset is successfully saved in DataFrame but when I try to find cov() it result an error. here is the code ...
16
votes
2answers
907 views

Where in the workflow should we deal with missing data?

I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data ...
25
votes
4answers
13k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...