Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
5
votes
2answers
17k views

Merging large CSV files in pandas

I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
1
vote
2answers
5k views

Replacing column values in pandas

I have a data frame which has three columns as shown below. There are about 10,000 entries in the data frame and there are duplicates as well. ...
2
votes
1answer
25 views

How to fill NaN based on the previous row that contains a specific value [closed]

I would like to fill the missing data using the value from the previous trading day of the same stock. In this example, the AAPL stock should be 100. I've tried with ...
-2
votes
0answers
12 views

isnull(). returning incorrect figure [duplicate]

In the below code, I am trying to bind the prediction result in new dataframe. However, in new dataframe it is showing incorrect values. I do not know why ...
1
vote
4answers
29k views

String Values in a data frame in Pandas

Suppose I have a data frame like this : ...
0
votes
2answers
39 views

Building a summary string in a Pandas groupby (Possibly cross-tab or pivot-table question)

I'm a n00b in Python Pandas. I want to do a particular aggregation / grouping / cross-tab but I know so little of the terminology that I don't even know how to look this up. But here's what I want. ...
2
votes
2answers
290 views

Feature Engineering

I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below: ...
1
vote
1answer
1k views

Generating a sequence based on value in another column in Python

I have the following data frame: I would like to add a column with value equal to 1 if flag is 0 and incrementally add 1 in the following rows until the next 0 is ...
1
vote
1answer
40 views

Running scikit-learn with large volume

I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the ...
2
votes
1answer
6k views
6
votes
2answers
7k views

Plotting different values in pandas histogram with different colors

I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used <...
0
votes
1answer
270 views

partial fitting, how to ensure one hot captures all features consistently

Doing some data science on ~4 million samples, with lots of columns being categorical. One column has ~1000 categories and my boss insists on including it in the analysis. My output is also ...
0
votes
2answers
20 views

Replace value of a column if the value of another column is a duplicate

I have a Pandas dataframe that contains three columns: ID, name and date. The name column is not unique and may contain duplicates. I want to change the date of the duplicated name to the earliest ...
0
votes
2answers
32 views

How to apply Geomertic mean in python

I am trying to built a function to calculate the Geometric mean for Growth column and Avg_growth. Year Company Name FH_PBIDT ...
0
votes
1answer
33 views

Ordering a material science dataset (properties names, properties scalars, formulas)

I'm dealing with a materials science dataset and I'm in the following situation, I have data organized like this: ...
0
votes
1answer
2k views

ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

I want to create a new variable for the dataframe details called lower after iterating through multiple ...
78
votes
10answers
267k views
0
votes
1answer
26 views

Pandas replace column values by condition with averages based on a value in another column

I have a dataframe with people's CV data. Among others, there's a column with years of experience, and a column with age. Some people stated their age and experience in a way that experience > age. ...
1
vote
1answer
180 views

Combine results from multiple models

I am using chunks of 100000 rows at a time from the CSV file to train the a simple LASSO model. How do i combine all of these models trained from these different chunks? I would like to use all ...
25
votes
4answers
19k views

Is pandas now faster than data.table?

https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping The data.table benchmarks hasn't been updated since 2014. I heard somewhere that Pandas is ...
112
votes
11answers
65k views

Why do people prefer Pandas to SQL?

I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle. The vast majority of the operations I've seen done ...
0
votes
2answers
5k views

How to measure correlation between several categorical features and a numerical label in Python?

I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product ...
2
votes
1answer
316 views

Convert Pandas Dataframe with mixed datatypes to LibSVM format

I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
0
votes
2answers
31 views

How to find if any variable which is normally distributed in pandas dataframe [closed]

Im having age column and how can i plot and say this is normally distributed or not
0
votes
0answers
14 views

replacing last duplicate values in pandas

I have a dataframe ,that looks like this | **site** | **Active**| 0 |: deals| Active:| 1 |: deals | InActive:| 2 | : deals | Active:| 3 |:...
0
votes
1answer
17 views

replace values based on Number of duplicate rows are occured

I have a dataframe ,that looks like this site Active 0 deals Active 1 deals Active 2 deals Active 3 discount Active 4 discount Active i ...
0
votes
0answers
8 views

Demand Forecasting - Error in predictions

I've been trying to do daily demand forecasting using the H2O AutoML package for using 4 years of daily sales data as the training set. However, the daily estimates always seems to find a range around ...
0
votes
1answer
82 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
3
votes
3answers
22k views

How to download a Jupyter Notebook from GitHub?

This is a fairly basic question. I am working on a data science project inside of a Pandas tutorial. I can access my Jupyter notebooks through my Anaconda installation. The only problem is that the ...
0
votes
0answers
25 views

ValueError: Found input variables with inconsistent numbers of samples: [10000, 11200]

I am receiving the following error. I can't seem to figure out the exact change I need to make to my code to get solve this problem. I feel like it should be straightforward but I'm banging my head ...
0
votes
1answer
3k views

how to export the tables into a csv file pandas

The following is a piece of code I wrote to create a pivot table for categorical vs continuous variable. ...
0
votes
1answer
176 views

Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data

I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with ...
1
vote
1answer
25 views

Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below: ...
2
votes
1answer
443 views

memory problem due to large file

I'm new in Python and definitely, I'm sure I do some mistakes. Here is my problem and thanks all for your help in advance. I have 2 files (one is Hive) and the other is ...
7
votes
2answers
26k views

Perform k-means clustering over multiple columns

I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not ...
0
votes
0answers
16 views

Using Downsampling with RFE using pandas and sklearn

I have dataset which consist of 2 class with heavily unbalance distribution (600 for class A and 3000 for class B). When i do RFE to select our feature and test them using Decission tree, it gives a ...
0
votes
1answer
19 views

My small script on value alteration in columns of a data not working

I have a data set which has "Speed" as one of the columns (features). The column contains both zero and non-zero values. I want to randomly set 10% of the non-zero values to zeros. This will change ...
0
votes
3answers
2k views

Remove columns with a certain number of consecutive zeros

I'm using pandas and I am dealing with time series of sales. What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...
0
votes
1answer
23 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
0
votes
0answers
20 views

Applying word embedding function on a large pandas dataset

We have a pandas dataframe with one column ('message') and 3.9 million rows and need to convert these messages to their word embeddings using Google's Universal Sentence encoder. ...
6
votes
6answers
14k views
0
votes
1answer
477 views
0
votes
0answers
9 views

How should I add reference key column to output after modeling is Complete?

0 I have created a csv of data with the following columns: (1) app_key (2) churn, (3)tenure https://i.stack.imgur.com/NAlFF.png I have performed the following code in order to drop app_key and churn <...
-1
votes
1answer
17 views

Similar values cleaning [closed]

can someone know algorithm how to identify account names that are similar enough to be potentially merged and imported as one Duplicates with different values: Geico val1 NaN =====>>...
1
vote
1answer
56 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
-1
votes
1answer
27 views

How can I get all column values from a panda dataframe and save them into one string [closed]

this question has been ask couple of times but none of the answer is working for that what I want. For example I have this excel Dokument: ...
1
vote
1answer
58 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
0
votes
2answers
394 views

Extracting data, sorting and removing duplicates from an array using python 2.7

I have a text file data set in the form of ...
0
votes
1answer
132 views

Choropleth Map with plotly.express doesn't show map, just legend

I copied the "Indexing by GeoJSON Properties" example as seen on the Choropleth Maps in Python page. The code was executed with no error but there was no map showing, only the legend. In ...
1
vote
2answers
255 views

plot multiple strings from a dataframe

I am trying to read and plot several files which looks like as below when I open with python using pandas read.table. The columns to plot are "dev" vs timestamp. The timestamp should be compose from ...

1
2
3 4 5
20