People who code: we want your input. Take the Survey

Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
1
vote
1answer
110 views

Set value for column based on two other columns in pandas dataframe

I have a dataframe that has contracts with different order dates and I need to create a new column that assign a number to each contract if it has more than one order date. For example my sample ...
1
vote
1answer
25 views

copy data between two dataframes in python

I have two different dataframes, both containing a common element (identifier). I want to copy data from df1 for each identifier in df2 (each identifier can repeat maximum 5 times in df1) df1 ---data ...
2
votes
1answer
78 views

Plotting three lines on the same plot (with four hr frequency)

I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used: ...
0
votes
1answer
44 views

Look up a number inside a list within a pandas cell, and return corresponding string value from a second DF

(I've edited the first column name in the labels_df for clarity) I have two DataFrames, train_df and labels_df. train_df has integers that map to attribute names in the labels_df. I would like to ...
0
votes
0answers
16 views

Pre-processing data - Dataframe manipulation Time Series

I have a question in which I'm not entirely sure in which path to take. I'd appreciate if you could point me in the right direction. Below a screenshot of the a few records of my dataset. As you can ...
1
vote
2answers
41 views

Update the saved model after training

Will saving a trained model this way give me a model trained on every chunk of data or just the last chunk? ...
-1
votes
1answer
1k views

How to group data and plot line graphs

This is the first time I am using pandas and iPython notebook and was not able to figure out the correct search terms for my problem. I have a .xls file for ...
0
votes
1answer
353 views

How to filter rows in Python pandas dataframe with duplicate values in the columns to be filtere

Overall context: I have a data frame that contains observations for every five minute starting at 5 AM in the morning and ending at 8 PM in the evening for several days. I need to filter all the ...
4
votes
1answer
37 views

Spatially constrained geospatial similarity

What's the current methodology for clustering geospatial data by features? Example: I have some demographic dataset. Let's say this contains average home price and population density. So, an example ...
0
votes
0answers
826 views

Excel error may be caused by pandas writing or big data? advise needed

I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do: open an xml file extract some elements create a pandas dataframe ...
1
vote
1answer
463 views

Plot Two Categorical Variables

How to plot two categorical variables in Python or using any library? I want to plot the Playing Role of a Cricketer (Batsman, Bowler, etc.) VS Bought_By (Franchise Names, e.g., CSK, DC, etc.). The ...
1
vote
1answer
686 views

how to implement Excel vlookup on Python

I have a dataset A and dataset B, both data has a column in common which is the ID, i want to check if dataset A ID values can be found in dataset B ID values. Also get the records of dataset B that ...
0
votes
0answers
21 views

What are the use-cases for heapsort option in numpy.sort()?

Could anybody share experience about use cases of the heapsort option in numpy.sort() (or pandas.DataFrame.sort_values()) in the data science field? I mean if the <...
1
vote
1answer
27 views

txt file into pandas

I have a txt file where values are randomly separated, sometimes there is a single space, and sometimes there are many. I am importing the file into pandas with <...
0
votes
1answer
73 views

IterativeImputer Evaluation

I am having a hard time evaluating my model of imputation. I used an iterative imputer model to fill in the missing values in all four columns. For the model on the iterative imputer, I am using a ...
1
vote
2answers
2k views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
0
votes
1answer
196 views

How to combine rows after Pandas Groupby function

May I know how to combine several rows into one single row after I used Pandas groupby function? In below example, I would like to to group the data by Employee ID, Customer Last Name and Customer ...
-1
votes
1answer
32 views

Multiplying a dataframe by a larger one [closed]

I have two dataframes df1 and df2 with the same columns but not the same row number. I want to multiply them element-wise such ...
4
votes
1answer
805 views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
0
votes
1answer
560 views

How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names. what should ...
1
vote
3answers
4k views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
2
votes
1answer
330 views

Breaking a DataFrame based on Specific Column Values [closed]

Assume I have a following dataframe which has years of experience and salary. I would like to create a new dataframe from the existing one, only including observations where salary is above 45000. I ...
-2
votes
1answer
188 views

ImportError: Pandas requires version '0.3.0' or newer of 's3fs' [closed]

I'm trying to read files from S3, using boto3, pandas, anaconda, but I have the following error: ...
0
votes
0answers
19 views

Accessing group elements in groupby

I had a doubt about groupby operations. Lets suppose I have grouped my data based on one column and got 5 groups as output. I know that we can iterate over these groups and apply functions to them as ...
1
vote
1answer
51 views

Perform PCA on columns of different length

I have about 20-30 columns, all with different lengths. The first column has 25000 rows, the second column has 19000 rows, and it's different for all the columns. All are the survey data with 0 (No),1(...
0
votes
1answer
92 views

Plotting clustered sentences in Python

I have the following three sentences, extracted from a dataframe. I would like to check the similarity and create clusters based on their level of similarity. ...
0
votes
0answers
24 views
1
vote
2answers
32 views

Sort Pandas Dataframe per column [closed]

I have a dataset with age + another 14 variables. I have created 13 bins representing different age groups like so: ...
1
vote
2answers
48 views
0
votes
3answers
21k views

How to calculate time difference in between rows using loop in panda python

I have a CSV file with columns date, time. I want to calculate row-by-row the time difference ...
0
votes
1answer
200 views

Time series plot using netcdf data in python

I am trying to plot a time series for a 3d plot. I know that the variable must be made 1d inorder to make it viable for the time series plot. Actually I want the data within a box of latitude and ...
1
vote
1answer
23 views

How to Iterate over rows in a dataframe [closed]

So I've been trying to iterate over rows of my dataframe and my goal is to find matching rows based on two particular columns (say C and P). Then i need to do some manipulations as well in the data ...
-1
votes
2answers
162 views

How to find the average over an area of center for a given radius

folks, I have an excel file which contains the lat and long values of the center of a Tropical Cyclone(TC). The excel file is as given below: Also, I have an NC(NetCDF) file which is of that of ...
0
votes
1answer
10 views

Dealing with dates in dataframe

I have a dataframe which has two date columns - Start Date and End Date, Let's suppose start date is 4/10/2019 and end date is 4/10/2020 (MM/DD/YYYY) now I want to split these dates in a list which ...
0
votes
2answers
158 views

Pandas - Create a Integer Primary Key based in string columns - python [closed]

I need to create a primary key based in string columns in my dataframe ...
0
votes
1answer
3k views

Conditionally replace dataframe cells with value from another cell

I have a couple pandas dataframe questions. I would like to replace the values in only certain cells (based on a boolean condition) with a value identified from another cell. I have defined the ...
0
votes
1answer
7k views

Pandas to_sql insert only some columns

I am new using pandas. I am asking about how to insert specific columns using to_sql. This is my explanation. I have a database with a table datasiswa with columns: id: as int, with autoincrement,...
2
votes
1answer
647 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
0
votes
2answers
209 views

Pandas get_dummies() rows dropping after joining back with X

I'm having an issue that I can't explain and am hoping I am missing something simple. I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning,...
1
vote
1answer
127 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
22
votes
3answers
15k views

Is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
28
votes
4answers
128k views

How to fill missing value based on other columns in Pandas dataframe?

Suppose I have a 5*3 data frame in which third column contains missing value 1 2 3 4 5 NaN 7 8 9 3 2 NaN 5 6 NaN I hope to generate value for missing value based ...
1
vote
1answer
112 views

What's an appropriate datastore for variable length sequence data for PyTorch consumption?

I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats. I need a datastore that can rapidly ...
0
votes
1answer
21 views

Why does storing the output of .count() result in all 1's or 0's? [closed]

I have a dataframe which I run loc on to find all values within the loc parameters. Whatever the loc returns gets stored in a variable. I try to use this variable to store it in another dataframe ...
-1
votes
1answer
26 views

Why my frame index is full of 0 when I make it to copy another frame?

I've got such a code: ...
-1
votes
1answer
34 views

Unable to successfully merge dataframes in pandas along labels

I have two different dataframes, they both share the same labels, "Country" and "Year", I am trying to merge these together as one by these two columns. This is my code: ...
1
vote
1answer
524 views

Replace values in dataframe with another dataframes values based on condition [closed]

I have 2 dataframes with the same columns names and rows number but the elements' values are different. I need to change the value of each element in the first dataframe to 1 if its value in the the ...
0
votes
0answers
711 views

Jupyter notebook kernel dies when reading CSV file on M1 Mac

Odd Jupyter notebook problem on M1 mac here. Using these steps I was able to get Jupyter Notebooks to run in the first place. It worked fine executing the following simple code: ...
0
votes
1answer
36 views

Why is np.where not returning '1'? Only returns '0'

This code should return a new column called orc_4 with the value of 1 if the value of the row in ...
0
votes
1answer
31 views

Access keys of pandas dataframe when using groupby

I have the following database: And I would like to know how many times a combination of BirthDate and Zipcode is repeated throughout the data table: Now, my question is: How can I access the keys of ...

1 2 3
4
5
21