People who code: we want your input. Take the Survey

Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
3
votes
4answers
60k views

Using pandas, check a column for matching text and update new column if TRUE

My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE. From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, ...
1
vote
2answers
18 views

How do I assign specific values to categorical variables

I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...
0
votes
3answers
40 views

converting array to a true/false matrices

I have a data set where each record is a json document with a label, and an array of signals. The signals will vary for each record: ...
0
votes
1answer
47 views

Error is occurring setting an array element with a sequence

I have the columns in my Data Frame as shown below: ...
12
votes
3answers
14k views

Using TF-IDF with other features in scikit-learn

What is the best/correct way to combine text analysis with other features? For example, I have a dataset with some text but also other features/categories. scikit-learn's TF-IDF vectorizer transforms ...
2
votes
1answer
425 views

How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
0
votes
1answer
19 views

Extracting structured data from semi structured data

I want to use machine learning and NLP to convert semi-structured data in text files to structured data by predicting the patterns in the files and splitting the fields for example if I have a text ...
0
votes
1answer
36 views

How to create multi-hot encoding from a list column in dataframe?

I have dataframe like this Label IDs 0 [10, 1] 1 [15] 0 [14] I want to create a multihot encoding of the feature ...
0
votes
1answer
64 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
1
vote
1answer
284 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
0
votes
0answers
32 views

dataframe merge on string not working

having two dataframes each having common column which is country_code. while merging it by 'inner' join, it's returns empty dataframe. but i can see there is many values are common like IND, USA.... <...
0
votes
1answer
20 views

Replace column in a dataframe with another column based on index

I have the following dataframe A: ...
1
vote
0answers
42 views

Pandas: saving timedelta to Parquet

Looks like Pandas doesn't translate Pandas timedelta to Parquet INTERVAL: ...
1
vote
1answer
2k views

Generating a sequence based on value in another column in Python

I have the following data frame: I would like to add a column with value equal to 1 if flag is 0 and incrementally add 1 in the following rows until the next 0 is ...
1
vote
1answer
17 views

How to calculate the distance between two locations using Haversine Formula? [closed]

I have the columns of Latitude and Longitude of city like shown below : ...
0
votes
1answer
704 views

Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies

I'm still in the process of learning, so I'm sorry if this doesn't make much sense. I'm doing Kaggle learns micro courses, and to work with missing tabular data we learned about using pipelines with ...
1
vote
1answer
19 views

Pandas: Group by Single Column Entries [closed]

So have this table above. I'm trying to aggregate the occupations such that the table results in: I've tried using df.groupby(['Occupation']) but I get an error. ...
1
vote
1answer
199 views

Combine results from multiple models

I am using chunks of 100000 rows at a time from the CSV file to train the a simple LASSO model. How do i combine all of these models trained from these different chunks? I would like to use all ...
1
vote
1answer
78 views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
0
votes
1answer
5k views

ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

I want to create a new variable for the dataframe details called lower after iterating through multiple ...
1
vote
1answer
124 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
0
votes
1answer
191 views

Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data

I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with ...
0
votes
1answer
25 views

How can I group by elements in a column in pandas? [closed]

I have a dataset that looks like this: I need to transform it so it looks like this: Meaning I need to show the balance by balance groups and time gaps. Now the time gaps are elements of a column. I ...
0
votes
1answer
41 views

Shaping columns (and column headers) into multi-index rows using pivot/pivot_table

Problem I am trying to shape a dataset which includes stacked dates/data as rows and half-hourly data as columns (Trading periods, eg. TP1, TP2), into a chronological half-hourly order with the ...
2
votes
1answer
587 views

Convert Pandas Dataframe with mixed datatypes to LibSVM format

I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
39
votes
6answers
266k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
-1
votes
1answer
36 views
6
votes
2answers
9k views

What does pandas describe() percentiles values tell about our data?

Let say this is my dataframe ...
0
votes
1answer
3k views

how to export the tables into a csv file pandas

The following is a piece of code I wrote to create a pivot table for categorical vs continuous variable. ...
1
vote
0answers
19 views

How to resolve IndexError while doing Monte Carlo for 1000 runs? [closed]

Below code runs without any problem, however when I run the same code using Monte Carlo Analysis for 1000 runs, it gives IndexError. Can someone explain why this happens. Thanks ...
0
votes
2answers
5k views

Removing duplicates and keeping the last entry in pandas

I have a table and I'm trying to remove all the duplicate and keep the the rows that has the latest datestamp. Here is the table: ...
0
votes
1answer
21 views

Still have the same error after reshaping

I set my column 1 to 13 as X and column 14 as my Y. My data set has 14 columns and 401 rows including the parameters ...
1
vote
1answer
46 views

Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below: ...
2
votes
0answers
36 views

Storing multiple time series in a python pandas dataframe efficiently

I got different time series. Each time series contains different attributes, which have data entries at different points of time (timestamps). Each time series is labeled and has a label attribute. I ...
0
votes
2answers
11k views

How do I read a dat file, for which I don't know its structure?

Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it. How do I figure out the contents in my dat file. ...
1
vote
1answer
22 views

My small script on value alteration in columns of a data not working

I have a data set which has "Speed" as one of the columns (features). The column contains both zero and non-zero values. I want to randomly set 10% of the non-zero values to zeros. This will change ...
0
votes
1answer
25 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
0
votes
1answer
42 views

How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
0
votes
3answers
3k views

Remove columns with a certain number of consecutive zeros

I'm using pandas and I am dealing with time series of sales. What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...
0
votes
1answer
696 views
1
vote
1answer
87 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
2
votes
1answer
123 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
1
vote
1answer
64 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
1
vote
2answers
385 views

plot multiple strings from a dataframe

I am trying to read and plot several files which looks like as below when I open with python using pandas read.table. The columns to plot are "dev" vs timestamp. The timestamp should be compose from ...
1
vote
1answer
71 views

pandas series match multiple keywords

Is there a direct python pandas method to match values of series and update different series with some string ? I couldn’t find any direct method of doing it. Here the match is to find a value in a ...
1
vote
1answer
15 views

how to get the number of columns that have the same value between 2 rows [closed]

I want to compare two records and get the number of columns that have the same value , for example I have two rows, the first row is : rank rank1 rank2 1 1 3 and the second row is rank rank1 rank2 ...
0
votes
1answer
1k views

Comparing dataframe object with string value in django

I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL ...
1
vote
1answer
17 views
0
votes
1answer
15 views

Organizing a csv file of multiple datasets into a list of Pandas dataframes

I have a csv file, containing results from a Computational Fluid Dynamics (CFD) simulation (a sample of my csv file is attached as a google drive link; file size: 1,392KB). In particular, the csv file ...
0
votes
1answer
44 views

What is the best solution to replace NaN values?

I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that ...

1
2
3 4 5
21