Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
912
questions
1
vote
1answer
18 views
Not enough memory for operations with Pandas
Wes McKinney, the author of Pandas, writes in his blog that
"... my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB ...
-2
votes
0answers
11 views
how to convert or interpolate monthly netcdf data to daily data [closed]
I have a monthly NetCDF data containing monthly data of potential temperature. It contains 12-time steps i.e. for 12 months. I was wondering if there is any way we can convert this monthly dataset to ...
-2
votes
0answers
8 views
Help me clearing out the error "Dataframe object has no Attribute; i have attached the screenshot [closed]
am doing univariate analysis
am unable to access the column share
i've attached the screenshot
-1
votes
1answer
14 views
VIF error in multiple regression [closed]
While trying VIF in multiple regression I came across this error, can anyone help me out of this?
here is my code:
...
0
votes
1answer
16 views
RAM crashed for XML to DataFrame conversion function
I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) ...
0
votes
0answers
19 views
Predict how many days until New Purchase
I am trying to introduce myself into predicting with dates and I am having big troubles understanding how to do it.
I have a dataset with customer orders from an e-commerce platform with the usual ...
0
votes
1answer
19 views
How to filter in pandas based on string value? [closed]
I want to filter a pandas dataframe based on the id of the row. The id is a sequence of patient names. For example, I have columns in this format:
ID | A | B
And ...
0
votes
0answers
16 views
0
votes
1answer
45 views
Pandas - Avoid boolean result when using groupby()
I have this script:
sectors = df.groupby(['company_sector']).mean()['investment_in_millions']
Output:
I wanted to keep the same groupy() but having a result ...
1
vote
0answers
18 views
Why is the behaviour of groupby command different in R and in python with pandas? [closed]
if I execute the following lines of codes in R as well as python for groupby command, the order of the resulting ...
0
votes
0answers
17 views
Time Series Plot for floating values
I have a Dataframe which looks as shown below
I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as
...
0
votes
1answer
25 views
How to play with .cat.codes and get_dummies
I may be lost here and this might be a stupid question.
I know that get_dummies returns multiple columns depending upon the number of variables and ...
0
votes
0answers
32 views
How to convert a .dat file to .csv format
I have a .dat file and I don't know it's structure. However, I would like to read it using Python and possibly convert it to .csv file.
Note: The file I have is encoded with ASN.1
0
votes
0answers
30 views
How to use a condition in rolling window function in pandas?
I am trying to calculate number of items sold for a category in the past. I have multiple conditions on which I want to do this: For example:
Get the number of items sold in last 10 days,
if none ...
-1
votes
1answer
22 views
Matrix of pairwise cosine similarities from matrix of vectors [closed]
I have a matrix of 200d vectors stored as follows:
$ X = \begin{pmatrix} \text{id}_1 & 0.5 & -2 & \dots & 10
\\ \text{id}_2 & -4 & 6 & \dots & -0.3
...
0
votes
0answers
8 views
Does it make sense to train kalman filter on unscaled (Timeseries-)data, to clean it?
I have Timeseries data to clean and I got the tip to use a kalman Filter.
My Question is, does it make sense to "fit" this kalman-filters parameters to an unscaled and unnormalized (...
0
votes
0answers
10 views
How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?
Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like
...
3
votes
1answer
49 views
Is it possible to have stratified train-test split of a set based on two columns?
Consider a dataframe that contains two columns, text and label. I can very easily create a stratified train-test split using ...
0
votes
1answer
70 views
Pandas/Python - comparing two columns for matches not in the same row
I have this data:
I wanted to compare A and B for matches not by row but rather search A0 if it is in column B and so on. Moreover, I wanted to ignore the .AX in column A because it would not find ...
1
vote
1answer
30 views
Pandas copy() different columns from different dataframes to a new dataframe
I have 2 dataframes that are coming from 2 different Excel files. I want to extract some columns from one file and other columns from the second file to print a new dataframe with the copied columns.
...
0
votes
1answer
23 views
Find and count similar price for each store and output the max count and its corresponding rows
My goal is:
for each item in each store, find the number of items in the same store within $2 price difference (i.e. number of items with similar price within the same store).
Then list all the ...
1
vote
1answer
48 views
How to access the data of the column on which some groupby operation has been carried out? [closed]
Suppose there is a pandas dataframe which has one column consisting of names of something, and there are multiple entries respective to each entry in the first column. To count the number of entries ...
3
votes
2answers
62 views
Is Label Encoding with arbitrary numbers ever useful at all?
From what I read online, there seems to be some confusion regarding the taxonomy and the terms used, so to avoid misunderstanding I'm going to define them here:
Label Encoding - encoding a nominal ...
0
votes
1answer
120 views
ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames
I want to create a new variable for the dataframe details called lower after iterating through multiple ...
1
vote
1answer
46 views
Date and time extraction from Excel file in Python or Pandas
I have an Excel file with a column that contain both date and time in a single cells as follows:
2020-07-10T13:32:01+00:00
I'm wondering how to extract this cell, split date and time (that are ...
1
vote
3answers
67 views
Count the max number of consecutive 1 and 0 in Pandas Dataframe [closed]
Hey I have the following Dataset
import pandas as pd
df = pd.DataFrame({
'column1': [0,0,1,0,1,0,0,1,1,0,1,1,1]})
I want to be able to count the number of ...
1
vote
2answers
552 views
AttributeError: 'DataFrame' object has no attribute 'ix' [closed]
I was learning Classification-based collaboration system and while running the code I faced the error AttributeError: 'DataFrame' object has no attribute 'ix'. Here is the code I write until now.
<...
1
vote
1answer
39 views
How to use df.groupby() to select and sum specific columns w/o pandas trimming total number of columns
I got Column1, Column2, Column3, Column4, Column5, Column6
I'd like to group Column1 and get the row sum of Column3,4 and 5
When I apply groupby() and get this that is correct but it's leaving out ...
0
votes
0answers
10 views
Estimate class proportions of a feature, central limit theorem
haven't been feeling smart lately and this is probably the most trivial question ever but I really need to know. I'm trying to point estimate some population parameters. I sampled from 1000 randomly ...
2
votes
1answer
32 views
pandas groupby and sort values
I am studying for an exam and encountered this problem from past worksheets:
This is the data frame called 'contest' with granularity as each submission of question from each contestant in the math ...
1
vote
1answer
34 views
parsing dictionary to JSON [closed]
I used the following code to pass a json string as a parameter in requests.post method,
...
0
votes
0answers
147 views
What are some standard or preferred battery of tests to perform to test for quality of my data before feeding to a ML algorithm?
we are designing a rules based engine to check the quality of the data before training our ML models. The data we have is time series data. We have about 3-4 features using which we have to make a ...
0
votes
1answer
57 views
conversion of dictionary to json to be sent to requests.post
I have a data frame like this.
df=pd.DataFrame({'sub1':[10,20,30,40],'sub2':[5,10,15,20],'sub3':[1,2,3,4]})
and a dictionary like this.
...
1
vote
0answers
35 views
Pandas how to change values in a column [closed]
I have data on mchip, chip schip and mcaid that contains the following columns/features: year (between 2000 and 2016), state, age and month. I have a problem with several years. For example, I want to ...
3
votes
2answers
73 views
Generate pdf from jupyter notebook without code
I have jupyter notebook which contains markdown, code and outputs (graphs). I want to generate pdf from this notebook.
I tried to hide code using html code which I get from here then I tried to ...
0
votes
1answer
23 views
Exploratory Data Analysis on dataset divided by winners and losers
I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players:
...
2
votes
1answer
22 views
Find the best interpolation method for missing observations
I have a database which has measurements of objects every day every hour.
However, some data is missing and I don't have measurements for all the hours.
in order to get over this challenge I have ...
0
votes
0answers
6 views
How to include the other variables at t=t to predict the target variable with time lags also in LSTM?
I am having a training data set for a time-series dataset like below:
...
1
vote
1answer
35 views
0
votes
1answer
29 views
Unhashable type 'list' while looping through dataframe in Python [closed]
I have the following dataframe comments. I have segregated a list of users based on certain conditions. I want to get the count of words based on those users which ...
2
votes
1answer
49 views
KeyError: Selecting text from a dataframe based on values of another dataframe
I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
1
vote
1answer
44 views
pandas series match multiple keywords
Is there a direct python pandas method to match values of series and update different series with some string ? I couldn’t find any direct method of doing it.
Here the match is to find a value in a ...
2
votes
4answers
36 views
Scikit-learn: train/test split to include have same representation of two different types of values in a column
I have a dataset of online purchase orders that contains two types of customers:
Customers who have an account and thus are known customers with unique customer number.
Customers who do not have an ...
0
votes
1answer
15 views
Creating a new Dataframe with specific row numbers from another
I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
1
vote
0answers
28 views
pandas: reshaping the dataset, moving column name as rows [closed]
I currently have the following table:
...
0
votes
1answer
27 views
How to infer from logarithmic scale scatter plot and line plot in Seaborn
I have generated the following 4 plots, plot 1 and plot 2 are on linear scale and plot 3 and plot 4 are on logarithmic scale. Can someone tell me what inferences can I draw from the plots?
NOTE: All ...
2
votes
0answers
26 views
Does statsmodels fully support MultiIndex?
The below code snippet shows how statsmodels seems to flatten MultiIndex tuples by joining them with an underscore "_".
...
1
vote
1answer
43 views
How to compare number of occurrences in two data frames for a column and extract the similarities
I'm trying to find the same number of occurrences in both data frames
This is a follow-up question for my previous question
I got 2 data frames
...
0
votes
1answer
45 views
Compare similarities between two data frames using more than one column in each data frame
This work started by comparing two columns in each data set in pandas.
Previous research:here
A lot of results online show how to compare 2 data frames with 1 column
I'm trying to learn how to ...
0
votes
2answers
47 views
Discarding non-english words in column
I have some non-english words/sentences in my data.
I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
