Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
974
questions
0
votes
0answers
8 views
Get latest Item by Date for a Recommender System
I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would ...
0
votes
1answer
30 views
How to load pre-processed .txt data and use it for training convolutional networks?
I have a lot of '.txt' (pre-processed) files related to C3D and I need to use this to train and test while using CNN model.
How does one go about this?
Most of data in each .txt files are formatted ...
0
votes
1answer
25 views
Replacing value not in list in Pandas [closed]
Say I have a data frame in Python.
If I want to replace all values in the say, the Size column that are not 'M' or 'S' or 'L' with nan, how do I do so?
Thanks in advance
1
vote
0answers
64 views
Email clustering using numpy and pandas
Im trying to cluster my raw email data using numpy and pandas only.
Hence Im Trying to create an email dataframe from raw emails that I have. However the issue that I have is that I keep getting an ...
0
votes
0answers
361 views
How to fix IndexError: Too many levels: Index has only 1 level, not 2 in Python
I have written the following code that has a function model_data to perform a particular set of tasks. I have to pass the list of ...
2
votes
1answer
77 views
1
vote
1answer
40 views
Why we can't Remove features with missing values in Data Preprocessing [closed]
In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values ...
1
vote
1answer
36 views
How to apply multiple filter in Data Frame? [closed]
How to implement multiple filters for checking data cell in a range ?
Suppose, I have a list of numbers like,
...
1
vote
1answer
17 views
How we can Identify Specific Feature from a larger amount of Dataset?
In Machine Learning, we need to play with any kind of datasets.
In the Dataset, There are too many records and features, Some datasets had lots of features (sometimes it's called ...
1
vote
0answers
10 views
How to find average of Duplicate Column Headers in Python dataframe [closed]
I have data in this format
...
0
votes
1answer
26 views
How to obtain a count of the classes of a categorical var within a certain time interval for a time-stamped data?
I have a dataframe with several categorical variables. But for simplicity let's assume there is only 1 categorical variables with 3 classes. I want to obtain the counts of these classes within a ...
1
vote
1answer
229 views
1
vote
1answer
28 views
Count occurences of rows based one cell content with pandas [closed]
I am currently struggling with the pandas framework.
Let's imagine the following Data Frame:
...
0
votes
1answer
81 views
Calculate all days for each years between two dates in a pandas dataframe
I'm new in python and coding. I'm doing a university project exercise. Last question is :
"
For each year, compute the total amount of loans. Each loan that has planned expiration time and ...
0
votes
1answer
95 views
Machine Learning: Predicting target based on a feature
I have a df looks as follow:
-It is very likely that the same feature1Xfeature2Xfeature3 combination will appear multiple times....
0
votes
1answer
28 views
What is the best way to visualize range of data?
If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this:
...
0
votes
1answer
35 views
Replace this for loop with a pandas vector operation [closed]
I am running this for loop in python to create a list of differences of means of two dataframes, and then using that list to generate a random values under a normal distribution.
...
0
votes
1answer
31 views
How to handle Tokenized text content which is given in number?
i have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed?
As I am encountering such data set the first time, so ...
0
votes
1answer
908 views
ValueError: Found array with 0 sample(s) (shape=(0, 13)) while a minimum of 1 is required [closed]
I am new to this logistic regression and finding this error. Please help.
...
-1
votes
2answers
941 views
Comparing a column value to list of values and if it contains the value then assigning that list value to new Column
Column
and wanted to check with this list
...
3
votes
4answers
5k views
Is there a package for using SQL to manipulate Pandas dataframes in Python?
Rather than learn a new package/language, I'd like to use my existing SQL skills to manipulate pandas dataframes in Python. Does anyone know of a way to do this, or ...
1
vote
1answer
39 views
What's the of all values above some percentile called? How do I get it in pandas?
Let's say I have a list of numbers, and I want the mean of all numbers that are greater than the 95th percentile. Is there some standard term for that value? ("Mean of a histogram bin"? &...
0
votes
0answers
21 views
Autocorrelation plot meaning!
I use pandas autocorrelation plot for a time series, and the result is:
How cany I inference from it?! This is my code :
...
0
votes
1answer
184 views
Linear Regression not working due to wrong kind of array
I try to deal with my homework. The Job is to take this Data and perform a linear regression on it.
The code is published here.
I am quite new to programming in Python and in data science. So I tried ...
5
votes
1answer
74 views
Text extraction from large pool of documents of different formats
I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, ...
2
votes
1answer
361 views
ValueError from statsmodels ExponentialSmoothing
I've been having a frustrating issue with the ExponentialSmoothing module from statsmodels.
My data is a pandas series with 74 weekly data points that looks like this:
...
0
votes
1answer
59 views
How to add a filter/query/condition in Plotly [closed]
I want to add a condition to the column where attractionName.value_counts() >=165. How can I add a query/filter/condition to a DataFrame before I plot the data.
<...
1
vote
1answer
67 views
Not enough memory for operations with Pandas
Wes McKinney, the author of Pandas, writes in his blog that
"... my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB ...
1
vote
0answers
750 views
Issues with pandas chunk merge
I'm trying to solve a kaggle competition
Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back to disk. This is the code I'm using:
...
1
vote
1answer
7k views
Numpy array from pandas dataframe
I am new in using python for data science.
What is the difference between selecting a a column with: df['name'].values and ...
1
vote
0answers
40 views
Chi Square Test Goodness of Fit
I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
2
votes
1answer
2k views
Python pandas index error
Encountred an index error when trying to run the code found here.
The error occurs in section In[10]:
...
2
votes
2answers
40 views
Help with replacing values in a data frame
I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine ...
2
votes
1answer
52 views
Converting a Pandas Series string of multiple attributes into individual attributes?
I'm working on an Airbnb dataset in order to predict the nightly price--where each example is one Airbnb listing (ie a listing for someone's house, room, apartment, etc). Each feature is a ...
1
vote
2answers
257 views
Should I apply PCA on the entire dataset or just the nominal values?
I have a data-set with 14~ attributes, roughly half of them nominal. I've used a binary vectorizer to convert these values to a number of attributes. The number of attributes, naturally, ballooned up; ...
1
vote
1answer
79 views
parsing dictionary to JSON [closed]
I used the following code to pass a json string as a parameter in requests.post method,
...
5
votes
2answers
7k views
What does pandas describe() percentiles values tell about our data?
Let say this is my dataframe
...
3
votes
1answer
149 views
2
votes
3answers
12k views
Pandas - read CSV with spanish characters
I have DataFrame which has column with spanish regions, when I import the .csv as following
df = pd.read_csv('...',delimiter=';',decimal=',')
And print the ...
3
votes
2answers
5k views
Pandas Conditional Fill NaN Forward/Backward
Updated 22 Oct. 2018:
I have the following dataset:
...
1
vote
1answer
2k views
How to Replace an object in Pandas array using replace with dictionary from excel file?
Suppose I have a data like this:
and I want to change the value of name and gender into an integer, and I have a dictionary like this:
I understand that I can use replace function in pandas, but I ...
1
vote
1answer
3k views
Segmenting pandas dataframe with lists as elements
I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. ...
5
votes
2answers
64 views
Which plot to use for data spanned on multiple years?
I've a dataset of - Number of accidents that happens in each state. The dataset spans from 2001 to 2014.
Which plot should I use to represent the information - number of accidents in each state per ...
4
votes
1answer
6k views
Stacked time series plot in python
In pandas I can set the date as index, and then run df.plot() to see a line chart. How do I make that line chart stacked as in the picture below?
3
votes
1answer
797 views
SVR is giving same prediction for all features
I'm creating a basic application to predict the 'Closing' value of a stock for day n+1, given features of stock n using Python and Scikit-learn
A sample row in my dataframe looks like this (2000 rows)
...
3
votes
1answer
978 views
Python: Fast indexing of strings in nested list without loop
I have an array dataset from which I need to build D whose each element is a list of strings (Third column of dataset). The dataset looks like:
...
0
votes
2answers
2k views
Remove Outliers from Dataframe using pandas in Python
I would like to remove outliers from my dataset. It looks like this:
...
0
votes
1answer
23 views
How to filter in pandas based on string value? [closed]
I want to filter a pandas dataframe based on the id of the row. The id is a sequence of patient names. For example, I have columns in this format:
ID | A | B
And ...
0
votes
1answer
91 views
Pandas - Avoid boolean result when using groupby()
I have this script:
sectors = df.groupby(['company_sector']).mean()['investment_in_millions']
Output:
I wanted to keep the same groupy() but having a result ...
0
votes
1answer
196 views
Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)
I am using a Dice Coefficient based function to calculate the similarity of two strings:
...
