Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Get latest Item by Date for a Recommender System

I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would ...
0
votes
1answer
30 views

How to load pre-processed .txt data and use it for training convolutional networks?

I have a lot of '.txt' (pre-processed) files related to C3D and I need to use this to train and test while using CNN model. How does one go about this? Most of data in each .txt files are formatted ...
0
votes
1answer
25 views

Replacing value not in list in Pandas [closed]

Say I have a data frame in Python. If I want to replace all values in the say, the Size column that are not 'M' or 'S' or 'L' with nan, how do I do so? Thanks in advance
1
vote
0answers
64 views

Email clustering using numpy and pandas

Im trying to cluster my raw email data using numpy and pandas only. Hence Im Trying to create an email dataframe from raw emails that I have. However the issue that I have is that I keep getting an ...
0
votes
0answers
361 views

How to fix IndexError: Too many levels: Index has only 1 level, not 2 in Python

I have written the following code that has a function model_data to perform a particular set of tasks. I have to pass the list of ...
1
vote
1answer
40 views

Why we can't Remove features with missing values in Data Preprocessing [closed]

In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values ...
1
vote
1answer
36 views

How to apply multiple filter in Data Frame? [closed]

How to implement multiple filters for checking data cell in a range ? Suppose, I have a list of numbers like, ...
1
vote
1answer
17 views

How we can Identify Specific Feature from a larger amount of Dataset?

In Machine Learning, we need to play with any kind of datasets. In the Dataset, There are too many records and features, Some datasets had lots of features (sometimes it's called ...
1
vote
0answers
10 views

How to find average of Duplicate Column Headers in Python dataframe [closed]

I have data in this format ...
0
votes
1answer
26 views

How to obtain a count of the classes of a categorical var within a certain time interval for a time-stamped data?

I have a dataframe with several categorical variables. But for simplicity let's assume there is only 1 categorical variables with 3 classes. I want to obtain the counts of these classes within a ...
1
vote
1answer
229 views
1
vote
1answer
28 views

Count occurences of rows based one cell content with pandas [closed]

I am currently struggling with the pandas framework. Let's imagine the following Data Frame: ...
0
votes
1answer
81 views

Calculate all days for each years between two dates in a pandas dataframe

I'm new in python and coding. I'm doing a university project exercise. Last question is : " For each year, compute the total amount of loans. Each loan that has planned expiration time and ...
0
votes
1answer
95 views

Machine Learning: Predicting target based on a feature

I have a df looks as follow: -It is very likely that the same feature1Xfeature2Xfeature3 combination will appear multiple times....
0
votes
1answer
28 views

What is the best way to visualize range of data?

If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this: ...
0
votes
1answer
35 views

Replace this for loop with a pandas vector operation [closed]

I am running this for loop in python to create a list of differences of means of two dataframes, and then using that list to generate a random values under a normal distribution. ...
0
votes
1answer
31 views

How to handle Tokenized text content which is given in number?

i have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed? As I am encountering such data set the first time, so ...
0
votes
1answer
908 views

ValueError: Found array with 0 sample(s) (shape=(0, 13)) while a minimum of 1 is required [closed]

I am new to this logistic regression and finding this error. Please help. ...
3
votes
4answers
5k views

Is there a package for using SQL to manipulate Pandas dataframes in Python?

Rather than learn a new package/language, I'd like to use my existing SQL skills to manipulate pandas dataframes in Python. Does anyone know of a way to do this, or ...
1
vote
1answer
39 views

What's the of all values above some percentile called? How do I get it in pandas?

Let's say I have a list of numbers, and I want the mean of all numbers that are greater than the 95th percentile. Is there some standard term for that value? ("Mean of a histogram bin"? &...
0
votes
0answers
21 views

Autocorrelation plot meaning!

I use pandas autocorrelation plot for a time series, and the result is: How cany I inference from it?! This is my code : ...
0
votes
1answer
184 views

Linear Regression not working due to wrong kind of array

I try to deal with my homework. The Job is to take this Data and perform a linear regression on it. The code is published here. I am quite new to programming in Python and in data science. So I tried ...
5
votes
1answer
74 views

Text extraction from large pool of documents of different formats

I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, ...
2
votes
1answer
361 views

ValueError from statsmodels ExponentialSmoothing

I've been having a frustrating issue with the ExponentialSmoothing module from statsmodels. My data is a pandas series with 74 weekly data points that looks like this: ...
0
votes
1answer
59 views

How to add a filter/query/condition in Plotly [closed]

I want to add a condition to the column where attractionName.value_counts() >=165. How can I add a query/filter/condition to a DataFrame before I plot the data. <...
1
vote
1answer
67 views

Not enough memory for operations with Pandas

Wes McKinney, the author of Pandas, writes in his blog that "... my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB ...
1
vote
0answers
750 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back to disk. This is the code I'm using: ...
1
vote
1answer
7k views

Numpy array from pandas dataframe

I am new in using python for data science. What is the difference between selecting a a column with: df['name'].values and ...
1
vote
0answers
40 views

Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
2
votes
1answer
2k views

Python pandas index error

Encountred an index error when trying to run the code found here. The error occurs in section In[10]: ...
2
votes
2answers
40 views

Help with replacing values in a data frame

I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine ...
2
votes
1answer
52 views

Converting a Pandas Series string of multiple attributes into individual attributes?

I'm working on an Airbnb dataset in order to predict the nightly price--where each example is one Airbnb listing (ie a listing for someone's house, room, apartment, etc). Each feature is a ...
1
vote
2answers
257 views

Should I apply PCA on the entire dataset or just the nominal values?

I have a data-set with 14~ attributes, roughly half of them nominal. I've used a binary vectorizer to convert these values to a number of attributes. The number of attributes, naturally, ballooned up; ...
1
vote
1answer
79 views

parsing dictionary to JSON [closed]

I used the following code to pass a json string as a parameter in requests.post method, ...
5
votes
2answers
7k views

What does pandas describe() percentiles values tell about our data?

Let say this is my dataframe ...
3
votes
1answer
149 views

Pandas dataframe with multiple hierarchical indices

I have a data frame which looks like this ...
2
votes
3answers
12k views

Pandas - read CSV with spanish characters

I have DataFrame which has column with spanish regions, when I import the .csv as following df = pd.read_csv('...',delimiter=';',decimal=',') And print the ...
3
votes
2answers
5k views

Pandas Conditional Fill NaN Forward/Backward

Updated 22 Oct. 2018: I have the following dataset: ...
1
vote
1answer
2k views

How to Replace an object in Pandas array using replace with dictionary from excel file?

Suppose I have a data like this: and I want to change the value of name and gender into an integer, and I have a dictionary like this: I understand that I can use replace function in pandas, but I ...
1
vote
1answer
3k views

Segmenting pandas dataframe with lists as elements

I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. ...
5
votes
2answers
64 views

Which plot to use for data spanned on multiple years?

I've a dataset of - Number of accidents that happens in each state. The dataset spans from 2001 to 2014. Which plot should I use to represent the information - number of accidents in each state per ...
4
votes
1answer
6k views

Stacked time series plot in python

In pandas I can set the date as index, and then run df.plot() to see a line chart. How do I make that line chart stacked as in the picture below?
3
votes
1answer
797 views

SVR is giving same prediction for all features

I'm creating a basic application to predict the 'Closing' value of a stock for day n+1, given features of stock n using Python and Scikit-learn A sample row in my dataframe looks like this (2000 rows) ...
3
votes
1answer
978 views

Python: Fast indexing of strings in nested list without loop

I have an array dataset from which I need to build D whose each element is a list of strings (Third column of dataset). The dataset looks like: ...
0
votes
2answers
2k views

Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
0
votes
1answer
23 views

How to filter in pandas based on string value? [closed]

I want to filter a pandas dataframe based on the id of the row. The id is a sequence of patient names. For example, I have columns in this format: ID | A | B And ...
0
votes
1answer
91 views

Pandas - Avoid boolean result when using groupby()

I have this script: sectors = df.groupby(['company_sector']).mean()['investment_in_millions'] Output: I wanted to keep the same groupy() but having a result ...
0
votes
1answer
196 views

Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)

I am using a Dice Coefficient based function to calculate the similarity of two strings: ...

1
3 4
5
6 7
20