Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

160 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
33 views

Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
4
votes
0answers
1k views

How does Seaborn calculate error bars when using estimators other than the arithmetic mean?

If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?
3
votes
0answers
221 views

How to apply rolling volume profile function with a pandas TimeDelta

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo. Here the author gets slice the index and apply the main function by ...
3
votes
1answer
83 views

Pandas DataFrame Rollup Error

I am trying to compute Percentage Contributions on a pandas dataframe. I have a data frame which has three columns (Buyer, ...
3
votes
0answers
78 views

Pandas: Calculating the best seller to incorporate number of houses sold

I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller ...
3
votes
0answers
113 views

Create top 10 index fund based on >100 stocks

Data I have a zip with ca M=160 csv files, with one csv file per stock. The ticker symbol is in the file name. Each csv file contain the same columns: date, txVolume(USD), txCount, marketcap(USD), ...
2
votes
0answers
28 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
2
votes
1answer
40 views

Data sets that have strings and numerical data all in one column

I m new to the field of Data Science, but I have a great passion for it. Here's a code I wrote for pre processing a data set. It works ...
2
votes
1answer
261 views

Any one read the over 100gb csv file and successfully concatenation?

I have been searching for the deal with large CSV file read method Its over 100gb and need to know how deal with the chunk file processing and make concatenation faster ...
2
votes
0answers
23 views

Finding IDs that have been consistently failing to meet target

It's my first time here on stack exchange. I have table with data columns including ID, Domain, Target and Dates (The table seen below). I am currently performing analysis on it and having trouble in ...
2
votes
1answer
49 views

How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
2
votes
1answer
3k views

Multiple filtering pandas columns based on values in another column

I have a pandas dataframe df1: Now, I want to filter the rows in df1 based on unique combinations of ...
2
votes
2answers
231 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
2
votes
2answers
101 views

Error plotting with datetime and value using matplotlib in python

Here I have a dataset with three values. I plot these three values in one graph using python. While I plot the graph first plot line is not coming properly. Because start point and end point combined ...
2
votes
1answer
62 views

Plotting three lines on the same plot (with four hr frequency)

I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used: ...
2
votes
1answer
133 views

memory problem due to large file

I'm new in Python and definitely, I'm sure I do some mistake. Here is my problem and thanks all to your help in advance. I have 2 files (one is Hive) and the other is csv and merging them. I have ...
2
votes
0answers
58 views

How to predict multiple iterations in the future based on current close prices? Keras Python

I have data from CoinMarketCap for the close prices of Bitcoin. How can I use this data to predict multiple days into the future using past days with a function? I want the function to look like: <...
2
votes
1answer
447 views

Partitioning data into features/labels and train/test after reading from csv file

I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
2
votes
0answers
76 views

How to prevent pd.ewma from changing the index?

I am resampling my time series data using the following code: bars = transactions.price.resample('4H', base = 3).ohlc() This allows me to account for DST ...
2
votes
3answers
417 views

Handling a combined dataset of numerical and categorical features for Regression

I have a dataset that has a large number of categorical features and a few numerical features, and I want to predict the probability that any given input is one of two types of a certain binary output ...
2
votes
1answer
67 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
2
votes
2answers
268 views

Feature Engineering

I have a dataframe with around 37,000 rows and 54 columns. Out of these 54 columns two columns namely 'user_id' and 'mail_id' are provided in avery creepy format as shown below: ...
1
vote
1answer
13 views

Is there a python code to perform the following task in python

Is there a way by which I can find how many people rated below 60% or how much percentage of people gave a rating of less than ...
1
vote
2answers
23 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
1
vote
0answers
32 views

Method for 'Simple' Clustering?

In a recent adult education class I took the prof shared a spreadsheet that was written in VBA code and takes ages to execute, I'm sure ...
1
vote
1answer
36 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
1
vote
1answer
37 views

How to put the variable names of pandas data frame on Diagonal of Seaborn PairGrid?

The given below code is not working now(earlier it was working) to put the column names on the diagonal of Seaborn PairGrid. ...
1
vote
3answers
333 views

convert dataframe without header to dictionary with a row of number

I am trying to put a dataframe into dictionary ,with the first column as the key ,the numbers in a row would be the value . how should I do it ? I tried to_dict but did not work , my df has no header ...
1
vote
1answer
163 views

String handling by OneHotEncoder

I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features. Moreover, the documentation is what looks more ambiguous. Here are the ...
1
vote
2answers
59 views

Pandas, make the bars of a bar graph have equal widths

What do I need to change so that ax1 uses the same width for bars as ax2? ...
1
vote
1answer
65 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
1
vote
1answer
52 views

ValueError from statsmodels ExponentialSmoothing

I've been having a frustrating issue with the ExponentialSmoothing module from statsmodels. My data is a pandas series with 74 weekly data points that looks like this: ...
1
vote
1answer
42 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
1
vote
0answers
31 views

How to write a code to display output value in every 60 minutes using panda python using csv file

I have temperature in a csv file. My file is updated with time . I have loaded this data to a pandas DataFrame. What I want to do is predict temperature value in next 60 minutes. I changed the start ...
1
vote
0answers
33 views

Email clustering using numpy and pandas

Im trying to cluster my raw email data using numpy and pandas only. Hence Im Trying to create an email dataframe from raw emails that I have. However the issue that I have is that I keep getting an ...
1
vote
0answers
136 views

linear interpolation on python

Given 1000 objects, the data set tracks each object's location ordered by time, such as 3/3/18 9:01, 3/3/18 9:04.......how to define the first observation of each object together with xy-...
1
vote
0answers
286 views

“Memory Error” - Kmeans in python using pandas DataFrame

I am trying to predict on my "dataset_to_predict" having size of (297000 x 5120). While Memory usage is under 50%. No Specific Error message. I'm trying to find # of k using elbow method - Got ...
1
vote
0answers
25 views

How to concatenate many .psv files in google collaboratory?

I have a folder named 'training' in my local drive which has 20000 .psv files. I zipped it and uploaded to google collaboratory, with the upload option in the Files section. I unzipped it with the ...
1
vote
0answers
27 views

Lookup from time series panda dataframe

I have below reference table with date time series defined in 20 mins interval ...
1
vote
1answer
48 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
1
vote
1answer
620 views

Generating a sequence based on value in another column in Python

I have the following data frame: I would like to add a column with value equal to 1 if flag is 0 and incrementally add 1 in the following rows until the next 0 is ...
1
vote
0answers
18 views

Visulazing a specific value of a column with the corresponding values

I have recently started using data studio to make visualizations before diving into analysis and data prep! I have a column named NAME OF STORE that good different names and other columns like Date, ...
1
vote
1answer
2k views

merge 2 dataframe with Memory Error

trying to merge 2 data frame size with (13647309, 48) and I'm using 32 memory. df_train = train.merge(train_lag,on=['ncodpers','int_date'], how='left') after I ...
1
vote
0answers
37 views

How to convert a time series data which can be fed to a Machine Leanring model for unsupervised learning?

I am very new to time series data. I am working with server data and want to classify if the bunch of server times are healthy or not. My data looks like this in pandas ...
1
vote
0answers
58 views

Split datetime64 column in dataframe on year

I have a column in my dataframe containing data in the datetime64 format. so example: 2005-02-01 00:00:00 and 2007:02-02 00:00:00. I want to split this into multiple columns which per year. Does ...
1
vote
1answer
2k views

'int' object is not iterable using panda python

Here I want to calculate time interval in between row by row in time column import from csv file. my start time 6.00 a.m and my end time is next day 6.00 a.m. In between this time periods how to find ...
1
vote
0answers
28 views

Allocation from a central stock in Pandas

I am struggling with creating an allocation algorithm against multiple demand. eg: consider 2 banks operating in multiple cities. For each bank in each city there are available funds that can be ...
1
vote
0answers
470 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition - https://www.kaggle.com/c/ga-customer-revenue-prediction Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back ...
1
vote
0answers
22 views

visualization idea on a panel data (roughly speaking)

I wanted to construct a heat map on the following data: ...
1
vote
2answers
72 views

How to group this dataframe in python?

I have this problem: ...