Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

209 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
1k views

How does Seaborn calculate error bars when using estimators other than the arithmetic mean?

If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?
4
votes
1answer
35 views

Spatially constrained geospatial similarity

What's the current methodology for clustering geospatial data by features? Example: I have some demographic dataset. Let's say this contains average home price and population density. So, an example ...
4
votes
1answer
219 views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
3
votes
0answers
36 views

Detecting pushups based on pose data

I've been playing with Google's MLKit, and decided to detect push ups. As a quick test, I took the position of the left shoulder, and plotted the Y Axis. Here's how a variety of trials look: Five ...
3
votes
0answers
129 views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
3
votes
0answers
998 views

How to apply a groupby rolling function to create multiple columns in the dataframe

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo and the link to the data is here and the example here. Some Sample of data ...
3
votes
1answer
426 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
3
votes
1answer
90 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
3
votes
0answers
95 views

Pandas: Calculating the best seller to incorporate number of houses sold

I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller ...
3
votes
0answers
130 views

Create top 10 index fund based on >100 stocks

Data I have a zip with ca M=160 csv files, with one csv file per stock. The ticker symbol is in the file name. Each csv file contain the same columns: date, txVolume(USD), txCount, marketcap(USD), ...
2
votes
1answer
64 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
2
votes
0answers
29 views

Does statsmodels fully support MultiIndex?

The below code snippet shows how statsmodels seems to flatten MultiIndex tuples by joining them with an underscore "_". ...
2
votes
1answer
235 views

Convert Pandas Dataframe with mixed datatypes to LibSVM format

I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
2
votes
1answer
41 views

How to apply .replace() inside a column in a pandas data frame to clean data

I have a pandas data frame where I want to replace all the values that are not numeric in one column for "" The error I'm getting in my code is the following: ...
2
votes
0answers
30 views

Using an unmerged branch from StatsModels for Anaconda

I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive. See this blog, which as some point mentions, almost ...
2
votes
1answer
409 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
2
votes
0answers
42 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
2
votes
1answer
208 views

How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
2
votes
2answers
643 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
2
votes
1answer
73 views

Plotting three lines on the same plot (with four hr frequency)

I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used: ...
2
votes
1answer
406 views

memory problem due to large file

I'm new in Python and definitely, I'm sure I do some mistake. Here is my problem and thanks all to your help in advance. I have 2 files (one is Hive) and the other is csv and merging them. I have ...
2
votes
0answers
101 views

How to predict multiple iterations in the future based on current close prices? Keras Python

I have data from CoinMarketCap for the close prices of Bitcoin. How can I use this data to predict multiple days into the future using past days with a function? I want the function to look like: <...
2
votes
2answers
761 views

Partitioning data into features/labels and train/test after reading from csv file

I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
2
votes
1answer
87 views

Need help on Time Series ARIMA Model

I'm working on forecasting daily volumes and have used time series model to check for data stationarity. However, I'm strugging at forecasting data with 90% accuracy. Right now variation is extremely ...
2
votes
0answers
84 views

How to prevent pd.ewma from changing the index?

I am resampling my time series data using the following code: bars = transactions.price.resample('4H', base = 3).ohlc() This allows me to account for DST ...
2
votes
0answers
1k views

Multidimensional Groupby to ND Array

My objective is to build an N-dimensional array in Python that has the mean of the intersection of a number of features. For example, I have the number of customers in a store at any given time. I ...
2
votes
1answer
75 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
2
votes
0answers
850 views

pandas dataframes memory

I have a question about memory usage. I want to do 4 things: ...
1
vote
0answers
14 views

Making a tagged Part-of-Speech corpus with the help of a lexicon

I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe. Also, I have a list of tokens (words) in another Dataframe. I want to take each ...
1
vote
0answers
34 views

Compare ranking across trials with pandas

I am carrying out several runs of an experiment with different configurations on many different inputs. The experiment has a random element and I'm trying to investigate whether this randomness ...
1
vote
0answers
19 views

PyTorch: Train without dataloader (loop trough dataframe instead)

I was wondering if it is bad practice to instead of using built in tools such as dataloader just loop trough each row in a pandas df. Lets say I am doing text classification and my training loop looks ...
1
vote
0answers
461 views

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python

I have written the following code for a neural network to perform regression on a dataset, but I am getting a ValueError. I have looked up to different answers and ...
1
vote
2answers
31 views

How to combine two time dependent datasets?

I'm very new to data science so please be gentle. I have a dataset that contains record of occurrence of fire for the past 35 years (+-700.000 rows). Each date and time can have more than one ...
1
vote
0answers
15 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
1
vote
1answer
53 views

pandas series match multiple keywords

Is there a direct python pandas method to match values of series and update different series with some string ? I couldn’t find any direct method of doing it. Here the match is to find a value in a ...
1
vote
0answers
37 views

Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
1
vote
0answers
27 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
1
vote
0answers
22 views

pandas memory issues

What are some good "best practices" for dealing with large datasets in pandas and numpy? Typically i read data from csv files and use numpy and pandas to prepare them for machine learning and AI ...
1
vote
1answer
83 views

How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?

I want to apply pd.cut as a transformer in a pipeline, like this: ...
1
vote
1answer
385 views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
1
vote
0answers
31 views

How to clean messy column and reshape data structure in pandas

I have a dataset that looks like this. There are 122 columns of coordinates excluding the date (Time) column and 7000 rows. What I want to do is split the lat and lon into two columns, and then ...
1
vote
1answer
70 views

What's an appropriate datastore for variable length sequence data for PyTorch consumption?

I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats. I need a datastore that can rapidly ...
1
vote
1answer
29 views

Normalize data from different groups

I have data that have been grouped in 27 groups by different criteria. The reason for these groupings is to show that each group has a different behavior. However, I would like to normalize everything ...
1
vote
2answers
35 views

Update the saved model after training

Will saving a trained model this way give me a model trained on every chunk of data or just the last chunk? ...
1
vote
0answers
32 views

How to find the position of a company given other companies in a Pandas DataFrame with Python

So I have a Pandas DataFrame. I am doing some data analysis with Python with some sets of ...
1
vote
1answer
24 views

Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below: ...
1
vote
2answers
73 views

How to deal with errors of defining data types in pandas' read_csv ()?

I have a table with 118,000 rows and 80 columns. I would like to select 8 columns from the table. I am reading the file using the pandas function pd.read_csv ...
1
vote
1answer
32 views

How to summarize multiple time series like dataset

I am running an experiment where I am sampling CPU utilization every second. A snippet of the data from each experiment run is something like this: ...
1
vote
0answers
19 views

Are there decisive leaders in programming with tabular data?

What are the most effective bread-and-butter in-memory open source tabular data frameworks today? I have been working with tabular data for years with an in-house solution that integrates with Excel ...
1
vote
0answers
96 views

Pandas - dataframe not storing images properly

I'm trying to organize image files I have into a Pandas dataframe. I'm using Keras's preprocessing library for working with them. Here's how I load them in: ...

1
2 3 4 5