Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

219 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
38 views

Spatially constrained geospatial similarity

What's the current methodology for clustering geospatial data by features? Example: I have some demographic dataset. Let's say this contains average home price and population density. So, an example ...
4
votes
1answer
825 views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
3
votes
0answers
48 views

Detecting pushups based on pose data

I've been playing with Google's MLKit, and decided to detect push ups. As a quick test, I took the position of the left shoulder, and plotted the Y Axis. Here's how a variety of trials look: Five ...
3
votes
3answers
11k views

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python

I have written the following code for a neural network to perform regression on a dataset, but I am getting a ValueError. I have looked up to different answers and ...
3
votes
0answers
854 views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
3
votes
0answers
2k views

How to apply a groupby rolling function to create multiple columns in the dataframe

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo and the link to the data is here and the example here. Some Sample of data ...
3
votes
1answer
673 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
3
votes
1answer
848 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
3
votes
1answer
147 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
3
votes
1answer
90 views

Need help on Time Series ARIMA Model

I'm working on forecasting daily volumes and have used time series model to check for data stationarity. However, I'm strugging at forecasting data with 90% accuracy. Right now variation is extremely ...
3
votes
0answers
114 views

Pandas: Calculating the best seller to incorporate number of houses sold

I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller ...
3
votes
0answers
142 views

Create top 10 index fund based on >100 stocks

Data I have a zip with ca M=160 csv files, with one csv file per stock. The ticker symbol is in the file name. Each csv file contain the same columns: date, txVolume(USD), txCount, marketcap(USD), ...
2
votes
0answers
40 views

Storing multiple time series in a python pandas dataframe efficiently

I got different time series. Each time series contains different attributes, which have data entries at different points of time (timestamps). Each time series is labeled and has a label attribute. I ...
2
votes
1answer
123 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
2
votes
0answers
54 views

Does statsmodels fully support MultiIndex?

The below code snippet shows how statsmodels seems to flatten MultiIndex tuples by joining them with an underscore "_". ...
2
votes
1answer
613 views

Convert Pandas Dataframe with mixed datatypes to LibSVM format

I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
2
votes
0answers
50 views

Using an unmerged branch from StatsModels for Anaconda

I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive. See this blog, which as some point mentions, almost ...
2
votes
1answer
15k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
2
votes
3answers
286 views

Is it a best practice to exclude retweets from the data set?

I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is ...
2
votes
0answers
43 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
2
votes
1answer
438 views

How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
2
votes
2answers
890 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
2
votes
1answer
79 views

Plotting three lines on the same plot (with four hr frequency)

I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used: ...
2
votes
0answers
131 views

How to predict multiple iterations in the future based on current close prices? Keras Python

I have data from CoinMarketCap for the close prices of Bitcoin. How can I use this data to predict multiple days into the future using past days with a function? I want the function to look like: <...
2
votes
2answers
1k views

Partitioning data into features/labels and train/test after reading from csv file

I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
2
votes
0answers
88 views

How to prevent pd.ewma from changing the index?

I am resampling my time series data using the following code: bars = transactions.price.resample('4H', base = 3).ohlc() This allows me to account for DST ...
2
votes
0answers
1k views

Multidimensional Groupby to ND Array

My objective is to build an N-dimensional array in Python that has the mean of the intersection of a number of features. For example, I have the number of customers in a store at any given time. I ...
2
votes
1answer
77 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
1
vote
0answers
12 views

creating dataset/storing vectors from c++ to python

I am new to the data science field. I need to create dataset which consists of n inputs (vectors of size 5) and n outputs (vectors of size > 10,000). These vectors are generated using c++ code. ...
1
vote
1answer
102 views

Big array and MemoryError: Unable to allocate memory (in Python)

I am trying to create a predictive model using linear regression with a dataset that has 157673 entries. The data (in a csv file) is in such format: ...
1
vote
2answers
18 views

How do I assign specific values to categorical variables

I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...
1
vote
0answers
53 views

Pandas: saving timedelta to Parquet

Looks like Pandas doesn't translate Pandas timedelta to Parquet INTERVAL: ...
1
vote
1answer
111 views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
1
vote
0answers
18 views

How to generate data for future price predictions with Linear regression and Python

I have written the following Python code that makes predictions using linear regression model. However, I'll appreciate you pointing out what went wrong with how I generated data['X_forecast']. I'm ...
1
vote
1answer
139 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
1
vote
0answers
37 views

Making a tagged Part-of-Speech corpus with the help of a lexicon

I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe. Also, I have a list of tokens (words) in another Dataframe. I want to take each ...
1
vote
0answers
32 views

Filtering redundant Association rules

After performing association rule mining, (using mlxtend library), I have gotten a pandas dataframe containing all the association rules. This contains too many rules, therefore we must reduce the ...
1
vote
0answers
37 views

Compare ranking across trials with pandas

I am carrying out several runs of an experiment with different configurations on many different inputs. The experiment has a random element and I'm trying to investigate whether this randomness ...
1
vote
0answers
177 views

PyTorch: Train without dataloader (loop trough dataframe instead)

I was wondering if it is bad practice to instead of using built in tools such as dataloader just loop trough each row in a pandas df. Lets say I am doing text classification and my training loop looks ...
1
vote
3answers
4k views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
1
vote
2answers
43 views

How to combine two time dependent datasets?

I'm very new to data science so please be gentle. I have a dataset that contains record of occurrence of fire for the past 35 years (+-700.000 rows). Each date and time can have more than one ...
1
vote
0answers
32 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
1
vote
1answer
28 views

Find and count similar price for each store and output the max count and its corresponding rows

My goal is: for each item in each store, find the number of items in the same store within $2 price difference (i.e. number of items with similar price within the same store). Then list all the ...
1
vote
1answer
72 views

pandas series match multiple keywords

Is there a direct python pandas method to match values of series and update different series with some string ? I couldn’t find any direct method of doing it. Here the match is to find a value in a ...
1
vote
1answer
2k views

Error with pandas dataframe (needs to be 1-dimensional)

I am trying to determine the conformal predictions for my model with my data. But it gives me following error that occurs at icp.calibrate(X_cal, y_cal) : ...
1
vote
0answers
43 views

Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
1
vote
0answers
30 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
1
vote
0answers
30 views

pandas memory issues

What are some good "best practices" for dealing with large datasets in pandas and numpy? Typically i read data from csv files and use numpy and pandas to prepare them for machine learning and AI ...
1
vote
1answer
2k views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
1
vote
0answers
70 views

How to clean messy column and reshape data structure in pandas

I have a dataset that looks like this. There are 122 columns of coordinates excluding the date (Time) column and 7000 rows. What I want to do is split the lat and lon into two columns, and then ...

1
2 3 4 5