Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

227 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
1answer
1k views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
4
votes
1answer
40 views

Spatially constrained geospatial similarity

What's the current methodology for clustering geospatial data by features? Example: I have some demographic dataset. Let's say this contains average home price and population density. So, an example ...
3
votes
0answers
59 views

Detecting pushups based on pose data

I've been playing with Google's MLKit, and decided to detect push ups. As a quick test, I took the position of the left shoulder, and plotted the Y Axis. Here's how a variety of trials look: Five ...
3
votes
3answers
20k views

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python

I have written the following code for a neural network to perform regression on a dataset, but I am getting a ValueError. I have looked up to different answers and ...
3
votes
0answers
1k views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
3
votes
0answers
2k views

How to apply a groupby rolling function to create multiple columns in the dataframe

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo and the link to the data is here and the example here. Some Sample of data ...
3
votes
1answer
796 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
3
votes
1answer
15k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
3
votes
3answers
366 views

Is it a best practice to exclude retweets from the data set?

I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is ...
3
votes
1answer
989 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
3
votes
1answer
179 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
3
votes
1answer
87 views

Plotting three lines on the same plot (with 4-hour frequency)

I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used: ...
3
votes
2answers
1k views

Partitioning data into features/labels and train/test after reading from csv file

I need to read data from a CSV file and then the first partition that data into features and labels and then into the training and testing set. However, there are several issues cropping up again and ...
3
votes
1answer
93 views

Need help on Time Series ARIMA Model

I'm working on forecasting daily volumes and have used time series model to check for data stationarity. However, I'm strugging at forecasting data with 90% accuracy. Right now variation is extremely ...
3
votes
0answers
125 views

Pandas: Calculating the best seller to incorporate number of houses sold

I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller ...
3
votes
0answers
146 views

Create top 10 index fund based on >100 stocks

Data I have a zip with ca M=160 csv files, with one csv file per stock. The ticker symbol is in the file name. Each csv file contain the same columns: date, txVolume(USD), txCount, marketcap(USD), ...
2
votes
1answer
18 views

How to calculate the difference based on matching criteria

Hello, I am trying to transfer over from excel to pandas. I want to add new column called 'daily_volume' where if the 'project_name' is equal to the above row project_name then calculate the ...
2
votes
0answers
257 views

Storing multiple time series in a python pandas dataframe efficiently

I got different time series. Each time series contains different attributes, which have data entries at different points of time (timestamps). Each time series is labeled and has a label attribute. I ...
2
votes
1answer
148 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
2
votes
0answers
89 views

Does statsmodels fully support MultiIndex?

The below code snippet shows how statsmodels seems to flatten MultiIndex tuples by joining them with an underscore "_". ...
2
votes
1answer
133 views

What's an appropriate datastore for variable length sequence data for PyTorch consumption?

I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats. I need a datastore that can rapidly ...
2
votes
1answer
913 views

Convert Pandas Dataframe with mixed datatypes to LibSVM format

I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
2
votes
1answer
88 views

Normalize data from different groups

I have data that has been grouped into 27 groups by different criteria. The reason for these groupings is to show that each group has different behavior. However, I would like to normalize everything ...
2
votes
0answers
58 views

Using an unmerged branch from StatsModels for Anaconda

I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive. See this blog, which as some point mentions, almost ...
2
votes
2answers
53 views

How would I load a database with multiple inconsistent headers?

I have a database with multiple headers like so: ...
2
votes
0answers
43 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
2
votes
1answer
568 views

How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
2
votes
2answers
1k views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
2
votes
0answers
147 views

How to predict multiple iterations in the future based on current close prices? Keras Python

I have data from CoinMarketCap for the close prices of Bitcoin. How can I use this data to predict multiple days into the future using past days with a function? I want the function to look like: <...
2
votes
0answers
88 views

How to prevent pd.ewma from changing the index?

I am resampling my time series data using the following code: bars = transactions.price.resample('4H', base = 3).ohlc() This allows me to account for DST ...
2
votes
0answers
1k views

Multidimensional Groupby to ND Array

My objective is to build an N-dimensional array in Python that has the mean of the intersection of a number of features. For example, I have the number of customers in a store at any given time. I ...
2
votes
1answer
81 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
1
vote
1answer
28 views

How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
1
vote
1answer
25 views

How to find difference between two dates in different dataframes

I want to create a difference between two date columns from different dataframes based on customer ID from the first dataframe. For example: I find max date from the first dataframe for all customers: ...
1
vote
0answers
27 views

What kind of data structure should I use for this data?

I've been using Pandas with the exhaust data from my job for about 2 years now, but I think I've somewhat outgrown strictly tabular data structures. My problem: I work for a large logistics company (...
1
vote
1answer
21 views

When should we use jaccard score?

I am a newbie in Machine Learning, I trained a binary classifier for bank loan prediction through Logistic Regression. I measured the accuracy of it with two methods: accuracy score and jaccard index. ...
1
vote
0answers
26 views

How to make the values in different columns in the correct order based on another data frame (mapping) in Python Pandas

I am pretty new to Python and Pandas and I struggle with combining a messy dataframe from excel with a mapping. I have tried to find some solutions on the Internet, however with no success. My first ...
1
vote
0answers
93 views

CDF plot overlay histogram in python

I have a dataframe called df['ProgressStep'] I would like to get overlaid CDF plot in histogram. Have tried 2 methods, neither one meet my target perfectly. please help to fine tune the code, either ...
1
vote
0answers
33 views

Numpy-like reduction of particular dimensions in Pandas DataFrame?

With a numpy multidimensional arrays, it is easy to reduce particular axes with functions such as np.mean or np.sum. For example,...
1
vote
1answer
21 views

Removing features that correlate with the target label

I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
1
vote
0answers
13 views

Using Quarterly Percent Change as Validation

this is a newbie question but it's very simple so I'm hoping it's okay to ask. I would like to calculate the percent change from one quarter to the next using Pandas and use that as my Y-Test (...
1
vote
3answers
103 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
1
vote
1answer
37 views

How to group by one column and find second occurance of values greater than a threshold

The Problem I have a pandas dataframe that contains series of people, the week number that a visit occurred, and their systolic and diastolic blood pressures. ...
1
vote
0answers
12 views

creating dataset/storing vectors from c++ to python

I am new to the data science field. I need to create dataset which consists of n inputs (vectors of size 5) and n outputs (vectors of size > 10,000). These vectors are generated using c++ code. ...
1
vote
0answers
125 views

How do I convert my dataframe column into vectors using word2vec?

I have the column of Categories of vary length in my DataFrame as shown below : Categories [Restaurants , Drinks] [Restaurants , Drinks , Rooftop] [Dinner] . . . ...
1
vote
2answers
111 views

How do I assign specific values to categorical variables

I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...
1
vote
1answer
107 views

Still have the same error after reshaping

I set my column 1 to 13 as X and column 14 as my Y. My data set has 14 columns and 401 rows including the parameters ...
1
vote
1answer
2k views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
1
vote
0answers
32 views

How to generate data for future price predictions with Linear regression and Python

I have written the following Python code that makes predictions using linear regression model. However, I'll appreciate you pointing out what went wrong with how I generated data['X_forecast']. I'm ...
1
vote
1answer
261 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...

1
2 3 4 5