Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

224 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
28 views

Spatially constrained geospatial similarity

What's the current methodology for clustering geospatial data by features? Example: I have some demographic dataset. Let's say this contains average home price and population density. So, an example ...
4
votes
0answers
1k views

How does Seaborn calculate error bars when using estimators other than the arithmetic mean?

If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?
3
votes
0answers
448 views

How to apply a groupby rolling function to create multiple columns in the dataframe

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo and the link to the data is here and the example here. Some Sample of data ...
3
votes
1answer
98 views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
3
votes
0answers
92 views

Pandas: Calculating the best seller to incorporate number of houses sold

I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller ...
3
votes
0answers
124 views

Create top 10 index fund based on >100 stocks

Data I have a zip with ca M=160 csv files, with one csv file per stock. The ticker symbol is in the file name. Each csv file contain the same columns: date, txVolume(USD), txCount, marketcap(USD), ...
2
votes
0answers
23 views

Using an unmerged branch from StatsModels for Anaconda

I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive. See this blog, which as some point mentions, almost ...
2
votes
1answer
240 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
2
votes
0answers
37 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
2
votes
1answer
248 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
2
votes
1answer
134 views

How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
2
votes
2answers
522 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
2
votes
1answer
67 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
2
votes
1answer
70 views

Plotting three lines on the same plot (with four hr frequency)

I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used: ...
2
votes
1answer
337 views

memory problem due to large file

I'm new in Python and definitely, I'm sure I do some mistake. Here is my problem and thanks all to your help in advance. I have 2 files (one is Hive) and the other is csv and merging them. I have ...
2
votes
0answers
83 views

How to predict multiple iterations in the future based on current close prices? Keras Python

I have data from CoinMarketCap for the close prices of Bitcoin. How can I use this data to predict multiple days into the future using past days with a function? I want the function to look like: <...
2
votes
2answers
606 views

Partitioning data into features/labels and train/test after reading from csv file

I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
2
votes
1answer
82 views

Need help on Time Series ARIMA Model

I'm working on forecasting daily volumes and have used time series model to check for data stationarity. However, I'm strugging at forecasting data with 90% accuracy. Right now variation is extremely ...
2
votes
0answers
81 views

How to prevent pd.ewma from changing the index?

I am resampling my time series data using the following code: bars = transactions.price.resample('4H', base = 3).ohlc() This allows me to account for DST ...
2
votes
0answers
1k views

Multidimensional Groupby to ND Array

My objective is to build an N-dimensional array in Python that has the mean of the intersection of a number of features. For example, I have the number of customers in a store at any given time. I ...
2
votes
1answer
70 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
2
votes
0answers
842 views

pandas dataframes memory

I have a question about memory usage. I want to do 4 things: ...
2
votes
0answers
481 views

How to fix similarity matrix in Pandas returning all NaNs?

I've been trying to create a similarity matrix in Pandas from with a matrix multiplication operation on a document-term count matrix with 2264 rows and 20475 columns. The calculation completes in ...
1
vote
1answer
39 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
1
vote
0answers
20 views

Does statsmodels fully support MultiIndex?

The below code snippet shows how statsmodels seems to flatten MultiIndex tuples by joining them with an underscore "_". ...
1
vote
0answers
17 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
1
vote
0answers
16 views

pandas memory issues

What are some good "best practices" for dealing with large datasets in pandas and numpy? Typically i read data from csv files and use numpy and pandas to prepare them for machine learning and AI ...
1
vote
0answers
15 views

Split-Half Reliability Python

I have a pandas dataframe of baseball stats that I have transposed that looks as follows: Each column represents a separate plate appearance for a player. Each player can (and usually is) represented ...
1
vote
1answer
22 views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
1
vote
1answer
43 views

Plot hurricane track

Hello folks, Recently while going through a research paper I happened to come across a plot showing the hurricane track(figure attached). Can anyone say how it is plotted or anyone has ...
1
vote
0answers
17 views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
1
vote
0answers
49 views

Convert Pandas Dataframe with mixed datatypes to LibSVM format

I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
1
vote
1answer
30 views

How to apply .replace() inside a column in a pandas data frame to clean data

I have a pandas data frame where I want to replace all the values that are not numeric in one column for "" The error I'm getting in my code is the following: ...
1
vote
0answers
28 views

How to find the position of a company given other companies in a Pandas DataFrame with Python

So I have a Pandas DataFrame. I am doing some data analysis with Python with some sets of ...
1
vote
1answer
60 views

Python dataframe iterate over rows

Dataset ...
1
vote
0answers
15 views

Resul Logit, meaning of values Python scikit-learn?

What does the marked paces mean ?: Current function value, Df Model, Df Residuals, Pseudo R-squared, Log-Likehood, LL-Null, LLR p-value, Coef, Std Err., z, [0,025 0.975] ? Could you give mean a short ...
1
vote
0answers
28 views

How do I pivot a table while keeping the other columns intact

I have looked around and I have not been able to exactly get the result I am looking for. I have the following Dataframe: My goal, is to have a dataframe that for every year-state combination, has ...
1
vote
1answer
17 views

In feature selection, I came across a situation where NaN were filled by median of the column values

Why the median value is used for NaN? Why not something else like mean? What is the logic behind using the median value?
1
vote
2answers
37 views

How to deal with errors of defining data types in pandas' read_csv ()?

I have a table with 118,000 rows and 80 columns. I would like to select 8 columns from the table. I am reading the file using the pandas function pd.read_csv ...
1
vote
1answer
19 views

How to summarize multiple time series like dataset

I am running an experiment where I am sampling CPU utilization every second. A snippet of the data from each experiment run is something like this: ...
1
vote
0answers
56 views

Pandas - dataframe not storing images properly

I'm trying to organize image files I have into a Pandas dataframe. I'm using Keras's preprocessing library for working with them. Here's how I load them in: ...
1
vote
1answer
656 views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
1
vote
0answers
22 views

Training on small, sparse vectors included in one big, denormalized table/dataframe?

So I have a table like this: ...
1
vote
0answers
352 views

Lemmatization of pandas column using Wordnet after POS

I have a pandas column df_travail[line_text] with text. I want to lemmatize each word of this column. First I Lowercase the text : ...
1
vote
1answer
222 views

Python Pandas agg error

I am trying to generate descriptive statistics using agg function in Pandas. I am having trouble with one line with a lambda function. They work when I run them as separate lines of code, but when I ...
1
vote
1answer
2k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
1
vote
0answers
62 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...
1
vote
1answer
46 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
1
vote
0answers
35 views

Method for 'Simple' Clustering?

In a recent adult education class I took the prof shared a spreadsheet that was written in VBA code and takes ages to execute, I'm sure ...
1
vote
0answers
36 views

Pipeline that cached the results

I use pandas to do feature extraction for machine learning. I hope to achieve the following: Consider I have five data processing steps done sequentially, and I execute it once, the results will be ...

1
2 3 4 5