Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1,284
questions
0
votes
1
answer
11
views
unable to bar plot the data for all the columns
I am trying to plot the data in a loop, I am using the below code to build the plot. but Its running for all the loop, I can see the column name is getting printed for all th but its building the plot ...
-1
votes
0
answers
8
views
Filtering or Querying Pandas MultiIndex Dataframe based on column values
I have a multi-index pandas DataFrame such as below, primarily indexed with DateTime object.
...
0
votes
0
answers
11
views
What is the best way to acess the data in a pandas Window object?
I have a pandas DataFrame object, and I call its rolling method to get a Window object. Now I want to modify the data in each of the windows to get a new set of windows.
Reading the source code, I ...
0
votes
0
answers
14
views
Deep learning models for time series forecasting changing the input of the experiment
For my project I received the following pandas dataframe which contains the dissipated heat from an experiment. Each column represents the power provided to the experiment. The distance between each ...
- 1
0
votes
1
answer
21
views
0
votes
0
answers
10
views
How to capture a column of a data type
I want to put 2nd column in an array of the following data like [1,1,1,1,..]:
...
- 1
0
votes
1
answer
26
views
Does Scikit-Learn's OneHotEncoder make all Columns Categorical?
I've been using Scikit-Learn's OneHotEncoder to turn categorical data into binary columns, however, it seems that fitting ...
- 227
0
votes
1
answer
21
views
What Should I do with NaN Values when Transforming Categorial Data to Numerical Data?
I'm currently working on a dataset with several categorial features, which I need to transform into numerical features through one hot encoding.
However, some of the features have ...
- 227
0
votes
0
answers
33
views
Should I apply log transform?
Hsre is my skewness and kurtosis value before applying log transform. For skewness:
I use -3 < x < 3 as acceptable value and kurtosis at -10 < x < 10 as acceptable value. Since my data has ...
0
votes
1
answer
27
views
Linear Regression line not showing in plot
It's a silly problem, I know, but it's getting my nerves. Everything seems fine, but I cannot get the line to show on the plot.
I've put it in a public Google notebook, for your convenience.
t ...
-1
votes
1
answer
16
views
How do you Specify which Columns Pandas reads in?
I have a huge dataset with ~470,000 rows and ~1400 columns. I only require 184 of the available columns. I'd like to limit the columns I read in to improve my dataset's loading times.
How do you ...
- 227
0
votes
1
answer
25
views
How does Pandas' Correlation Method Handle Non-Numeric Columns?
I'm using Pandas' .corr() method to figure out which columns I can eliminate from a large dataset. Some of those columns have non-numeric types.
How does Pandas ...
- 227
1
vote
1
answer
32
views
How do you Speed up the Calculation of a Correlation Matrix on a Large Dataset in Pandas?
I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' ...
- 227
0
votes
0
answers
17
views
Should I remove outlier in features after group based on the result of target variable
I wonder should i remove outliers in case like this? or should i just left it be?
Also if i should remove the outlier, how should i do that? I know how to remove outliers in 1 feature like this
...
0
votes
1
answer
15
views
Conversion of DataFrame into array for Keras sequential Conv 1D model
I am new in the area of Sequential Conv 1D models.
My question is the following: When I am converting the DataFrame (X_train) into an array, doesn't that impact ...
0
votes
0
answers
13
views
How to decide which feature in DataFrame can be deleted (feature selection)?
I'm new to Machine Learning, and I'm working on dataset "Combined Cycle Power Plant over 6 years (2006-2011)", when the power plant was set to work with ...
- 1
0
votes
0
answers
4
views
During preprocessing when I apply the func on few rows as the data is big it throws error of 'Series' object has no attribute 'lower'
def transform_text(Status_information):
Status_information = Status_information.lower()
#Status_information = Status_information.apply(lambda x: x.lower())
Status_information = nltk.word_tokenize(...
- 1
0
votes
1
answer
16
views
select rows containing max value basing on another duplicated rows of a group
I want to select rows by the maximum values of another column which would be the duplicated rows containing duplicated maximum values of a group.
This should contain three steps:
(1) group dataframe ...
- 103
0
votes
2
answers
67
views
Understanding the relationship between two features
I am trying to solve classification task. Could you suggest me, if these two features are independent? the plot looks strange
0
votes
1
answer
29
views
0
votes
0
answers
13
views
Pandas dataframe reindex/resample datetime64
I have multiple Pandas dataframes where the indexes are datetime64.
I would like to merge them into one dataframe with a common time index, NaN padding for the shorter dataframes and NaNs filling for ...
0
votes
1
answer
53
views
Using Simple imputer replace NaN values with mean error
I am trying to replace 2 missing NaN values in data using the SimpleImputer.
I load my data as follow;
...
1
vote
1
answer
38
views
Pandas Data with Stacking Models
I am working in a ML project where I would like to predict the final result of a soccer game. There are three output categories: Local Win, Tie, Visitor Win.
In order to do so, I am using stacking ...
1
vote
1
answer
38
views
Is it advisable to save an ML model as a Joblib/Pickle file?
Part of our thesis project is to create a Diabetes predictor web application, and I have something I like to clarify. Is it a common practice to save an ML model as a Joblib/Pickle file like this one? ...
1
vote
1
answer
21
views
Creating a dataframe using roll-forward window on multivariate time series
Based on the simplifed sample dataframe
...
- 121
0
votes
1
answer
21
views
pd.get_dummies One-Hot Encoding but Keep the Order of Appearence
So I have this data which which has the 12 months as below:
When I one-hot encode it using pd.get_dummies the result is this:
You see that the new columns are ...
0
votes
0
answers
38
views
Python - stock price for different stocks plotting (without yfinance)
I have a dataset which contains a few stocks in a range of dates.
I searched in internet but most of the solutions gives solutions with yfinance and I'm not ...
0
votes
0
answers
42
views
Creating a scatter plot with matplotlib with large dataset
I am trying to create a scatter plot from instrument peak area data recorded every tenth of a second or so. For some reason, when I try to plot it with Matplotlib, I get a straight line, which is not ...
- 11
0
votes
1
answer
22
views
ignore exponent of a word in dataframe
I have a dataframe where there are columns which have exponent in their values(strings/words).
Example
Pandas reads them as Rasha, Fatiguec,Pyrexiab. is there any way i can make it read properly. or ...
- 13
0
votes
1
answer
21
views
Append Existing Columns to another Column in Pandas Dataframe
I have a data that looks like this:
The T2M indicates the temperature, and the followed row is the year. I want to append all the similar parameters columns under a single column having all the years,...
0
votes
1
answer
9
views
Transforming pandas dataframe from multiple series to categorical label
I have a dataframe in set up in this way:
depth
val1
val2
1
0.5
0.6
2
0.4
0.7
3
0.3
0.8
and I want to get it into this layout:
depth
y
category
1
0.5
val1
1
0.6
val2
2
0.4
val1
2
0.7
val2
3
...
- 101
0
votes
1
answer
38
views
Using Python/Pandas to compare columns of unequal length
https://ibb.co/cyCzVgy
I would like to be able to compare the NEW_PRICE to the LINE_TOTAL_ALLOWED amounts based on the ...
- 1
-1
votes
1
answer
89
views
how to delete duplicate dictionaries in list of dictionaries and extract the values of dictionary? [closed]
delete duplicate dictionaries in list of dictionary and extract the values of dictionary
...
- 1,714
0
votes
1
answer
20
views
Filter out transactions occurring within a timeframe with the same amount [closed]
I need to apply some filtering on a data frame using pandas.
Basically my data frame has the following column:
ID - The row id of the transaction
Timestamp - object was transformed to datetime format
...
- 13
0
votes
1
answer
144
views
when I run Random Forest classification model then at every rows of my train data set show this error (ValueError: could not convert string to float:)
'''
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 500, max_depth = None, min_samples_split=2, min_samples_leaf =1,
bootstrap = True, random_state=0)...
0
votes
1
answer
15
views
Different Correlation Coefficents with different Time Ranges
I built a Time-Series that displays the price of the Electricty Price in South Italy and two of their most important commodities (commodities, gas) used to produce the eletrical energy. So I ordered ...
0
votes
1
answer
29
views
Compute the pdf from pandas kde
I have data (features/targets in machine learning terminology), e.g. X1(t), X2(t), ... XN(t) and dependent variable y(t). I can use pandas to plot the kde's of the independent variables (X1(t),...).
I ...
- 81
0
votes
0
answers
19
views
why when I find the best accuracy for logistic regression then it give me this error (AttributeError: split not found)
after run this code I face the split not found error.
...
0
votes
1
answer
35
views
read_csv, non-printing ascii delimiters, and multi index
I have some text data which uses the ASCII data characters 0x1e "group separator" and 0x1d "record separator"...
- 101
0
votes
1
answer
79
views
How does one perform a Canova-Hansen test in Python?
I am referring to the documentation here, but it does not give many examples on how to actually perform the test. I have a pandas dataframe with two columns:
Column 1 is first day of every week,
...
- 5
0
votes
0
answers
13
views
Creating a prediction matrix for kernel regression in numpy
I need to create a rectangular matrix from timeseries data which is T time bins by L, the sum of the length of the set of events of interest occurring in the timeseries within certain times referred ...
- 103
0
votes
1
answer
68
views
Cumsum starting value not 0 in Pandas
I have the following dataframe df of athletes (indexed by Athlete_ID) and their Rank (indexed by ...
0
votes
1
answer
90
views
Set column names to column value in pandas forloop, add prefix to column names
I have a dataframe with repeat column names, and I am trying to add a prefix label to column names with an appropriate identifier that comes from a column within the dataframe. I have this data frame:
...
- 11
0
votes
0
answers
631
views
How to correctly create a PyTorch Tensor from a Pandas DataFrame?
I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data.
Obviously, This new tensor do ...
- 101
0
votes
1
answer
27
views
How to generate presentation tables out of pandas data frames?
After exploring and analyzing data with pandas I often create summarize tables (e.g. "table 1"). Currently I use pandas...
- 101
0
votes
1
answer
23
views
best clustering algorithm or model for clustering areas on map?
I have a database that has information such as Latitude, longitude, plus other information such as sightseeing locations, restaurants and shopping centers, if it's rural or suburb,... It also has ...
- 1
1
vote
2
answers
76
views
Average number of records by ID
I have a dataframe with IDs and booking refs, looking like the simplified example below.
ID
BookingRef
001
2019/32323
002
2011/23232
002
2017/7u4922
In the above example, 001 has one booking and ...
- 113
1
vote
2
answers
117
views
How can I do mathematical operations to two columns of a CSV file and save the result in a new CSV file?
I have a CSV file (such as test1.csv). There are tabular values like the following.
S1 S2 S3
4.6 3.2 2.1
3.2 4.3 5.4
1.4 3.4 6.1
I want ...
- 131
1
vote
0
answers
139
views
Turn duplicate rows in pandas dataframe into new columns in the same dataframe
I have a large list of dataframe with duplicate observations for the same index, analagous to the 'df' below:
...
- 11
1
vote
2
answers
41
views
How can I create new dataset maintaining the same trend from an existing dataset?
I am trying to create several new datasets based on an existing dataset. The new datasets should follow the trends (i.e., the correlation within the data) of that existing dataset.
As an example, I ...
- 131