Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
11 views

unable to bar plot the data for all the columns

I am trying to plot the data in a loop, I am using the below code to build the plot. but Its running for all the loop, I can see the column name is getting printed for all th but its building the plot ...
-1 votes
0 answers
8 views

Filtering or Querying Pandas MultiIndex Dataframe based on column values

I have a multi-index pandas DataFrame such as below, primarily indexed with DateTime object. ...
0 votes
0 answers
11 views

What is the best way to acess the data in a pandas Window object?

I have a pandas DataFrame object, and I call its rolling method to get a Window object. Now I want to modify the data in each of the windows to get a new set of windows. Reading the source code, I ...
0 votes
0 answers
14 views

Deep learning models for time series forecasting changing the input of the experiment

For my project I received the following pandas dataframe which contains the dissipated heat from an experiment. Each column represents the power provided to the experiment. The distance between each ...
  • 1
0 votes
1 answer
21 views

Creating function to write excel sheets using pandas

...
0 votes
0 answers
10 views

How to capture a column of a data type

I want to put 2nd column in an array of the following data like [1,1,1,1,..]: ...
  • 1
0 votes
1 answer
26 views

Does Scikit-Learn's OneHotEncoder make all Columns Categorical?

I've been using Scikit-Learn's OneHotEncoder to turn categorical data into binary columns, however, it seems that fitting ...
  • 227
0 votes
1 answer
21 views

What Should I do with NaN Values when Transforming Categorial Data to Numerical Data?

I'm currently working on a dataset with several categorial features, which I need to transform into numerical features through one hot encoding. However, some of the features have ...
  • 227
0 votes
0 answers
33 views

Should I apply log transform?

Hsre is my skewness and kurtosis value before applying log transform. For skewness: I use -3 < x < 3 as acceptable value and kurtosis at -10 < x < 10 as acceptable value. Since my data has ...
0 votes
1 answer
27 views

Linear Regression line not showing in plot

It's a silly problem, I know, but it's getting my nerves. Everything seems fine, but I cannot get the line to show on the plot. I've put it in a public Google notebook, for your convenience. t ...
-1 votes
1 answer
16 views

How do you Specify which Columns Pandas reads in?

I have a huge dataset with ~470,000 rows and ~1400 columns. I only require 184 of the available columns. I'd like to limit the columns I read in to improve my dataset's loading times. How do you ...
  • 227
0 votes
1 answer
25 views

How does Pandas' Correlation Method Handle Non-Numeric Columns?

I'm using Pandas' .corr() method to figure out which columns I can eliminate from a large dataset. Some of those columns have non-numeric types. How does Pandas ...
  • 227
1 vote
1 answer
32 views

How do you Speed up the Calculation of a Correlation Matrix on a Large Dataset in Pandas?

I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' ...
  • 227
0 votes
0 answers
17 views

Should I remove outlier in features after group based on the result of target variable

I wonder should i remove outliers in case like this? or should i just left it be? Also if i should remove the outlier, how should i do that? I know how to remove outliers in 1 feature like this ...
0 votes
1 answer
15 views

Conversion of DataFrame into array for Keras sequential Conv 1D model

I am new in the area of Sequential Conv 1D models. My question is the following: When I am converting the DataFrame (X_train) into an array, doesn't that impact ...
0 votes
0 answers
13 views

How to decide which feature in DataFrame can be deleted (feature selection)?

I'm new to Machine Learning, and I'm working on dataset "Combined Cycle Power Plant over 6 years (2006-2011)", when the power plant was set to work with ...
  • 1
0 votes
0 answers
4 views

During preprocessing when I apply the func on few rows as the data is big it throws error of 'Series' object has no attribute 'lower'

def transform_text(Status_information): Status_information = Status_information.lower() #Status_information = Status_information.apply(lambda x: x.lower()) Status_information = nltk.word_tokenize(...
  • 1
0 votes
1 answer
16 views

select rows containing max value basing on another duplicated rows of a group

I want to select rows by the maximum values of another column which would be the duplicated rows containing duplicated maximum values of a group. This should contain three steps: (1) group dataframe ...
  • 103
0 votes
2 answers
67 views

Understanding the relationship between two features

I am trying to solve classification task. Could you suggest me, if these two features are independent? the plot looks strange
0 votes
1 answer
29 views

calculate sklearn metrics from 2d array

I have the following frame of actual value, ...
0 votes
0 answers
13 views

Pandas dataframe reindex/resample datetime64

I have multiple Pandas dataframes where the indexes are datetime64. I would like to merge them into one dataframe with a common time index, NaN padding for the shorter dataframes and NaNs filling for ...
0 votes
1 answer
53 views

Using Simple imputer replace NaN values with mean error

I am trying to replace 2 missing NaN values in data using the SimpleImputer. I load my data as follow; ...
1 vote
1 answer
38 views

Pandas Data with Stacking Models

I am working in a ML project where I would like to predict the final result of a soccer game. There are three output categories: Local Win, Tie, Visitor Win. In order to do so, I am using stacking ...
1 vote
1 answer
38 views

Is it advisable to save an ML model as a Joblib/Pickle file?

Part of our thesis project is to create a Diabetes predictor web application, and I have something I like to clarify. Is it a common practice to save an ML model as a Joblib/Pickle file like this one? ...
1 vote
1 answer
21 views

Creating a dataframe using roll-forward window on multivariate time series

Based on the simplifed sample dataframe ...
0 votes
1 answer
21 views

pd.get_dummies One-Hot Encoding but Keep the Order of Appearence

So I have this data which which has the 12 months as below: When I one-hot encode it using pd.get_dummies the result is this: You see that the new columns are ...
0 votes
0 answers
38 views

Python - stock price for different stocks plotting (without yfinance)

I have a dataset which contains a few stocks in a range of dates. I searched in internet but most of the solutions gives solutions with yfinance and I'm not ...
0 votes
0 answers
42 views

Creating a scatter plot with matplotlib with large dataset

I am trying to create a scatter plot from instrument peak area data recorded every tenth of a second or so. For some reason, when I try to plot it with Matplotlib, I get a straight line, which is not ...
0 votes
1 answer
22 views

ignore exponent of a word in dataframe

I have a dataframe where there are columns which have exponent in their values(strings/words). Example Pandas reads them as Rasha, Fatiguec,Pyrexiab. is there any way i can make it read properly. or ...
0 votes
1 answer
21 views

Append Existing Columns to another Column in Pandas Dataframe

I have a data that looks like this: The T2M indicates the temperature, and the followed row is the year. I want to append all the similar parameters columns under a single column having all the years,...
0 votes
1 answer
9 views

Transforming pandas dataframe from multiple series to categorical label

I have a dataframe in set up in this way: depth val1 val2 1 0.5 0.6 2 0.4 0.7 3 0.3 0.8 and I want to get it into this layout: depth y category 1 0.5 val1 1 0.6 val2 2 0.4 val1 2 0.7 val2 3 ...
0 votes
1 answer
38 views

Using Python/Pandas to compare columns of unequal length

https://ibb.co/cyCzVgy I would like to be able to compare the NEW_PRICE to the LINE_TOTAL_ALLOWED amounts based on the ...
-1 votes
1 answer
89 views

how to delete duplicate dictionaries in list of dictionaries and extract the values of dictionary? [closed]

delete duplicate dictionaries in list of dictionary and extract the values of dictionary ...
  • 1,714
0 votes
1 answer
20 views

Filter out transactions occurring within a timeframe with the same amount [closed]

I need to apply some filtering on a data frame using pandas. Basically my data frame has the following column: ID - The row id of the transaction Timestamp - object was transformed to datetime format ...
  • 13
0 votes
1 answer
144 views

when I run Random Forest classification model then at every rows of my train data set show this error (ValueError: could not convert string to float:)

''' from sklearn.ensemble import RandomForestClassifier forest = RandomForestClassifier(n_estimators = 500, max_depth = None, min_samples_split=2, min_samples_leaf =1, bootstrap = True, random_state=0)...
0 votes
1 answer
15 views

Different Correlation Coefficents with different Time Ranges

I built a Time-Series that displays the price of the Electricty Price in South Italy and two of their most important commodities (commodities, gas) used to produce the eletrical energy. So I ordered ...
0 votes
1 answer
29 views

Compute the pdf from pandas kde

I have data (features/targets in machine learning terminology), e.g. X1(t), X2(t), ... XN(t) and dependent variable y(t). I can use pandas to plot the kde's of the independent variables (X1(t),...). I ...
  • 81
0 votes
0 answers
19 views

why when I find the best accuracy for logistic regression then it give me this error (AttributeError: split not found)

after run this code I face the split not found error. ...
0 votes
1 answer
35 views

read_csv, non-printing ascii delimiters, and multi index

I have some text data which uses the ASCII data characters 0x1e "group separator" and 0x1d "record separator"...
  • 101
0 votes
1 answer
79 views

How does one perform a Canova-Hansen test in Python?

I am referring to the documentation here, but it does not give many examples on how to actually perform the test. I have a pandas dataframe with two columns: Column 1 is first day of every week, ...
  • 5
0 votes
0 answers
13 views

Creating a prediction matrix for kernel regression in numpy

I need to create a rectangular matrix from timeseries data which is T time bins by L, the sum of the length of the set of events of interest occurring in the timeseries within certain times referred ...
0 votes
1 answer
68 views

Cumsum starting value not 0 in Pandas

I have the following dataframe df of athletes (indexed by Athlete_ID) and their Rank (indexed by ...
0 votes
1 answer
90 views

Set column names to column value in pandas forloop, add prefix to column names

I have a dataframe with repeat column names, and I am trying to add a prefix label to column names with an appropriate identifier that comes from a column within the dataframe. I have this data frame: ...
0 votes
0 answers
631 views

How to correctly create a PyTorch Tensor from a Pandas DataFrame?

I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data. Obviously, This new tensor do ...
  • 101
0 votes
1 answer
27 views

How to generate presentation tables out of pandas data frames?

After exploring and analyzing data with pandas I often create summarize tables (e.g. "table 1"). Currently I use pandas...
  • 101
0 votes
1 answer
23 views

best clustering algorithm or model for clustering areas on map?

I have a database that has information such as Latitude, longitude, plus other information such as sightseeing locations, restaurants and shopping centers, if it's rural or suburb,... It also has ...
  • 1
1 vote
2 answers
76 views

Average number of records by ID

I have a dataframe with IDs and booking refs, looking like the simplified example below. ID BookingRef 001 2019/32323 002 2011/23232 002 2017/7u4922 In the above example, 001 has one booking and ...
  • 113
1 vote
2 answers
117 views

How can I do mathematical operations to two columns of a CSV file and save the result in a new CSV file?

I have a CSV file (such as test1.csv). There are tabular values like the following. S1 S2 S3 4.6 3.2 2.1 3.2 4.3 5.4 1.4 3.4 6.1 I want ...
  • 131
1 vote
0 answers
139 views

Turn duplicate rows in pandas dataframe into new columns in the same dataframe

I have a large list of dataframe with duplicate observations for the same index, analagous to the 'df' below: ...
1 vote
2 answers
41 views

How can I create new dataset maintaining the same trend from an existing dataset?

I am trying to create several new datasets based on an existing dataset. The new datasets should follow the trends (i.e., the correlation within the data) of that existing dataset. As an example, I ...
  • 131

1
2 3 4 5
26