Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Can label encoder be used to deal with unordered category variables?

Many people say that one hot should be used to deal with unordered category variables, but I see that there is a high praise project on kaggle, which directly uses label encoder to deal with unordered ...
user avatar
1 vote
2 answers
45 views

python compare 2 dataframes [closed]

in reference to this post, python-pandas-compare-columns-in-separate-dataframes-then-delete-non-matching, I can't get the isin function to work with my 2 example dataframes : df 1 : ...
user avatar
0 votes
2 answers
43 views
+50

Global feature importance for classification problem where some values are irrelevant

I have a binary classification problem where most of the features are categorical with 4 possible values: yes, no, irrelevant, nan. I am trying to find the modular global feature importance of those ...
user avatar
  • 51
0 votes
0 answers
11 views

need to get value based on condition in pandas [closed]

I have a excel where I have CustomerName and ID. I have a separate customer transaction in anther excel I need to assign coustomerID in the Transaction excel based on the name of the customername. ...
user avatar
0 votes
0 answers
31 views

I'm iterating a dataframe with a for loop, does anybody know a faster way of doing this? [migrated]

I'm working on a dataframe where I need to add a new column which is based on other existing rows in the dataframe. Here is a simplified version of what I'm trying to do. Basically there is a ...
user avatar
0 votes
1 answer
15 views

How do I not display rows that have an empty value when trying to output a dataframe with pandas [closed]

I have this table where there are missing values under the value2 column. Value1 Value2 1000 1000 1000 500 1000 560 1000 560 What I would like to do is to display the above table but without ...
user avatar
  • 15
0 votes
0 answers
11 views

Constructing pandas DF with model trained categorical variable

I've trained a lightGBM model on a dataset X where X has a categorical (in the pandas sense) variable. This model trains fine and when I predict using it all looks good - I can even change the value ...
user avatar
0 votes
0 answers
15 views

Changing a 0-1 column datatype from int64 to uint8 such as in pandas.get_dummies()

Is it advisable to change the datatype int64 of a 0-1's column to uint8 such as ...
user avatar
1 vote
0 answers
16 views

Create a Nested Dictionary from a Pandas Dataframe [migrated]

I have a dataframe with columns like this: Name, Education, DOB, ...
user avatar
0 votes
0 answers
15 views

How to find location for stretch of values more than fixed number and their average in dataframe python3 [migrated]

I have a data-frame something like this: And I wish to find the the location (start and end) and the average of the stretch of stretch where values in column 2 are more than 0.2. Something like this: ...
user avatar
  • 3
0 votes
1 answer
41 views

Merging two datasets with different features for machine learning prediction

I'm trying to create a model which predicts Real estate prices with xgboost in machine learning, my question is : Can i combine two datasets to do it ? First dataset : 13 features Second dataset : ...
user avatar
0 votes
0 answers
8 views

Turning multiple binary columns into categorical (with less columns) with Python Pandas

I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
user avatar
  • 21
0 votes
1 answer
49 views

How to groupby and sum values of only one column based on value of another column

I have a dataset that has the following columns: Category, Product, Launch_Year, and columns named 2010, 2011 and 2012. These 13 columns contain sales of the product in that year. The goal is to ...
user avatar
1 vote
1 answer
28 views

How to return the number of values that has a specific count

I would like to find how many occurrences of a specific value count a column contains. For example, based on the data frame below, I want to find how many values in the ID column are repeated twice <...
user avatar
  • 15
0 votes
1 answer
14 views

How does Pandas store timezone information?

Timestamp columns are stored internally in UTC, regardless of whether they have a timezone or not. This is stated in the docs: https://pandas.pydata.org/docs/user_guide/timeseries.html#working-with-...
user avatar
  • 101
1 vote
2 answers
175 views

Compare string entries of columns in different pandas dataframes

I have two dataframes, df1 and df2, both with different number of rows. df1 has a column 'NAME', a short string; and df2 has a column 'LOCAL_NAME', a much longer string that may contain the exact ...
user avatar
0 votes
0 answers
4 views

Identifing this dataset for sanitising

I am beginner here starting with data science for analytics. I am trying to figure out what data set this is and how to read it from python. I have an idea of the steps but not sure how to code it in ...
user avatar
  • 1
0 votes
0 answers
51 views

ValueError: continuous is not supported

I am working on a regression problem and building a model using Random Forest Regressor but while trying to get the accuracy I am getting ValueError: continuous is not supported. ...
user avatar
-1 votes
2 answers
35 views

unable to pass X_train and y_train in my regressor variable. i got a ValueError

...
user avatar
0 votes
2 answers
106 views

Exploratory data analysis (EDA) on large dataset

I am working with lots of data (we have a table that produces 30 million rows daily). What is the best way to explore it (do on EDA)? Take a frictional slicing of the data randomly (100000 rows) or ...
user avatar
  • 25
0 votes
0 answers
19 views

How to optimize Pandas DataFrame reorganizing?

I have a DataFrame which looks like this: date person value 2022-05-01 A 5 2022-05-01 B 4 2022-05-02 A 5 2022-05-02 B 9 I want to convert it to that form: ...
user avatar
0 votes
1 answer
87 views

Python: calculate the weighted average correlation coefficient

I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
user avatar
  • 101
0 votes
0 answers
26 views

write a dataframe using a column that contains a cycle (number of the weekin every year) in python

I want to write a dataframe with a column of the number of week and make it as a row that contains 53 columns (I mean weeks). I really need your help. NOTE THAT: each year has 53 week, and the column ...
user avatar
  • 1
0 votes
0 answers
18 views

NaNs in predictions with LSTM

I have an LSTM model that I have trained and tested it with a dataset. Now I want to test it to an other dataset and I use the following snippet: ...
user avatar
  • 101
0 votes
2 answers
34 views

Create new rows based on a value in a column

My dateset is generated like the example df = {'event':['A','B','C','D'], 'budget':['123','433','1000','1299'], 'duration_days':['6','3','4','2']} I need ...
user avatar
2 votes
1 answer
23 views

The single CSV created by combining a large number of CSV files is too large to process. What options do I have?

The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
user avatar
1 vote
0 answers
43 views

Encode each comma separated value in Pandas

I have a dataset ...
user avatar
  • 19
0 votes
0 answers
23 views

How can i merge two datasets with similar words in python?

For instance i have a row value on the dataset_1 : "Entity" = Apple dataset_2 : "Entity" = iCloud Apple (Entity is the column) I need to merge one dataset to the other by the ...
user avatar
0 votes
0 answers
23 views

How to sample a dataframe or numpy array with a particular interval?

I have the following dataframe : ...
user avatar
  • 1
0 votes
1 answer
54 views

Plot multiple time series from single dataframe

I have a dataframe with multiple time series and columns with labels. My goal is to plot all time series in a single plot, where the labels should be used in the legend of the plot. The important ...
user avatar
2 votes
0 answers
23 views

Not able to encode multiple categorical columns at once

I have written the following code for encoding categorical features of the dataframe( named 't') - ...
user avatar
2 votes
0 answers
24 views

How to perform Timeseries Forecasting on dataset with repeating dates?

I have a dataset, I have to perform timeseries forecasting on that dataset. The dataset has date column, the dates in date column are duplicated. We have 5 classes, a date will have a sales record for ...
user avatar
0 votes
0 answers
8 views

Autocorrelation Chart showing different colours and no Shadow for checking the Orders to be used for ARIMA

I have been doing a project on Time Series data and one of the items was to test for AutoCorrelation and found the charts in a different color than what I expected. I also don't see the shaded part ...
user avatar
0 votes
1 answer
14 views

What to do with missing origin city values

Hello fellow data scientists. I am new in this field, and I face to a problem, for what I need an advice. So I have data, where one column is product ID, and another which say from which city it ...
user avatar
0 votes
0 answers
6 views

iteration and if condition -python

...
user avatar
0 votes
0 answers
6 views

Prophet Weight Cloumn On Time Series (Python)

Can we add a weight column for times on Prophet? So I want the values ​​of old dates to be less important while recent dates are more important. Is there a parameter or method for this? Or does the ...
user avatar
0 votes
0 answers
6 views

Handling missing values in IP addresses and key-like features

I have a log dataset that contains +30 features. One group of these features are of the following type, for example, request_id, user_partyrole_id, authentication_id, user_login_key and such ip and ...
user avatar
0 votes
1 answer
23 views

How to set a new numeric index

I have read data from a json file, and the dataframe looks like: Col1 Col2 Col3 A: 1 Hi 7 B: 22 CC 1 I think it is because the file was a ...
user avatar
0 votes
0 answers
12 views

Formatting df's multi index

I have a multi-index dataframe used in a block of code. It's index looks like this: ...
user avatar
  • 101
0 votes
0 answers
41 views

regression.fit(x_train, y_train) is not working on python

I try to deal with my homework. The Job is to take the Achievement data and perform a multi-linear regression on it. The code is published here. I am quite new to programming in Python and in data ...
user avatar
0 votes
0 answers
135 views

ValueError: Input contains NaN, infinity or a value too large for dtype('float64') - fitting a model

I am attempting to train a model on a pandas DataFrame and am running into ValueError issue. I have verified that there are no ...
user avatar
  • 302
1 vote
1 answer
40 views

max temperature for every month given several months and several years

I'm not sure if this is the right place to put this. So I have a dataframe consisting of just 4 columns but around 2000 rows. It's in a csv format. A typical value of the first column which is titled ...
user avatar
  • 133
1 vote
0 answers
15 views

How should I OneHotEncod a column of (8128 rows and) 2058 nuniques?

The title, pretty much. I just want to know the best and most efficient way to OneHotEncode a column with like 2058 nuniques. Doing a ...
user avatar
0 votes
0 answers
24 views

Pandas Dataframe Filtering based on rows

I have a dataset like this: ...
user avatar
  • 1
0 votes
0 answers
96 views

pandas - Find rows where text in one column is in text in another column using vectorized operations

I wish to find instances where text in col2 is contained in col1; ultimately, I want to remove the text from col1 (replace the col2 text in col1 with ''). I want to achieve this using vectorized ...
user avatar
  • 121
1 vote
0 answers
30 views

How to solve this IndexError? [closed]

I have created a training dataframe Traindata as following: dataFile='/content/drive/Colab Notebooks/.../Normal_Anomalous_8Digits.csv' data8=pd.read_csv(dataFile) ...
user avatar
  • 21
0 votes
2 answers
80 views

How can I add a column indicating clusters on my original df? Python

I have clustered pitchers in baseball based off of averages of advanced metrics. I am working with statcast data, so every data point is a pitch thrown as follows: pitcher_name batter_name spin_rate ...
user avatar
0 votes
1 answer
46 views

what exactly is the "order"-parameter in pandas interpolation?

what exactly is the "order"-parameter in pandas interpolation? it is mentioned in the doc: ...
user avatar
1 vote
1 answer
139 views

how many rows does pandas' interpolate consider? [closed]

How does pandas' DataFrame.interpolate() work in relation to the amount of rows it considers: is it just the row before the NaNs and the row right after? Or is it the whole DataFrame (how does that ...
user avatar
1 vote
1 answer
39 views

Natural language processing [closed]

I am new to NLP. I converted my JSON file to CSV with the Jupyter notebook. I am unsure how to proceed in pre-processing my data using techniques such as tokenization and lemmatization etc. I ...
user avatar
  • 11

1
2 3 4 5
25