Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
949
questions
-1
votes
0answers
13 views
How to create a new column containing part of the day (night, morning, afternoon,evening) [closed]
I wish to create a new column that contains the part of the day .I would consider {night(00:01 to 6:00),morning(6:01 to 12:00), afternoon(12:01 to 18:00), evening(18:011 to 00:00)}
here is the ...
-1
votes
0answers
18 views
Cannot install pandas-profiling because of “llvmlite” [closed]
Windows 10
I have been trying to install pandas-profiling package for the past 3 hours
I keep getting an error when it tries to install "llvmlite" which is included with the package I guess
...
1
vote
0answers
13 views
Making a tagged Part-of-Speech corpus with the help of a lexicon
I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe.
Also, I have a list of tokens (words) in another Dataframe.
I want to take each ...
-1
votes
0answers
12 views
how to convert a large xml file containing multiple xml pages into python pandas? [closed]
Every xml page is separated by a \n in the large file. So i have a large xml file sometimes about 0.5gb and in it there are multiple xml pages separated by \n.
imagine that i need to take this file ...
-1
votes
2answers
24 views
How can I get all column values from a panda dataframe and save them into one string [closed]
this question has been ask couple of times but none of the answer is working for that what I want.
For example I have this excel Dokument:
...
-1
votes
0answers
16 views
What is the wrong in my histogram code? [closed]
The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
...
0
votes
1answer
19 views
List value in Pandas DataFrame column makes analysis harder
Should I move to database?
I have a list of courses data in JSON format that looks like this:
...
0
votes
1answer
20 views
Pandas: Compare two Dataframe and Groupby categorical [closed]
My Question is about pandas DataFrame, I have two DataFrame both follow the same structure.
df_1:
...
0
votes
1answer
29 views
How do I transform a file to .txt file using pandas?
I have to submit a machine learning project, and it has to be node in a .txt file. I know that if I am using pandas, and I want to transform a file from another format to .csv format I can use .to_csv(...
0
votes
0answers
24 views
Clean up a pandas column using regex
I have a Python Pandas dataframe with a column named "codes". This column is very messy and contains the following values:
...
0
votes
2answers
26 views
What is the best way to treat datetime in the preprocessing step of machine learning
I have two datetime columns in my dataset. What I have done so far I have extracted year, ...
0
votes
1answer
18 views
Looping 2 DataFrame Pandas [closed]
I am learning about pandas and find an interesting case
i try to find total sales for every genre that exist in the data frame,I try looping a few time and get a dead end,maybe you can give me a few ...
0
votes
0answers
12 views
Interpolate function isn't returning values
My input DataFrame is structured like this:
I have a list of unique IDs (UIDs) which I would like to compare with each other, year on year (based on the "Age" column).
The challenge is, the ...
0
votes
1answer
34 views
What pandas function is most similar to the transpose procedure in SAS?
I am trying to transpose Amt by Txn_ID across Item_Desc in Python using pandas. Which pandas ...
0
votes
2answers
28 views
Query relating to Pandas Rows manipulation
I have a query regarding Pandas data manipulation.
Let's say I have a dataframe, df with following structure.
A B C
1 1 7
5 3 3
3 3 2
7 5 2
5 NaN 2
We have 3 ...
0
votes
1answer
25 views
Choropleth Map with plotly.express doesn't show map, just legend
I copied the "Indexing by GeoJSON Properties" example as seen on the Choropleth Maps in Python page. The code was executed with no error but there was no map showing, only the legend. In ...
3
votes
3answers
70 views
In a pandas dataframe can I convert my column values into numbers?
So I am new to all this. I was wondering in pandas can I convert my column values into numbers?
I'll try and give an example to explain what I mean
So say for example I have a column called, 'animals',...
0
votes
2answers
84 views
Problem extracting words from dataframe
I have the following dataset which is a .json file:
and I would like to get the first word for every string inside lista_asm, so I would like to get: jmp,push,...
0
votes
1answer
15 views
How to sample with guaranteed minimum spacing?
I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of ...
0
votes
0answers
5 views
Help understanding ValueError: setting an array element with a sequence [migrated]
my journey to understand the subtleties of DL in organometallic chemistry continues so, I’m sorry if this question is, let’s say, “too basic”.
I have finished the Feature Engineering part, optimizing ...
0
votes
0answers
11 views
Filtering redundant Association rules
After performing association rule mining, (using mlxtend library), I have gotten
a pandas dataframe containing all the association rules. This contains too many rules,
therefore we must reduce the ...
0
votes
1answer
17 views
Running scikit-learn with large volume
I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the ...
0
votes
1answer
44 views
How to effectively use np.where function to change NaN values in one column based on a substring in another column
I am trying to remove the NaN values from a column in a table. I used the numpy.where() function, but it just fills all the <...
0
votes
0answers
9 views
time syncronisation
Introduction
Given is a vehicle which is moving past a station. If the vehicle passes the station (for simplicity determined by GPS) it sends RF-signals, to the station saying "Hello I am here ...
0
votes
2answers
25 views
Skip NA in Mean function within Pandas agg function [closed]
Here's my working code:
new_df = old_df.groupby('store', as_index = False).agg({'reviewScore': 'mean'})
I would like to add an argument to the mean function: ...
0
votes
1answer
25 views
Train and Test Data Split for a Tweet Classification Task
I am trying to train a few Machine Learning (ML) algorithms such as SVM, NB and Random Forest to do binary classification on disaster tweets. During this project, I want to train ML algorithms for a ...
3
votes
0answers
36 views
Detecting pushups based on pose data
I've been playing with Google's MLKit, and decided to detect push ups.
As a quick test, I took the position of the left shoulder, and plotted the Y Axis. Here's how a variety of trials look:
Five ...
0
votes
2answers
35 views
Pandas loads a csv file incorrectly, but without throwing an error [closed]
So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas loads the data improperly. And it does so consistently. Let me show you ...
1
vote
0answers
34 views
Compare ranking across trials with pandas
I am carrying out several runs of an experiment with different configurations on many different inputs. The experiment has a random element and I'm trying to investigate whether this randomness ...
-1
votes
2answers
72 views
How to deal with nan values after merging / joining two dataframes? [closed]
A lot of time after merging two pandas dataframes, I end up with NaNs in the new dataframe, that's just how the way it is, because one csv does not have all the ID's that the other has (Two dataframes ...
0
votes
1answer
26 views
-1
votes
1answer
26 views
Unable to successfully merge dataframes in pandas along labels
I have two different dataframes, they both share the same labels, "Country" and "Year", I am trying to merge these together as one by these two columns.
This is my code:
joined = ...
2
votes
1answer
17 views
How to Set the Same Categorical Codes to Train and Test data? Python-Pandas
NOTE:
If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that:
You encode categorical values as ...
0
votes
1answer
44 views
How to convert DataFrame column to Rows in Python?
I have the following dataset in df_1 which I want to convert into the format of df_2. In df_2...
2
votes
2answers
230 views
How to split train/test datasets having equal classes proportion
I would like to know how I can split in an equal number the following
Target
0 1586
1 318
in order to have the same proportion of 0 and 1 classes in a ...
0
votes
1answer
12 views
How to add Error Bar to Matplotlib line plot?
I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
1
vote
3answers
35 views
How to interprete percentile information from the describe function in Pandas?
I am a bit stumped on how to interpret the percentile information you see when you call the describe function on dataframes in Pandas.
I believe I have a basic understanding of what percentile means. ...
1
vote
1answer
57 views
My corr() function in Python keeps resulting in an “ValueError: The truth value of a Series is ambiguous…” [closed]
I am a very inexperienced programmer, this is my first question on the Data Science StackExchange, I sorry if it is formatted poorly or comes across as basic. For some strange reason, in Python, ...
1
vote
0answers
19 views
PyTorch: Train without dataloader (loop trough dataframe instead)
I was wondering if it is bad practice to instead of using built in tools such as dataloader just loop trough each row in a pandas df. Lets say I am doing text classification and my training loop looks ...
1
vote
1answer
24 views
How do I control sensor data readings against (measured) outside influences?
I need some hints in this problem I have.
I have a dataset of electro-dermal activity readings that are, by nature, influenced by the movements of the person that is being surveyed. In the same ...
1
vote
0answers
416 views
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python
I have written the following code for a neural network to perform regression on a dataset, but I am getting a ValueError. I have looked up to different answers and ...
-1
votes
2answers
70 views
How do I use lagged independent variable in statsmodel OLS regression?
If there is good reason to believe that an independent variable (x) has a lagged effect on dependent ...
2
votes
1answer
41 views
How to visualise a large correlation matrix?
I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21).
Now ...
0
votes
2answers
44 views
including Zeros counts categories with pandas value_counts()
What i want is including zero counts categories while generating frequencies for categorical variables
...
0
votes
0answers
8 views
Get latest Item by Date for a Recommender System
I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would ...
0
votes
1answer
20 views
Replacing value not in list in Pandas [closed]
Say I have a data frame in Python.
If I want to replace all values in the say, the Size column that are not 'M' or 'S' or 'L' with nan, how do I do so?
Thanks in advance
0
votes
0answers
156 views
How to fix IndexError: Too many levels: Index has only 1 level, not 2 in Python
I have written the following code that has a function model_data to perform a particular set of tasks. I have to pass the list of ...
1
vote
1answer
39 views
Why we can't Remove features with missing values in Data Preprocessing [closed]
In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values ...
1
vote
1answer
15 views
How we can Identify Specific Feature from a larger amount of Dataset?
In Machine Learning, we need to play with any kind of datasets.
In the Dataset, There are too many records and features, Some datasets had lots of features (sometimes it's called ...
0
votes
3answers
490 views
How to combine and separate test and train data for data cleaning?
I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
