Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
0
votes
1answer
10 views
How to load pre-processed .txt data and use it for training convolutional networks?
I have a lot of '.txt' (pre-processed) files related to C3D and I need to use this to train and test while using CNN model.
How does one go about this?
Most of data in each .txt files are formatted ...
0
votes
1answer
19 views
Normalization(minmax) gives me worse results than before in KNN, follow up actions?
Hello I'm studying a classification problem with KNN right now.
I have many numeric features that I normalized with MinMaxScaler, I also got some OHE categorical features that not seem to cause the ...
0
votes
1answer
11 views
Convert a repetitive list into Pandas Dataframe [on hold]
How do you convert something like this:
A: 1
B: 2
C: 3
###
A: 5
B: 5
C: 6
###
A: 2
B: 5
C: 7
into a dataset where the first row would be the first section with
...
1
vote
1answer
64 views
retrieve food groups from food item list
I have a dataframe of food items as follows: I have to create a food_group list that gives the food group it belongs to, for-example all type of yogurts should be in one group called yogurt.
I used ...
-1
votes
1answer
15 views
How to format strings in a pandas df column? [on hold]
How do I format all strings in a pandas df column? I need to remove all text before a comma, e.g. "Warsaw, Poland" to "Poland".
0
votes
1answer
12 views
Remove columns with a certain number of consecutive zeros
I'm using pandas and I am dealing with time series of sales.
What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...
-1
votes
0answers
27 views
How to read in and process a txt-file in Python [on hold]
I know this is a very basic question - I still hope this is the correct forum to post it:
Consider a txt-file that consists of 32 columns and a huge number of rows. Columns are separated through <...
0
votes
0answers
19 views
Add Header name without using pandas
I want to add header name without using pandas. In my program I have given two columns in my txt file where header name is missing:
...
1
vote
2answers
22 views
Combining Latitude/Longitude position into single feature
I have been playing with 2 dimensional machine learning using pandas (Trying to do something like this: https://github.com/freeman-lab/spark-ml-streaming), and I'd like to combine Lat/Long into a ...
0
votes
1answer
46 views
How to create dictionary with multiple keys from dataframe in python?
I have a pandas dataframe as follows, I want to convert it to a dictionary format with 2 keys as shown:
...
1
vote
1answer
22 views
2
votes
1answer
47 views
Delete all rows between two values in a dataframe that repeat multiple times in a column
I have dataframe and let's say inside of it is a column_A. This column_A has 3 strings as values, call them 'new_records', 'deletions', 'changes' that repeat across the dataframe multiple times in ...
0
votes
1answer
29 views
filling missing values of a column by comparing with another column [closed]
I have two columns which are heavily dependent on each other. IS_OWNER(a binary column with 1 for true and 0 for false also has some missing values which I need to fill in) and OWNERSHIP_SHARE(some ...
1
vote
0answers
37 views
Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?
It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model ...
0
votes
1answer
7 views
Groupby date and find number of occurrences of a value a in another column using pandas
My Dataset is looking like this
I would like to get the output something like this.
I need to group by date and find the occurrences if each feedback. In this case, i have 5 feedback types (namely 5,...
0
votes
1answer
26 views
Segmenting data in a Dataframe and assigning order numbers (Python using Pandas)
I have a large Excel data set which I load as Pandas. In one column I have repeating strings/numbers (the information can be a text and or integers), in another column I have a unique identifier of ...
0
votes
2answers
37 views
Replace method in pandas not giving expected result
I have data as below:
123.12.23.2
110.22.21.23
I want to mask this data as below one
1xx.xx.xx.x
So I tried below code :
...
0
votes
0answers
25 views
Pandas to_sql insert only some columns
I am new using pandas.
I am asking about how to insert specific columns using to_sql.
This is my explanation.
I have a database with a table datasiswa with columns:
id: as int, with autoincrement,...
0
votes
0answers
24 views
How to pass inputs (interactively) to a model?
Let me give you a high-level design (blueprint) of my model.
Input data::
...
0
votes
0answers
11 views
1
vote
0answers
12 views
How to concatenate many .psv files in google collaboratory?
I have a folder named 'training' in my local drive which has 20000 .psv files. I zipped it and uploaded to google collaboratory, with the upload option in the Files
section. I unzipped it with the ...
0
votes
2answers
59 views
Perform k-means clustering over multiple columns
I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not ...
1
vote
1answer
26 views
How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)
I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
1
vote
0answers
6 views
1
vote
1answer
44 views
I have this data set for crimes of a 12 month time period, over 250k rows. I want to predict future crimes by date and location
I have this 250k data set with these features
...
0
votes
2answers
21 views
How to get date columns in pandas?
I need a separate date column(in this format2018-10-08) in this table. please help.
...
0
votes
2answers
26 views
Adding a column to a dataframe in pandas using another Column
So I have a column called "plot" in a dataframe and i want to create a new one called "keywords" which only has the important words of plot.
here is the code:
...
0
votes
0answers
17 views
How to graphically present the statistical interpretations when using Kolmogorov–Smirnov test (K-S) test?
I am referring to my previous question (Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?) related to visualising the Pandas dataframes to show that they have ...
1
vote
1answer
49 views
How to add date column in python pandas dataframe
I want to add a date column (from 1/1/1979 upto the data is) in pandas data frame.
Currently, my data frame looks like this:
...
1
vote
1answer
54 views
Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?
I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same ...
2
votes
1answer
71 views
How to penalize for empty fields in a DataFrame?
I have to calculate the consistency of racing car drivers during the whole season. My DataFrame consists of 10 columns (10 circuit names) and for each of those columns I have the standard deviation ...
1
vote
0answers
20 views
How to analyse data after applying pandas' groupby function? [migrated]
I have a data set of Olympic games medal winners. I am trying to find the country with most medals. How do I go about working with the series after applying groupby function?
Here is my data frame.
...
0
votes
1answer
38 views
compare pandas dataframes with different lengths
I have two dataframes of different lengths and I need to add a column to the first one with filtered values, e.g.
...
0
votes
0answers
15 views
Improving pandas speed for deriving data from other datasets
I don't want to filter the set I'm starting with or perform mathematical operations on it. Instead, I want to count the number of entries which appear in another set based on the observations here:
...
4
votes
1answer
25 views
How to use a one-hot encoded nominal feature in a classifier in Scikit Learn?
Im working on a genre classification problem on a songs dataset. Since genre is a nominal feature, I used sklearn's LabelBinarizer to get the one-hot encoding for this feature for every row in the ...
2
votes
2answers
50 views
sklearn and pandas in AWS Lambda
I made a front end where I would like to make REST calls to an AWS Lambda interfaced with AWS API Gateway.
I dumped my model as a pickle file (and so my encoders) which I initially trained locally. ...
0
votes
1answer
19 views
1
vote
2answers
55 views
Low accuracy in multi-class classification despite all data being generated from rules
I have a well defined data where i have cleaned up my data to final form which has 20 features mapping to a number between 1 to 100. Upto 5 features are enabled(value set to 1) for each row. The data ...
1
vote
1answer
30 views
How to deal with memory insufficient read by pandas in python
I use pandas.read_csv to read a huge file for machine learning, but I got memory error.
Someone recommend me to set arg ...
0
votes
1answer
107 views
Multiple filtering pandas columns based on values in another column
I have a pandas dataframe df1:
Now, I want to filter the rows in df1 based on unique combinations of ...
0
votes
3answers
31 views
1
vote
1answer
14 views
does index of my data which is of type “Date time index” plays a part in reggression?
I'm new to data science and I'm working on a regression problem. My question is the index of my data which is of type "Date time index" plays a part in regression? I mean is it Okay if i drop the ...
0
votes
1answer
14 views
My small script on value alteration in columns of a data not working
I have a data set which has "Speed" as one of the columns (features). The column contains both zero and non-zero values. I want to randomly set 10% of the non-zero values to zeros. This will change ...
1
vote
1answer
17 views
Manipulating multi-indices for a pandas dataframe
I have a pandas dataframe with multi-index. I have couple of questions on this.
The indices are week numbers (38 to 42) and for each week, day of the week (DOW). So it looks like
The problem is ...
1
vote
0answers
23 views
Lookup from time series panda dataframe
I have below reference table with date time series defined in 20 mins interval
...
0
votes
1answer
42 views
How to create new feature based on clustering result
I'm still a beginner in machine learning and I want to know how to code this situation based on python and machine learning (clustering).
I have data like:
...
0
votes
1answer
34 views
How to transform entire pandas data frame in one hot representation?
I want all the columns one hot encoded without the need of listing out the columns or apply one hot encode one by one. I know how to do it one column then another.
0
votes
1answer
30 views
Using pandas get_dummies() on real world unseen data
I made a ML model, trained and tested it with my data containing categorical variables.
To create dummy variables I used pd.get_dummies() before the split.
I now ...
2
votes
1answer
108 views
Pandas merge column duplicate and sum value
How to merge duplicate column and sum their value?
What I have
A 30
A 40
B 50
What I need
A 70
B 50
DF for this ...
0
votes
0answers
7 views
Is there a well-recognized way to map specific events (with a timestamp/s) to a pandas dataframe with a timeseries index?
So let's say I have this dataframe:
...
