Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
0
votes
1answer
12 views
Combine results from multiple models
I am using chunks of 100000 rows at a time from the CSV file to train the a simple LASSO model.
How do i combine all of these models trained from these different chunks? I would like to use all ...
0
votes
1answer
8 views
How can I merge 2+ DataFrame objects without duplicating column names?
This is for work. TLDR: Bottom-line question at the bottom.
I am gathering and parsing test results produced by an old test setup whose output formatting is not likely to change anytime soon. I've ...
0
votes
0answers
6 views
how assign header by other column header and concate header and first row in pandas?
i have a data like this
my desire format is this
how can i do this?
1
vote
1answer
19 views
Mapping column values of one DataFrame to another DataFrame using a key with different header names
I have two data frames df1 and df2 which look something like this.
...
1
vote
2answers
31 views
0
votes
1answer
26 views
pandas: How to impute the categorical column by the nearest neighbors?
I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...
-1
votes
0answers
17 views
how to repeat some row data in python? [duplicate]
i have a dataset like this
i want to change first column of data and my desire data frame is like this
how can i do this?
-2
votes
0answers
32 views
how to handle rows and columns in python? [closed]
I have a dataset which has large data and is like:
I tried to replace these number of first columns by their respective header but I did get a solution. I want this table in format:
-1
votes
2answers
37 views
how to transformation of row to column and column to row in python pandas? [closed]
I have a large dataset
I want to transform this dataset into this format
I have try it through transpose but i couldn't figure out
0
votes
3answers
44 views
How to access substrings in pandas column and store it into new columns?
I'm working on a dataset for building permits. In the dataset there is a column that gives the location (lattitude and longitude) for the building permit. The data in the location column look like ...
0
votes
3answers
58 views
How to replace a part string value of a column using another column
How to replace a part string value of a column using another column.
My DataSet here is :
...
0
votes
1answer
31 views
Missing Values in Classification
I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't.
I also have ...
0
votes
1answer
26 views
Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?
I am running the Kaggle Video Games sales dataset through an XGboost algo.
I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5
when I use:
data = pd.get_dummies(data=...
1
vote
1answer
25 views
Pandas - how can I calculate a field to count the number of occurences that values in the ID field appear in sequential order
I know how to read in the data frame in Pandas and do the basic manipulation, but how do I populate the order column based on the ID column? For instance, if bike occurs twice such as seen below, how ...
1
vote
0answers
30 views
Improving population weighting
Bit of a noob in this stats world, so apologies in advance for any naiveté. I did a fair bit of stats long ago in college but it's a distant memory, so please assume little knowledge!
The dataset I'...
2
votes
2answers
46 views
0
votes
0answers
18 views
Chrome OS and Data Science
At this moment in time is there sufficient data science, data engineering, visualization and machine learning/AI software available to justify buying a Chrome OS box like HP G2? Is the Chrome Linux ...
0
votes
0answers
24 views
Working with many unique values in a column
I have only learnt basics of data science about 4 months ago. And now I am working on project.
This is my code: https://pastebin.com/er1EA7pZ
This is my data: https://www.kaggle.com/grosvenpaul/...
1
vote
0answers
22 views
How to read first row data as start time , end of the day last row as end time and continue to next day import from csv file using python
Here I have time series with date from csv file. So in time column first time I want to read it as start time and it will continue with one hour delta time till to end of the day time. Then again it ...
1
vote
1answer
25 views
How to integrate nvidia gpu with jupyter notebook for scikit-learn
I know there are answers saying that tells how to integrate/use tensorflow or for deep learning libraries. I want to use it for classification or regression scikit-learn libraries. Is it possible. If ...
2
votes
2answers
24 views
How to get a count the number of observations for each year with a Pandas datetime column?
I'm working on a column that I converted from a object to a datetime datatype in Pandas. I'm trying to get a count of how many of observations are there for each year.
This is the column: ...
2
votes
1answer
15 views
A single column has many values per row, separated by a comma. How to create an individual column for each of these?
As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
-1
votes
0answers
5 views
replace missing value with n-1 [migrated]
for example : I have, df[0]= 420, NaN, 455; NaN,NaN, NaN, then using :
...
1
vote
2answers
39 views
Error in using fit() on RandomForest Classifier where X was a pandas.DataFRame object
On using fit() method on sklearn.ensemble.RandomForestClassifier I am getting a value error that says.
ValueError: could not convert string to float: 'male'
The ...
1
vote
1answer
20 views
shifting the last column in the dataframe to the first place [closed]
While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other ...
0
votes
1answer
42 views
Resampling pandas Dataframe keeping other columns
I'm facing a problem with a pandas dataframe. Actually my Dataframe contains 3 columns: DATE_TIME, SITE_NB, VALUE.
For some SITE_NB there are missing rows. For example:
DATE_TIME;SITE_NB; VALUE
2011-...
2
votes
1answer
27 views
Adding a new custom column in a python data frame [closed]
I need to add a new column in a python data frame that has the dates of January in a column. Each date is repeated 24 times in the column without any intervention. The total number of entries will ...
1
vote
3answers
31 views
Data aggregation and split train test samples
I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....
0
votes
0answers
24 views
How to read line delimited json chunk by chunk into pandas dataframe?
I have many very large line delimited json files. I know that I can use pandas json_read to setup an interator that will give me the file chunk by chunk.
But I do not have a clue how I can actually ...
-1
votes
1answer
52 views
How do I add a column to a Pandas dataframe based on other rows and columns in the dataframe? [closed]
I've tried a lot of different methods, but I can't seem to find the right way to do this. I want to create a new column based on the time and ...
0
votes
0answers
18 views
getting zero values after applying cat.codes on pandas dataframe
I am using a pandas dataframe to process live network data. I first retrieve the network data using the python library scapy and then put the data into a pandas dataframe. I change the columns to "...
1
vote
0answers
30 views
Converting time series data to MNIST like data? [closed]
There are some time series data like this:
...
1
vote
1answer
14 views
How do I assign a title to each of the histograms I have posted at the same time? [closed]
fig, axes = plt.subplots(1,7, sharey=True)
df2.loc[df2.country == 'BR','date_created'].hist(xrot=90,bins=30,ax=axes[0],figsize=(20,4));
df2.loc[df2.country == 'CH','date_created'].hist(xrot=90,bins=...
1
vote
1answer
49 views
Value extraction from a python dataframe [ problem statement specific ]
This is an update on one of the older posts of a friend of mine (That post stopped getting responses as I got a partial solution to my problem) and I'm reworking that problem statement. Link to that ...
4
votes
2answers
60 views
Binary text classification with TfidfVectorizer gives ValueError: setting an array element with a sequence
I am using pandas and scikti-learn to do binary text classification using text features encoded using TfidfVectorizer on a DataFrame. Here is some dummy code that illustrates what I'm doing:
...
1
vote
1answer
28 views
Appending DataFrames to lists in a dictionary - why does it seem like the list is being referenced by each new DataFrame?
I have a DataFrame that pairs one or more labels to a sample group and id, for a given sample stored in a database at SampleGroup/SampleID:
There are ~100 labels. I want to create binary models to do ...
4
votes
2answers
58 views
Pandas vs Linux Datascience [closed]
I joined a data science learning community in my college and we are using linux terminal commands and awk commands to practice gathering some information from big datasets stored in csv files. About ...
2
votes
1answer
165 views
Display Images (url) Inside Pandas Dataframe
I would like to display images (mostly jpg and png formats) directly from their url link inside a pandas dataframe. Imagine I already have the following dataframe:
...
1
vote
2answers
22 views
plot multiple strings from a dataframe
I am trying to read and plot several files which looks like as below when I open with python using pandas read.table. The columns to plot are "dev" vs timestamp.
The timestamp should be compose from ...
5
votes
1answer
351 views
Difference between isna() and isnull() in pandas
I have been using pandas for quite some time but i do not understand what is the difference between isna() and isnull() in ...
1
vote
2answers
77 views
How to save Tensorflow predictions to data frame?
I am new to Tensorflow. I have trained Tensorflow model, but I need to take model predictions and add them to my original test set as a column. How can I do that?
...
2
votes
1answer
49 views
ML Models: How to handle categorical feature with over 1000 unique values
I am trying to build an ML Classification model on a data set that contains quite a few categorical columns. However, few of them have over 1000 unique values. I am concerned that if I run one-hot ...
1
vote
0answers
15 views
Spearmanr on two pandas dataframes
I have two pandas dataframes that look about the same but with different information stored in them. My question will be about how to compare the two dataframes to ensure column and row match before ...
2
votes
1answer
27 views
How to remove rows from a data frame that are identical to other df?
I have two data frames.
So how do I remove rows that have identical email addresses in df2 from df1?
...
1
vote
1answer
38 views
Partitioning data into features/labels and train/test after reading from csv file
I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
0
votes
0answers
45 views
Timeseries sensor data for one process regression modeling
The dataset:
I have a dataset containing the data of a manufactoring process. The dataset contains the process ID ("Sarzs_no"), the ID of the manufactoring machine ("Unit"), the data logged by two ...
0
votes
0answers
26 views
fit distribution to the data
I want to fit a distribution that is as close as possible to my data. The data is regarding the number of events that arrive in a time period. In theory, Poisson distribution is used for similar cases,...
0
votes
1answer
33 views
Dataframe looks the same but the structure is different when loop
I am generating a dataframe from a JSON file, this JSON file can come from 2 different sources, so the internal structure is slightly different, so what I am doing is first detecting the source and ...
2
votes
1answer
35 views
how to improve searching index in dataframe
Given a pandas dataframe with a timestamp index, sorted.
I have a label and I need to find the closest index to that label.
Also, I need to find a smaller timestamp, so the search should be computed ...
0
votes
1answer
14 views
lengthy criteria in dataframe selector
I'd like to get records with country codes not in a long list , something along the lines of
...
