People who code: we want your input. Take the Survey

Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
1
vote
2answers
7k views

How to save Tensorflow predictions to data frame?

I am new to Tensorflow. I have trained Tensorflow model, but I need to take model predictions and add them to my original test set as a column. How can I do that? ...
0
votes
0answers
12 views

Is it mandatory to have a DateTime index for a pandas dataframe to run ARIMA models on it?

I'm trying to learn about Arima models. In every tutorial, the Dataframe considered has a DateTime index. When I looked upon the mathematical formula for Arima model, it only depends on the past ...
0
votes
1answer
28 views

Getting extra characters while loading data

While loading data using read_sas from sas dataset in Python getting b' along with every string. How can I fix this?
0
votes
3answers
3k views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
0
votes
1answer
494 views

Python: How to plot time interval from a Dataframe in Pandas

I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job: ...
4
votes
3answers
997 views

Best method to deal with too many zeroes in regression problem?

I'm trying to solve a simple regression problem using TensorFlow and Pandas to see what's the expected conversion rate for a given product given how much I'm willing to pay for each visit (cost per ...
-1
votes
1answer
27 views

Having an error when converting to data format in python [closed]

I have gone through a lot of forums and discussions. However, can't get my problem solved. Note: 'Released' column has same format for all entries and there are no null values or unwanted spaces in it....
1
vote
1answer
2k views

Error with pandas dataframe (needs to be 1-dimensional)

I am trying to determine the conformal predictions for my model with my data. But it gives me following error that occurs at icp.calibrate(X_cal, y_cal) : ...
0
votes
1answer
3k views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...
4
votes
3answers
27k views

How to download a Jupyter Notebook from GitHub?

This is a fairly basic question. I am working on a data science project inside of a Pandas tutorial. I can access my Jupyter notebooks through my Anaconda installation. The only problem is that the ...
0
votes
1answer
27 views

Adding and multiplying higher values based on different columns of a dataframe

I am trying to calculate scores based on different values in the dataframe. Since these scores are based on different conditions, I am having problems where in the final calculation, which I need in a ...
2
votes
1answer
77 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
2
votes
2answers
870 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
0
votes
1answer
41 views

.1 after column name while read csv file from pandas

I am getting .1 after column name while read csv file from pandas
3
votes
1answer
144 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
2
votes
2answers
1k views

Partitioning data into features/labels and train/test after reading from csv file

I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
1
vote
0answers
18 views

How to generate data for future price predictions with Linear regression and Python

I have written the following Python code that makes predictions using linear regression model. However, I'll appreciate you pointing out what went wrong with how I generated data['X_forecast']. I'm ...
0
votes
1answer
389 views

how to limit datetime in x axis

Hello folks, I am trying to plot three variables in a single XY plot. I am having a problem setting the x-axis limit which is for time. I tried setting limits using x_limit but getting ...
1
vote
0answers
1k views

ValueError: Length of values does not match length of index [closed]

Why when I concatenate DataFrames my code shows this error? I trained a model to detect and classify 6 classes, and then store the coordinates and probabilities of these classes in a DataFrame, but I ...
8
votes
3answers
22k views

How to remove outliers using box-plot?

I have data of a metric grouped date wise. I have plotted the data, now, how do I remove the values outside the range of the boxplot (outliers)? All the ['AVG'] data is in a single column, I need it ...
0
votes
1answer
89 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
0
votes
1answer
22 views

Pandas - avoid iterrows() assembling a multi-index data frame from another time-series multi-index data frame

For analyzing pairs of currencies, I have two dataframes. The first one contains information about the symbols constituting every unique pair (as in "pair metadata"): ...
26
votes
4answers
21k views

Is pandas now faster than data.table?

Here is the GitHub link to the most recent data.table benchmark. The data.table benchmarks has not been updated since 2014. I ...
0
votes
2answers
159 views
0
votes
1answer
11 views

Find differences of two protein sequence files

I have two separate files A and B of more than 100000 records of protein sequences. Now I need to find the sequences that are in ...
11
votes
4answers
46k views

Export pandas to dictionary by combining multiple row values

I have a pandas dataframe df that looks like this ...
0
votes
2answers
29 views

Confusion between doing .copy() and using [[ ]]

I am reading "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. Sometimes he does xyz = DataFrame[["col1"]] Which ...
0
votes
1answer
62 views

Imputing missing value based on filtering result of another column [closed]

C1 C2 A x A y A z A x A NaN A x A x A x B y B y B z B y B NaN B y B x B x I have to impute missing values in C2 , the imputation should be such that if the missing values ...
1
vote
0answers
23 views

How can i find the accurcy for time series? [closed]

I have to work for the first time with time series and I have some question about this interesting field of machine learning. What I have to do: I should make a forecast for quantity for some special ...
0
votes
1answer
397 views

Is there a way to force a transformer to return a pandas dataframe?

I am having issues with scikit-learn converting dataframes to numpy arrays. For instance, the following code ...
15
votes
2answers
107k views

How to plot two columns of single DataFrame on Y axis

I have two data frames (Action, Comedy). Action contains two columns (year, rating) ratings columns contains average rating with respect to year. The Comedy data frame contains the same two columns ...
0
votes
1answer
44 views

How to Manipulate data for multiple visits per person?

I have a query to solve. I have data regarding customers and number of visits done to them. These are in two tables. So I want to join two table and create different features so that I can find better/...
10
votes
3answers
16k views

How to perform one hot encoding on multiple categorical columns

I am trying to perform one-hot encoding on some categorical columns. From the tutorial I am following, I am supposed to do LabelEncoding before One hot encoding. I have successfully performed the ...
0
votes
1answer
22 views

Split a string and create a news columns with parents

let's say I have this dataframe, and I would like to split it into several after (.) and have it like that, with levels
0
votes
1answer
14 views

How can I replace NaN values with other values in the CustomerID column if they have a same InvoiceNo?

InvoiceNo StockCode Quantity InvoiceDate UnitPrice CustomerID Country 573415 23434 20 2011-10-30 16:17:00 0.79 13607.0 UK :-------- :--------- :--------- :------------------- :--------- :---------- :...
19
votes
3answers
31k views

Pandas Dataframe to DMatrix

I am trying to run xgboost in scikit learn. And I am only using Pandas to load the data into a dataframe. How am I supposed to use pandas df with xgboost? I am confused by the DMatrix routine ...
1
vote
1answer
41 views

Normalize data from different groups

I have data that have been grouped in 27 groups by different criteria. The reason for these groupings is to show that each group has a different behavior. However, I would like to normalize everything ...
0
votes
1answer
35 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
0
votes
1answer
47 views

Read csv file and save images from the output

I have created some code that reads my CSV file and converts the dataset to a grayscale image. I want to know if there is any possible way to read through each row in the dataset and save each of the ...
0
votes
1answer
29 views

Using user defined function in groupby

I am trying to use the groupby functionality in order to do the following given this example dataframe: ...
0
votes
1answer
85 views

Applying a robust scaler to column 0 to 6 of my data frame?

I have a dataframe X and would like to scale columns 0 to 6 using robust scaler, how do i achieve this please? the following code only returns the scaled section of the data frame without the ...
0
votes
1answer
22 views

Vertical concatenation in a df based on column value_python

Is there any way to concatenate vertically a specific column from my df (concat1), considering/filtering values from another column (col_value)? My df looks like this: ...
3
votes
1answer
36 views

How to summarize multiple time series like dataset

I am running an experiment where I am sampling CPU utilization every second. A snippet of the data from each experiment run is something like this: ...
0
votes
2answers
357 views

The actual results and results from pickle files are not matching in pandas for DBSCAN clustering

I've built a DBSCAN clustering model. The output result and the result after using the pickle files are not matching. Based on HD and MC column, I am clustering WT column. ...
1
vote
0answers
21 views

Optimize Yahoo Finance Code for Analysis [closed]

I am trying to analyze a number of companies using financial data I gathered from Yahoo Finance. I am also using the yfinance API to get some more details about the company using functions. Since I am ...
1
vote
1answer
2k views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
0
votes
2answers
267 views

Error when trying .transform for OrdinalEncoder from Scikit Learn

I'm having a lot of issues using scikit learn recently and was hoping someone could help me with my problem. I can use other methods to ordinal encode but i want to figure this one out. ...
1
vote
1answer
31 views

Replace part column value with value from another column of same dataframe

I have a dataframe with two columns: Name DATE Name1 20200126 Name2 20200127 Name#DATE# 20200210 I need to replace all the #DATE# with ...
0
votes
0answers
18 views

How to determine my pacf and acf in ARIMA with daily time series data?

I have daily time series data for revenue sales ranging from 2018-04-01 to 2021-02-19. I want to implement an ARIMA model and the plot of the data looks like this: Based on my augmented-dickey fuller ...

1 2
3
4 5
21