Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
1
vote
0answers
21 views

Linear Regression not working due to wrong kind of array

I try to deal with my homework. The Job is to take the following Data: http://www.statsci.org/data/general/uscrime.html and perform a linear regression on it. The code is published here: https://...
0
votes
1answer
13 views

Clustering based on missing values

I have a dataframe of the form: ...
0
votes
0answers
17 views

Long, Wide or Tidy?

I find myself converting back and forth between those formats a lot. It seems like to do time series interpolation, Tidy is best. It seems like for graphing Wide is best. Or not? Is there a best ...
0
votes
0answers
5 views

how to make contour plot centered using netcdf data

hello everyone, I am trying to produce a contour plot using contourf function in python for showing interannual monthly climatology. I do have got my plot but my concern is when I am ...
0
votes
0answers
8 views

error in pandas dataframe when trying to convert dtype object to float

I am trying to aggregate data converted from an xlsx file to a dataframe but it is all dtype object. I tried to convert it to strings and float so I can aggregate it but this is the error I get : <...
0
votes
0answers
12 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
0
votes
0answers
7 views

Coding advice/tricks to create a Seaborn heatmap for a high number of columns or most relevant only

The number of columns (more than 30) gives the Seaborn heatmap a messy appearance and impossible to glean important information from. What I am wondering is, can I map ONLY variables with the most ...
0
votes
0answers
14 views

Pandas retrieve unwanted things later [closed]

If I'm trying to make some data processing and I want to hide unwanted symbols or html codes during processing stage and I want to popup those unwanted things again on their origin place within the ...
0
votes
1answer
9 views

How to combine rows after Pandas Groupby function

May I know how to combine several rows into one single row after I used Pandas groupby function? In below example, I would like to to group the data by Employee ID, Customer Last Name and Customer ...
0
votes
1answer
27 views

find monthly climatology for a particular month

I have a 10-year sst monthly data. I want to plot the interannual monthly climatology for the month of October. By using the code I have for python with me I am able to plot the interannual monthly ...
0
votes
0answers
8 views

how to limit datetime in x axis

Hello folks, I am trying to plot three variables in a single XY plot. I am having a problem setting the x-axis limit which is for time. I tried setting limits using x_limit but getting ...
0
votes
0answers
7 views

Pandas.cut() on categorical values [migrated]

This is my dataframe ...
0
votes
0answers
3 views

Loading a Single Series from a Pickled DataFrame in Pandas [migrated]

After saving a Pandas After saving a Pandas DataFrame with df.to_pickle(file_name), it can be loaded with ...
0
votes
0answers
10 views

How do I show my pandas dataframe that I have in my jupyter notebook so that I can ask a question on stack exchange/stack overflow?

I've noticed that people produce such an amazing pandas dataframe which allows people to answer the questions. However, whenever I try to copy my table after doing ...
1
vote
1answer
39 views

Plot hurricane track

Hello folks, Recently while going through a research paper I happened to come across a plot showing the hurricane track(figure attached). Can anyone say how it is plotted or anyone has ...
0
votes
0answers
15 views

Python Pandas, how to select cases for which timestamps should be rounded?

I need to only select the cases for sensor type == air to be rounded to the nearest 5 seconds but do not how I should use a function to make this happen. I do have the following lines: ...
0
votes
0answers
26 views

Python Pandas, make date time rounding based on value in another column [migrated]

I am trying to round the timestamp of sensor data on the nearest 5 minutes based on the type of sensor. I have a column called 'sensor type' with two options: 'air' or 'sound'. For sensor type air the ...
0
votes
1answer
18 views

Converting a column to a DataTimeIndex and query by day

I've a time series dataset which I thrown into Pandas. I have later converted the Date column into a DateTime column and then transformed to an index. To here, everything is ok. Problems arise when I ...
0
votes
0answers
39 views

Python Numpy: TypeError ufunc 'isfinite' not supported for the input types

I'm new to data science, I'm using to ARIMA to analysis the time-series data of energy production. i'm following this tutorial. my code after few tweaks is: ...
0
votes
1answer
18 views

Converting a pandas dataframe into a csv with multiple columns

When I convert my pandas dataframe into a csv it all puts it in one column, I would like it to be in multiple columns. this my code ...
0
votes
0answers
12 views

I would like to merge 2 xlsx data frames - match and copy data to multiple rows [migrated]

I would like to copy a single row of dataframe2 to multiple rows in dataframe 1 based on postcode matching Example dataframe1 ...
0
votes
0answers
22 views

Parse JSON to CSV using Python

I have a request that is giving me a JSON answer: ...
1
vote
0answers
13 views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
0
votes
1answer
19 views

How to seperate text lines from txt files in python?

I am playing with a .txt file in python for EDA and I want to seperate the lines from this fashion: ...
0
votes
1answer
22 views

Plotting clustered sentences in Python

I have the following three sentences, extracted from a dataframe. I would like to check the similarity and create clusters based on their level of similarity. ...
0
votes
1answer
18 views

Split a string and create a news columns with parents

let's say I have this dataframe, and I would like to split it into several after (.) and have it like that, with levels, help please Thanks!!
0
votes
1answer
21 views

Extract seperated value from columns

I am working on this CSV file which is a collection of movie details from IMDB. In this I have a genres column in the dataframe with all the genres of the movies seperated by a pipe (|) What I need ...
0
votes
0answers
11 views

Question regarding Gaussian window function using pandas

I'm trying to aggregate my timeseries data using a rolling window. Within a particular window, I would like to use a bell curve to assign weights to individual data points. To be more specific: the ...
4
votes
1answer
23 views

Spatially constrained geospatial similarity

What's the current methodology for clustering geospatial data by features? Example: I have some demographic dataset. Let's say this contains average home price and population density. So, an example ...
0
votes
1answer
54 views

How to add a calculated column in a Pandas dataframe?

I am new to Python/Pandas so I'm struggling a bit here. I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare ...
0
votes
4answers
46 views

How to view entire 5788 rows and 7 columns.?

I want to see entire 5788 rows and 7 columns of my CSV data. I used below code and it is giving little data by stating [5788 rows x 7 columns] but showing only 10 rows and 5 columns in python prompt. ...
0
votes
2answers
58 views

How to find the average over an area of center for a given radius

folks, I have an excel file which contains the lat and long values of the center of a Tropical Cyclone(TC). The excel file is as given below: Also, I have an NC(NetCDF) file which is of that of ...
0
votes
0answers
21 views

How to clean messy column and reshape data structure in pandas

I have a dataset that looks like this. There are 122 columns of coordinates excluding the date (Time) column and 7000 rows. What I want to do is split the lat and lon into two columns, and then ...
1
vote
1answer
20 views

Set value of a column based on values of other columns (Pandas)

The last time I used Python is some time ago so some things have faded away. I have a given dataset, with multiple columns. I want to create a new column and set the values based on multiple values (...
0
votes
1answer
19 views

Bar plot with varying length

Hello folks, I am trying to plot a grouped bar plot of two variables with varying lengths, which means the x and y length of both the variables are different. The format of the data is ...
0
votes
1answer
15 views

Better way of writing following Pandas Python code

Following code is given, we need to sum up the medals. ...
0
votes
1answer
33 views

Implementing single variable Linear Regression in python

I'm trying to implement LMS algorithm in python. I have the following code: ...
0
votes
1answer
42 views

What's an appropriate datastore for variable length sequence data for PyTorch consumption?

I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats. I need a datastore that can rapidly ...
0
votes
0answers
9 views

How to define schema so that I can store sklearn object in pyspark dataframe

Using @pandas_udf(schema, PandasUDFType.GROUPED_MAP), is there any workaround to return sklearn fitted models in data frame ? Something like this - ...
0
votes
1answer
22 views

Find the time between two events by customer id [closed]

I need to find a customer has bought P1, and after how many days customer will buy P2. I am unable to find the days between order of P1 and the next order of P2 by the same customer. I have data as ...
1
vote
2answers
31 views

Sort dataframe by date column stored as string

I have dataframe named df1. I want to sort data frame month column according to month (Jan, Feb, March..). For that I used code: ...
0
votes
0answers
14 views

Weighting Value Of Timer Series Event Based On Subsequent Events

I am new to Data Science forum. I post a lot on StackOverflow, but this issue is more conceptual. I am doing analysis on time series data and weighting the value of an event based on the outcome. ...
-1
votes
0answers
9 views

how to approach Kaggle Challenge for Restaurant Revenue Prediction ? which models gives best prediction and why?

project for restaurant revenue prediction approach for 1. Random forest works the best on our prepossessed data.? 2. what is mean by the P1....P37?? 3. how feature contribute on revenue? https://www....
0
votes
1answer
28 views

Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
-1
votes
0answers
17 views

Trying to run a piece of code in Python, receiving error in Spyder IDE

I've been trying to execute the following code in Spyder IDE, but I keep getting the error: File "/Users/triveni/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py", line 1646, in ...
0
votes
0answers
15 views

How to Approach “Additivity” Analysis (Factor Analysis) for Suppliers?

I am trying to summarise an additivity value of the i-th supplier, which supplies unpredictable sets of inventory daily over a time series in Python. I have a data set of the form ...
1
vote
1answer
20 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
1
vote
0answers
24 views

Convert Pandas Dataframe with mixed datatypes to LibSVM format

I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
0
votes
1answer
37 views

How to fill missing values by looking at another row with same value in one column(or more)?

Let's say we have a 6*4 data frame in which third and fourth column contain missing value ...

1
2 3 4 5
18