Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
1
vote
2answers
35 views

Update the saved model after training

Will saving a trained model this way give me a model trained on every chunk of data or just the last chunk? ...
3
votes
1answer
112 views

Data sets that have strings and numerical data all in one column [closed]

Here's a code I wrote for pre-processing a data set. It works ...
0
votes
2answers
45 views

Pandas loads a csv file incorrectly, but without throwing an error [closed]

So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas loads the data improperly. And it does so consistently. Let me show you ...
-1
votes
1answer
28 views

Unable to successfully merge dataframes in pandas along labels

I have two different dataframes, they both share the same labels, "Country" and "Year", I am trying to merge these together as one by these two columns. This is my code: joined = ...
4
votes
1answer
35 views

Spatially constrained geospatial similarity

What's the current methodology for clustering geospatial data by features? Example: I have some demographic dataset. Let's say this contains average home price and population density. So, an example ...
0
votes
1answer
300 views

How to filter rows in Python pandas dataframe with duplicate values in the columns to be filtere

Overall context: I have a data frame that contains observations for every five minute starting at 5 AM in the morning and ending at 8 PM in the evening for several days. I need to filter all the ...
0
votes
2answers
67 views

including Zeros counts categories with pandas value_counts()

What i want is including zero counts categories while generating frequencies for categorical variables ...
-1
votes
2answers
526 views

How to deal with nan values after merging / joining two dataframes? [closed]

A lot of time after merging two pandas dataframes, I end up with NaNs in the new dataframe, that's just how the way it is, because one csv does not have all the ID's that the other has (Two dataframes ...
1
vote
0answers
34 views

Compare ranking across trials with pandas

I am carrying out several runs of an experiment with different configurations on many different inputs. The experiment has a random element and I'm trying to investigate whether this randomness ...
1
vote
1answer
609 views

how to implement Excel vlookup on Python

I have a dataset A and dataset B, both data has a column in common which is the ID, i want to check if dataset A ID values can be found in dataset B ID values. Also get the records of dataset B that ...
0
votes
1answer
105 views

How to combine rows after Pandas Groupby function

May I know how to combine several rows into one single row after I used Pandas groupby function? In below example, I would like to to group the data by Employee ID, Customer Last Name and Customer ...
4
votes
1answer
332 views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
0
votes
3answers
1k views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
1
vote
2answers
62 views

How can I save a final model after training it on chunks of data?

After training a model on chunks, how can I save the final model? ...
0
votes
1answer
44 views

Plotting clustered sentences in Python

I have the following three sentences, extracted from a dataframe. I would like to check the similarity and create clusters based on their level of similarity. ...
0
votes
2answers
68 views

How to find the average over an area of center for a given radius

folks, I have an excel file which contains the lat and long values of the center of a Tropical Cyclone(TC). The excel file is as given below: Also, I have an NC(NetCDF) file which is of that of ...
0
votes
2answers
46 views

Extract seperated value from columns

I am working on this CSV file which is a collection of movie details from IMDB. In this I have a genres column in the dataframe with all the genres of the movies seperated by a pipe (|) What I need ...
0
votes
1answer
52 views

Comparing 2 columns from separate dataframes and copy some row values from one df to another if column value matches in pandas

Im relatively new to pandas andpython . I have a simple problem but I can not find the solution . I have 2 dataframes DF1 and DF2 as shown in image Both dataframes have ID coluumns. I wannt to check ...
2
votes
1answer
440 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
0
votes
1answer
60 views

How to convert DataFrame column to Rows in Python?

I have the following dataset in df_1 which I want to convert into the format of df_2. In df_2...
0
votes
1answer
2k views

Conditionally replace dataframe cells with value from another cell

I have a couple pandas dataframe questions. I would like to replace the values in only certain cells (based on a boolean condition) with a value identified from another cell. I have defined the ...
0
votes
2answers
47 views

Pandas get_dummies() rows dropping after joining back with X

I'm having an issue that I can't explain and am hoping I am missing something simple. I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning,...
0
votes
1answer
5k views

Pandas to_sql insert only some columns

I am new using pandas. I am asking about how to insert specific columns using to_sql. This is my explanation. I have a database with a table datasiswa with columns: id: as int, with autoincrement,...
2
votes
2answers
535 views

How to split train/test datasets having equal classes proportion

I would like to know how I can split in an equal number the following Target 0 1586 1 318 in order to have the same proportion of 0 and 1 classes in a ...
0
votes
1answer
46 views
1
vote
1answer
83 views

What's an appropriate datastore for variable length sequence data for PyTorch consumption?

I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats. I need a datastore that can rapidly ...
0
votes
1answer
46 views

Time series plot using netcdf data in python

I am trying to plot a time series for a 3d plot. I know that the variable must be made 1d inorder to make it viable for the time series plot. Actually I want the data within a box of latitude and ...
4
votes
3answers
6k views

Is there any use to running Pandas on Spark?

I just started working with PySpark this week, and the instance I have access two has Pandas installed. But what use is having Pandas on Spark? Isn't the whole purpose of running scripts on PySpark ...
1
vote
3answers
46 views

How to interprete percentile information from the describe function in Pandas?

I am a bit stumped on how to interpret the percentile information you see when you call the describe function on dataframes in Pandas. I believe I have a basic understanding of what percentile means. ...
0
votes
1answer
20 views

Time Series Plot for floating values

I have a Dataframe which looks as shown below I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as ...
1
vote
1answer
164 views

My corr() function in Python keeps resulting in an “ValueError: The truth value of a Series is ambiguous…” [closed]

I am a very inexperienced programmer, this is my first question on the Data Science StackExchange, I sorry if it is formatted poorly or comes across as basic. For some strange reason, in Python, ...
1
vote
0answers
23 views

PyTorch: Train without dataloader (loop trough dataframe instead)

I was wondering if it is bad practice to instead of using built in tools such as dataloader just loop trough each row in a pandas df. Lets say I am doing text classification and my training loop looks ...
1
vote
2answers
31 views

How to combine two time dependent datasets?

I'm very new to data science so please be gentle. I have a dataset that contains record of occurrence of fire for the past 35 years (+-700.000 rows). Each date and time can have more than one ...
1
vote
1answer
25 views

How do I control sensor data readings against (measured) outside influences?

I need some hints in this problem I have. I have a dataset of electro-dermal activity readings that are, by nature, influenced by the movements of the person that is being surveyed. In the same ...
2
votes
2answers
3k views

how to calculate number of datapoints within a given time interval?

I have a dataframe with one column which is a timestamp. I have converted that column to a datetime object using pandas to_datetime() method. However what I want is to ...
3
votes
1answer
498 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
1
vote
1answer
532 views

Python Pandas agg error

I am trying to generate descriptive statistics using agg function in Pandas. I am having trouble with one line with a lambda function. They work when I run them as separate lines of code, but when I ...
0
votes
1answer
23 views

How to split pieces of dataframe and create new dataframes based on it?

I have the following dataframe that I need to split bu Product and create a new dataframe for each product contain the series ds and y. How can I do that?
1
vote
1answer
8k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
0
votes
1answer
23 views

Azure Cloud SQL - Querying large number of rows with Python

I have a Python Flask application that connects to an Azure Cloud SQL Database, and uses the Pandas read_sql method with SQLAlchemy to perform a select operation on a table and load it into a ...
1
vote
1answer
360 views

How to fill missing values by looking at another row with same value in one column(or more)?

Let's say we have a 6*4 data frame in which third and fourth column contain missing value ...
2
votes
1answer
52 views

How to visualise a large correlation matrix?

I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21). Now ...
-1
votes
3answers
173 views

Pandas Groupby datetime by multiple hours [closed]

I have a log dataset which looks like the following:- ...
1
vote
2answers
124 views

How to deal with errors of defining data types in pandas' read_csv ()?

I have a table with 118,000 rows and 80 columns. I would like to select 8 columns from the table. I am reading the file using the pandas function pd.read_csv ...
14
votes
3answers
62k views

How do I merge two data frames in Python Pandas?

I have two data frames df1 and df2 and I would like to merge them into a single data frame. It is as if df1 and df2 were created by splitting a single data frame down the center vertically, like ...
1
vote
3answers
44 views

Updating 1st dataframe columns from 2nd data frame coulmns

I have two data frames df1 and df2 which look something like this. ...
1
vote
2answers
154 views

Predicting Customer Activity Absence

Could you please assist me with to following question? I have a customer activity dataframe that looks like this: It contains at least 500.000 customers and a "timeseries" of 42 months. The ones and ...

1 2 3
4
5
20