pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
0
votes
1answer
7 views
Down-sampling the data
I'm trying to do a regression analysis with two different datasets. One of them has 965 samples and the other 2275. I wish to downsample the latter to 965. Please suggest ways to do so.
Thank you!
1
vote
1answer
15 views
Partitioning data into features/labels and train/test after reading from csv file
I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
0
votes
0answers
18 views
How to change a field in a Dataframe based on the values in another Dataframe in python [migrated]
A change at work has meant that our old department numbers will no longer be used and we will be using new department numbers in the next month. However; we still need to keep track of the old ...
0
votes
0answers
26 views
Timeseries sensor data for one process regression modeling
The dataset:
I have a dataset containing the data of a manufactoring process. The dataset contains the process ID ("Sarzs_no"), the ID of the manufactoring machine ("Unit"), the data logged by two ...
0
votes
0answers
23 views
fit distribution to the data
I want to fit a distribution that is as close as possible to my data. The data is regarding the number of events that arrive in a time period. In theory, Poisson distribution is used for similar cases,...
0
votes
1answer
29 views
Dataframe looks the same but the structure is different when loop
I am generating a dataframe from a JSON file, this JSON file can come from 2 different sources, so the internal structure is slightly different, so what I am doing is first detecting the source and ...
1
vote
0answers
7 views
how to improve searching index in dataframe
Given a pandas dataframe with a timestamp index, sorted.
I have a label and I need to find the closest index to that label.
Also, I need to find a smaller timestamp, so the search should be computed ...
-1
votes
0answers
18 views
Does feature occurrences affect feature importance
Let’s say we have a data frame that looks like this: Df:
A B
0 45 78
1 5 34
2 3 0
3 56 0
4 34 0
5 23 0
Does the fact that we have more values ...
0
votes
1answer
13 views
lengthy criteria in dataframe selector
I'd like to get records with country codes not in a long list , something along the lines of
...
1
vote
0answers
20 views
Create a new row when a character exists in Python Dataframe
I have an excel sheet which look like this:
ColumnA ColumnB ColumnC
a 1|2|3 b
r 4|5 d
b 6|7|8|9|10 z
When columnB ...
0
votes
1answer
18 views
Aggregating small values in a frequency bar plot
I have a pandas Series of sorted percentage values like this :
...
3
votes
1answer
19 views
How to read html tables under multiple headers and combine them in a single pandas dataframe?
I have an html file like this:
...
-1
votes
1answer
18 views
Drop Duplicate but conserve data in other columns with pandas
I'm trying to use the pandas drop_duplicates method and I'm wondering if I have a table of this form
...
0
votes
1answer
11 views
Fill missing values AND normalise
I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.)
For example
...
0
votes
0answers
32 views
Aggregating a pandas dataframe using groupby, then using apply… but how to then add the output back into original dataframe? [migrated]
I have some data with 4 features of interest: account_id, location_id, date_from and ...
0
votes
1answer
32 views
How to populate pandas series w/ values from another df? [closed]
I need help figuring out how to populate a series of one dataframe w/ specific values from another dataframe.
Here's a sample of what I'm working with:
...
0
votes
1answer
15 views
Dealing with a dataset having target values on different scales?
I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
0
votes
2answers
37 views
How to get windspeed when temperature is maximum for each city? [closed]
Here is the sample dataset, I have Weather_Data,
I want to calculate
"What is the windspeed when temperature is maximum for each city"
I have tried following code:...
-1
votes
2answers
51 views
Data Cleansing - Handling CSV files
I would like to hear some views on a problem I have with my dataset (which I presume to be a common one).
Let's say I have the following dataset
...
0
votes
3answers
42 views
How to download a Jupyter Notebook from GitHub? [closed]
This is a fairly basic question. I am trying to work through a Pandas Tutorial presented on a Jupyter Notebook. I can access my Jupyter notebook through my Anaconda Installation, but my question is, ...
0
votes
2answers
54 views
print a specific column with a condition using pandas
I have a data set which contains 5 columns, I want to print the content of a column called 'CONTENT' only when the column 'CLASS' equals one.
I know that using ...
30
votes
7answers
12k views
Why do people prefer Pandas to SQL?
I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle.
The vast majority of the operations I've seen done ...
0
votes
0answers
44 views
Encode Presence/Absence as Binary Columns
I have a large tsv file of phonemes, where each row is a phoneme and some of the columns are "LanguageCode", "Phoneme", etc. Here is a toy example:
...
0
votes
1answer
33 views
Need help on Time Series ARIMA Model
I'm working on forecasting daily volumes and have used time series model
to check for data stationarity. However, I'm strugging at forecasting data with 90% accuracy. Right now variation is extremely ...
0
votes
1answer
26 views
How to use features when predicting aggregation?
Let's say I have to predict the total monthly sales of a store. I have the data in the following format:
I don't think I can use this data directly for predicting the sales for a month, as in it's ...
0
votes
1answer
53 views
Pre-processing irregular, high frequency time-series data in python
...posted originally in StackOverflow (might be better suited here)
Small Picture:
I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data ...
0
votes
1answer
17 views
Giving a score (from 1 - 5) using weights
I have a dataset with 8 features (columns). Each row is a customer, for which I want to give a score from 1-5 using the 8 features for each customer. The range of values are as follows:
Feature 1: 0-...
2
votes
2answers
65 views
panda grouping by month with transpose
Based on the following dataframe, I am trying to create a grouping by month, type and text,
I think I am close to what I want, however I am unable to group by month the way I want, so I have to use
...
0
votes
2answers
174 views
Scatter plot for binary class dataset with two features in python
I have my dataset that has multiple features and based on that the dependent variable is defined to be 0 or 1.
I want to get a scatter plot such that all my positive examples are marked with 'o' and ...
-3
votes
2answers
102 views
Get top 5 performers from team using pandas [closed]
I am new to both python and pandas, numpy etc. I am trying to do some data analysis on batsmen performances for a given team. I have successfully written python code to scrape data and store it in ...
0
votes
0answers
147 views
Remove Local Outliers from Dataframe using pandas
Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe....
1
vote
0answers
54 views
Cannot feed appropriate datatype to model from CSV
DESCRIPTION: When training my model and 3/4 of one epoch completes I get an error "ValueError: could not convert string to float" by running dataset( see below ) I can see the first column is int64 ...
2
votes
0answers
17 views
How to prevent pd.ewma from changing the index?
I am resampling my time series data using the following code:
bars = transactions.price.resample('4H', base = 3).ohlc()
This allows me to account for DST ...
0
votes
0answers
143 views
how to resolve this error “Grouper for <variable> not 1-dimensional”?
The below-written code gives an error. The variables categorical and category are a list of column names I need to iterate through.
...
1
vote
1answer
43 views
Is there an analog to SQL's STRING_AGG (or FOR XML PATH) function in Python?
Asked this in SE but maybe this is too data-oriented so trying to post it here. I am trying to find the analog to the SQL function STRING_AGG so that I can concatenate across columns and rows of a ...
0
votes
1answer
162 views
how to export the tables into a csv file pandas
The following is a piece of code I wrote to create a pivot table for categorical vs continuous variable.
...
0
votes
2answers
190 views
ValueError: not enough values to unpack (expected 4, got 2)
I have written this code
fig, (axis1, axis2,axis3, axis4)=plt.subplots(2,2,figsize=(10,4))
and I am getting this error
...
0
votes
1answer
52 views
Vertical and horizontal lines appearing on large confusion matrix?
I have produced a large heatmap-like confusion matrix and am seeing horizontal and vertical lines on it, so I'm trying to determine:
What they mean
Why they are there
How I can improve on this
...
1
vote
1answer
47 views
Predicting Customer Activity Absence
Could you please assist me with to following question?
I have a customer activity dataframe that looks like this:
It contains at least 500.000 customers and a "timeseries" of 42 months. The ones and ...
0
votes
1answer
224 views
How to create a historical timeline using Pandas Dataframe and matplotlib
So I've seen a few answers on here that helped a bit, but my dataset is larger than the ones that have been answered previously. To give a sense of what I'm working with, here's a link to the full ...
0
votes
1answer
129 views
What’s the best way to save many pandas dataframes together?
I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted,...
1
vote
4answers
1k views
How do I compare columns in different data frames?
I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
1
vote
1answer
250 views
How do I properly use the pandas_datareader package
I am trying to do a basic project where I grab some data from Morningstar or Google Finance but when I import the package according to the Usage instructions on GitHub and run Python in Pycharm, it ...
0
votes
1answer
450 views
How to deal with TypeError: ufunc 'isnan' not supported for the input types [closed]
I have dealt with all the Nan values in the features dataframe, then why I am still getting this error?
...
1
vote
1answer
27 views
Dropping less frequently used categorical data?
I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite ...
-1
votes
3answers
60 views
Combine Pandas DataFrames with year columns
I'm struggling to find the most efficient way to combine multiple dataframes with columns that are years and country names for the index. As an example:
GDP.csv
...
0
votes
2answers
33 views
Stata-style replace in Python
In Stata, I can perform a conditional replace using the following code:
replace target_var = new_value if condition_var1 == x & condition_var2 == y
What's ...
2
votes
2answers
173 views
Tidy data in panda dataframe
There is a quite famous article by H. Wickham, Tidy data, where he defines a certain type of cleaned data and calls it (dataframe-)tidy, and illustrates in on several example using R. At the end, he ...
1
vote
3answers
44 views
Need help in understanding a small example
Pardon me, I agree the title of the question is not clear. I would like to know the understanding of below steps which are picked from the textbook "Hands on machine learning".
...
0
votes
1answer
25 views
Why is it important to have sufficient number of instances in your dataset for each stratum?
As per the figure 1, most of the median-income values are clustered around \$20,000-\$50,000, but some median incomes go far beyond \$60,000.
I didn't understand the explanation behind why housing['...
