Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
2
votes
1answer
21 views

how is validator created in python and what are most popular libraries / modules to learn first

I have a df which has a serial number generated with each new record. The serial number combines with some other part like state code, year of registration and ...
0
votes
1answer
15 views

Where are the predicted results located in this model?

I am new at this and am pretty sure this is a stupid question, but here it goes: Where can I see the results of a model's prediction? I did this course about deep learning, followed the tutorial, ran ...
1
vote
0answers
26 views

How to make the values in different columns in the correct order based on another data frame (mapping) in Python Pandas

I am pretty new to Python and Pandas and I struggle with combining a messy dataframe from excel with a mapping. I have tried to find some solutions on the Internet, however with no success. My first ...
0
votes
0answers
16 views

Calculating all near-tangential lines for inflection points of a time series

I have been thinking about this problem for a while and I'm curious if anyone knows of a good paper on this, has any ideas for algorithms or improvements to the framework. The task is to store the ...
1
vote
0answers
91 views

CDF plot overlay histogram in python

I have a dataframe called df['ProgressStep'] I would like to get overlaid CDF plot in histogram. Have tried 2 methods, neither one meet my target perfectly. please help to fine tune the code, either ...
3
votes
1answer
48 views

Using output of ML algorithms as an input to different ML algorithm (Ensemble Learning)

I want to assign weights to multiple models and make an single ensemble model. I want to use my outputs as the input to a new machine learning algorithm and the algorithm will learn the correct ...
1
vote
0answers
33 views

Numpy-like reduction of particular dimensions in Pandas DataFrame?

With a numpy multidimensional arrays, it is easy to reduce particular axes with functions such as np.mean or np.sum. For example,...
0
votes
0answers
16 views

Best way to store large amounts of time series data? Relational database (SQL) or NoSQL route. Also additional python/pandas question inside

so I'm working on a project and I'm sort of stuck as to how to store my data. I have a concept I want to propose but am unsure whether it is possible, if it is not I would appreciate any help ...
0
votes
0answers
10 views

pandas reduce time for na checks (is_valid_na_for_dtype / is_na)

I have a small project that I iterate over a dataframe and do stuff with this data.. I know that if I can I should avoid iterate the rows but I'm afraid that I must so I use itertuples. Now I want to ...
0
votes
1answer
30 views

How to get column names after One Hot Encoding when using Pipelines?

I am using Pipeline and ColumnTransformer to preprocess the data. Basically I am using them to impute null values, scale the ...
1
vote
0answers
10 views

How to compute similarity matrix for strings efficiently? [duplicate]

Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
0
votes
0answers
13 views

How to merge one table to another by multiple columns?

I have a table that looks like this: prodh1 prodh2 prodh3 prodh4 prodh5 prodh6 7 77XS 77XSBRC 77XSBRCBRC1 77XSBRCBRC17S0C 77XSBRCBRC17S0C70R 7 77XA 77XA7TR 77XA7TRPRC1 77XA7TRPRC17S0C ...
0
votes
2answers
28 views

Compare 2 Pandas Column, over different rows and Output the resultant in a 3rd column

I have a data frame like this. EMPLOYEE_ID NAME MANAGER_EMPLOYEE_ID 0 42 S 40 1 40 G NaN 2 T M 40 3 0c H 42 I want a separate column that outputs the Name of the Manager of every person. So for ...
0
votes
2answers
110 views

Fastest way to replace a value in a pandas DataFrame?

I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
1
vote
1answer
16 views

Why does isna()/isnull() function show different values compared to value_counts()?

I am quite new to Machine Learning/Data science so I started doing my first Kaggle competition problem from here: https://www.kaggle.com/c/house-prices-advanced-regression-techniques I was cleaning ...
0
votes
0answers
85 views

Preserve column order after ColumTransformer

I'm using Pipeline and ColumnTransformer modules from sklearn library to perform feature ...
2
votes
2answers
352 views

How to convert mixed datetime formats into single one in pandas?

I am working with DataFrame which contains multiple datetime formats in one column. For example: ...
1
vote
1answer
21 views

Removing features that correlate with the target label

I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
0
votes
1answer
45 views

How to do EDA on large datasets

I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory. ...
0
votes
2answers
19 views

What is '[0]'in boxcox transformation with scipy.stats.boxcox?

While performing a boxcox transformation on a series of data, I noticed a certain [0], which I don't quite understand
0
votes
0answers
17 views

formatting data from pandas into CNN-compatible array

I have a pandas dataframe of this form: time x y 0 a b 1 c d For my CNN model training, I want to slice the first 5 rows (from t=0 to t=4) and apply my custom function to label this sliced set, ...
0
votes
1answer
17 views

Select current column when iterating dataframe [closed]

I have a dataframe with 1k+ columns like this(i cut it). I want to iterate the dataframe and check row by row if value is True then assign it the ...
0
votes
0answers
28 views

Is there row limit when saving pandas dataframe as hdf5?

I am making a dataframe with more than 100 million rows while preprocessing the data. csv file knows that there is a limit on the number of rows, so I want to save it in a different format. I'm trying ...
0
votes
1answer
223 views

Create a tooltip for each dot on a scatter plot created for a Pandas dataframe

I have a pandas based scatter plot that is created from a dataframe, with the following code: ...
1
vote
0answers
13 views

Using Quarterly Percent Change as Validation

this is a newbie question but it's very simple so I'm hoping it's okay to ask. I would like to calculate the percent change from one quarter to the next using Pandas and use that as my Y-Test (...
0
votes
1answer
100 views

Error reproducing Eland Demo Notebook

I was trying to reproduce Python Elasticsearch client Eland notebook, available in Eland documentation. Here are the first lines of code for producing an eland dataframe from my local Elasticsearch ...
0
votes
1answer
22 views
0
votes
0answers
12 views

Model on non-iid data performing badly

I am on this lecture about non-iid data where we generated a timeseries data using the function below: ...
-1
votes
1answer
53 views

Anomaly Detection and Removal/Interpolate [closed]

I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection ...
1
vote
1answer
18 views

How to move column in rows direction in Pandas DataFrame?

For example, I want get right from left DataFrame like on picture (move column B on 2 steps down): PS. "Na" is not necessary, it can be any controlled value like null, zero or empty string.
0
votes
1answer
284 views

Replacing rows of dataframe with rows of another dataframe that have the same index

I have a dataframe that has rows with indices 0 to 128 and a smaller dataframe with indices 4, 8, 105, and 107. I made edits to the rows in the smaller dataframe and am now trying to replace rows ...
1
vote
2answers
44 views

Handling conflicting cases pandas python

I have a data set where some rows are same but belong to different classes. Example - index Heading 1 Heading 2 Heading 1b Heading 2b Class/Target row -1 a b c d 0 row -2 t r f k 0 row -3 m u p l ...
0
votes
1answer
24 views

Extra feature on test set

Suppose I convert categorical data into dummy variables with get_dummies and I get these columns in the training dataset: x_A x_B x_C 0 1 0 0 0 1 1 1 0 But in ...
0
votes
1answer
213 views

OneHotEncoder Showing error while encoding two columns

I am encoding two columns with index number 1 and 2 that is column number 2 and 3 , using the following code, however I am facing error of invalid syntax : If I am using only 1 in the index value it ...
0
votes
1answer
127 views

Separate numerical and categorical variables

I have a dataset (42000, 10) which contains 7 categorical features and 3 numerical. I would like to separate both the numerical and categorical features into 2 different data frames i.e I would like 2 ...
0
votes
1answer
56 views

Concat function increases row values and returns df with null values

I am trying to one hot encode my train and test dataset. For my train dataset, I have 2 dataframes with different number of columns but same number of rows.A (with encoded features) = (34164, 293) and ...
1
vote
3answers
103 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
0
votes
2answers
37 views

Assign value of existing column to new columns in pandas based on conditions [closed]

I am trying to create a new column in pandas data frame based on existing columns. column1 column2 column3 y1 y2 y3 100 200 300 2020 2021 2022 100 200 300 2021 2022 2023 100 200 300 2019 2020 2021 ...
0
votes
1answer
95 views

Convert any format of phone number to (111) 111-1111 using python [closed]

I have a phone number column in my dataset which has numbers in different formats. Here are some examples: ...
2
votes
1answer
26 views

Discrepency in 25th percentile of standard deviation calculation

For each of the following sample sizes [3,5,7,9], I Need to calculate the 25th percentile of the standard deviation for the values sampled. And the total number of trials will be 10. DataFrame ...
0
votes
1answer
48 views

sklearn text analysis - dealing with missing values

I'm working on a multi-class text classification project. My goal is simple: given a "bug", I'd like to predict to which final group owner it will be assigned to. I was able to achieve ~...
1
vote
2answers
1k views

Check if a particular value is a datetime and assigning a particular column value in pandas

I have a pandas data frame that contains a partially corrupted data field as below. It has numbers (which are not a date) or nans. The real data frame has an incredibly large number of rows as well. I ...
1
vote
1answer
94 views

Returning a DataFrama with nlargest values based on a particular column

This is my sample DataFrame: ...
0
votes
1answer
30 views

How can I calculate total days past due between billing events?

I am dealing with a dataframe with subscription events partitioned by username, subscription status, and relative timestamps. For each of the dates, there are changes in time when the subscription ...
0
votes
1answer
757 views

Bar plot not showing up in Jupyter Notebook

I am trying to create a bar plot for a Pandas Series and the bar plot is not showing up in Jupyter notebook. When I run the cell, I only get the following and I do not see the bar plot. ...
1
vote
1answer
44 views

How to convert float type nan in a dictionary value to 0.50?

I have a dictionary as below: { '$175000-199999': nan, '$698506': nan } I want to convert the nan to 0.50. I tried using dictionary comprehensions ...
0
votes
0answers
29 views

Binning the data of one cell in a data frame using python

I have a dataframe as below: class 1 class 2 0 M NaN 1 17245 NaN 2 graduate Bach I want to use pd.cut() function and bin the data 17245 to 15000-25000. How do I do it?
0
votes
1answer
29 views

What is the fastest way to run predictions on a big dataset using pandas, numpy and keras?

I am a bit new to the field of data science and could really use some help. I used a natural language dictionary to train and test an ml model using keras and tensorflow. It detects the sentiments in ...
1
vote
2answers
35 views

How to convert a dataframe into a single dictionary that is not nested?

I have a dataframe as below: ...
0
votes
0answers
35 views

How to adjust the unadjusted charts in Pandas

I would like to adjust the unadjusted charts in Pandas. Suppose I have a data frame of the price of a stock symbol that there is a price gap between two candlesticks due to a capital increase. Sites ...

1
2
3 4 5
23