Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

Pandas nested/recursive groupby count [on hold]

I have a DataFrame with more than 2 million rows, that looks like this: ...
0
votes
0answers
22 views

Find the most relevant columns for each single class in pandas

The following question (this one) did not help me. I have a big dataset, and I want to know which Columns are the most relevant for the Target Variable. I know that, in my case, for each class in the ...
0
votes
1answer
22 views

Gradient decent in Python

I just finished working on my first machine learning algorithm i.e Linear regression. I want to reduce the rmse by optimising the model. I found out that gradient decent does the same job. But i dont ...
0
votes
1answer
32 views

Sublime Text 2 with Pandas for Excel (Combining Data) & Data Visualization

I'm new to programming with Python, and so far it's been a headache to create a build environment- and need your support and expertise in this area. Background I'm running a MacBook, and using ...
0
votes
1answer
25 views

General approach on time series for customer retention/churn in retail

I have a time series of data in the following form: ...
0
votes
1answer
15 views

Interpreting DataFrame.where() documentation

From examples outside of the documentation, I thought I understood the examples of the .where() method. Basically, it seems to be a another way to filter a dataframe. However, when I checked the ...
0
votes
0answers
4 views

Pandas dataframe manipulation [migrated]

I am working on cleaning a huge dataset using python. I have a data frame with 20,000 rows that looks like this: ...
-1
votes
1answer
16 views

How to plot datasets 1 factors for K mean clustering python

I'm unable to plot the data for K mean clusering algo usingsklearn as it throws this error : TypeError: scatter() missing 1 required positional argument: 'y' Here is the function I have written to ...
0
votes
1answer
22 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
1
vote
1answer
23 views

pandas.isna() vs pandas.DataFrame.isna()

I've seen the two documentation pages for pandas.isna() and pandas.DataFrame.isna() but the difference is still unclear to me. Could someone explain the difference to me using examples?
0
votes
0answers
12 views

How to use countVectorizer and TfidfVectorizer?

I am new to data science and trying to figure things out. I have been asked to create a countVectorizer with binary counts of 1-grams and TfidfVecortizer with 1-gram, both avoiding English stop words. ...
-1
votes
1answer
28 views

Finding most selling item everyday from dataset [closed]

Given a dataset: ...
1
vote
2answers
34 views

Pandas: How can I update dataframe values? [closed]

I have two spreadsheets where one is updating the other. How can I update this data using the pandas library? Example, where 'b' updates 'a': ...
0
votes
1answer
21 views

How do i find the most frequently occuring category for each unique customer?

I have a dataset with 30k unique users with each customer having multiple transaction. I aggregated the dataset with one record per user and containing aggregated information such as average ...
0
votes
1answer
39 views

how to see in pandas the element of a csv table with many columns (>30) which the names of its columns is more than 10 character?

how to see in pandas the element of a csv table with many columns (>25) which the names of its columns is more than 10 character? I have 5000 rows and 32 columns and the label of some columns are more ...
-1
votes
1answer
47 views

Compare 2 dataframe based on Column names and provide the miss-matches

I have 2 DF created based on 2 SQL databases. The order of columns is different in both SQL databases. I need to compare 2 dataframes based on a common column and give difference as output. SQL1: <...
0
votes
1answer
24 views

Look up a number inside a list within a pandas cell, and return corresponding string value from a second DF

(I've edited the first column name in the labels_df for clarity) I have two DataFrames, train_df and labels_df. train_df has integers that map to attribute names in the labels_df. I would like to ...
0
votes
1answer
33 views

How to filter rows in Python pandas dataframe with duplicate values in the columns to be filtere

Overall context: I have a data frame that contains observations for every five minute starting at 5 AM in the morning and ending at 8 PM in the evening for several days. I need to filter all the ...
1
vote
1answer
31 views

How to handle out of scope value?

I have a data set in which the score column has to be between 0 to 100 and the subject column has to be one of ['Math','Science','English']. However, my data set ...
1
vote
1answer
42 views

how to check all values in particular column has same data type or not?

I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. ...
2
votes
1answer
21 views

Pandas pivot table, creating ad hoc columns per dimension values

I'm new to pivot tables and have the following dataset: ...
0
votes
0answers
7 views

Predicting when the State of a variable will change based on past behavior

I have a equipment which Has two possible states, Responding & Not Responding, I have with me the amount of time the equipment spent over the last couple of months being in both the states, for ...
0
votes
1answer
14 views

appending file text into list with file name in python

Here 20_newsgroups is a folder which contain 20 folder and each folder contain some file , i just want to make datasets containind words and file name its gives Error ...
0
votes
0answers
14 views

Event prediction. What method to choose

Being newbie in DS and ML, I have some troubles on choosing an adequate way of conducting data processing and modelling. I have the prepared dataset with the following structure ...
-1
votes
1answer
40 views

Finding number of common elements between different columns of a DataFrame

I have a dataframe which has almost 70-80 columns. Each column consists of 100-150 rows in which values are stored as strings. I would like to find, for each column, what is the number of common ...
2
votes
2answers
83 views

Best way to visualize huge amount of data

I have a data set of around 3M row. I has only 2 category (category- 2:1 ratio). Now i want to visualize(scatter plot) it's distribution to understand can the data linearly separable or not(In order ...
0
votes
2answers
29 views

Compare dataframes columns

...
0
votes
0answers
13 views

Calculating Coefficients for a LR Model

quiet simple problem I would assume but I try it for the first time and no Version I find online works for me. I have a working Model and now I want to get the Coefficients for it. Which doesnt work. ...
2
votes
1answer
26 views

How to deal with memory exceptions while processing large amount of data?

Am trying to develop a python script which reads a large CSV file (approx 1.2 GB historical data) in chunks and performs following steps: Take backup of the file Extract new records of previous day ...
1
vote
1answer
49 views

Load large .jsons file into Pandas dataframe

I'm trying to load a large jsons-file (2.5 GB) into a Pandas dataframe. Due to the large size of the file pandas.read_json() will result in a memory error. ...
0
votes
1answer
31 views

Pandas Merge question

I have a dataset where for each date I have two values say A and B, there are dates when one of the values is missing ...
2
votes
1answer
37 views

Data sets that have strings and numerical data all in one column

I m new to the field of Data Science, but I have a great passion for it. Here's a code I wrote for pre processing a data set. It works ...
1
vote
3answers
22 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
2answers
28 views

Pandas Series Sort [closed]

I want to sort digit written in object form as given below: ...
0
votes
0answers
13 views

Table flattening with spread in R

I am trying to use the spread function in dplyr to "flatten" a table but it does not appear to be working as intended. Example table: I am trying to flatten this one so that each Record Number is ...
-2
votes
1answer
23 views

Python Pandas agregation

I have a data set recording daily values of several metrics say R1, R2, for example: ...
0
votes
0answers
13 views

Anomaly detection when doing clustering of text data

Im doing clustering of text data and Im wondering is there a way to do anomaly detection when doing clustering of text data. I have a code that works, I have vectorized the data but I do not know how ...
1
vote
2answers
21 views

Finding correlation between MNIST digits

What way would be correct to calculate the correlation between say digit '1' and digit '7' images from MNIST? Taking average values of all digit '1' pixels and digit '7' pixels to compute correlation ...
0
votes
1answer
24 views

Returning rows from a list of indexes in Python Pandas

Context I have a CSV containing two types of rows, an observation record and on the row below, an observation value which is related to the observation record above. The record line contains a four-...
0
votes
1answer
20 views

how to create multiple plot from a panda Dataframe

I want to plot multiple plots. The data is stored in a pandas dataframe and each row should be a seperate plot. Each row has an ID (ZRD_ID) which doenst matter and a date (TAG) and 24 values to be ...
0
votes
2answers
23 views

The actual results and results from pickle files are not matching in pandas for DBSCAN clustering

I've built a DBSCAN clustering model. The output result and the result after using the pickle files are not matching. Based on HD and MC column, I am clustering WT column. ...
0
votes
1answer
27 views

Key error: count

I get the above mentioned error in this problem. Can anyone please help me correct it? Example data frame: column ID with 3 numbers and column ...
0
votes
2answers
131 views

Delete/Drop only the rows which has all values as NaN in pandas [closed]

I have a Dataframe, i need to drop the rows which has all the values as NaN. ID Age Gender 601 21 M 501 NaN F NaN NaN NaN The ...
1
vote
0answers
28 views

visualize images inside the Dataframe [closed]

I use python 3.7.3 and I run this code on Jupyter Notebook . I have a column named 'image' in my Dataframe named df that contains Url of images I wanna visualize inside the df. I saw some answers ...
0
votes
1answer
36 views

Python can't take input while using functions

While building my model, I figured that it makes sense to reuse some of the code that I've been using for the train dataset, on the test set as well so I took the code performing mutual operations ...
1
vote
1answer
21 views

Starting search from rightmost column and moving left pandas

I'm very new to pandas and I'm trying to implement the code to search for something on an excel spreadsheet. Basically, the most recent price can be found on the furthest right column named ...
0
votes
1answer
229 views

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

Firstly I have a pandas series of recommended product (recmd_prdt_list). In this series there is a possibility of presence of deleted products. So as to remove deleted products from the recommended ...
0
votes
1answer
32 views

How to change a Pandas dataframe into feature vector?

I have a Pandas dataframe with 10 columns, 9 of which are features to be used to predict the 10th column. How is it ossible to convert this Pandas dataframe into X and y vectors to use in a linear ...
0
votes
1answer
56 views

How to create dictionaries out of pandas dataframes?

I import three csv-files with characteristics of countries (rows) and years (columns): ...