Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Creating a huge sparse data frame in Python without getting memory error

I have a scipy.sparse.csr_matrix, $X$, created by TfidfVectorizer and its shape is $(500000, 200000)$. I want to create a data frame from $X$ but every effort has resulted in memory error ! I have ...
0
votes
0answers
9 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
-1
votes
1answer
34 views

pandas count and show percentage

I have a df like this I would like to count the occurrence and also present in percentage, like below. I can easily do this in excel pivot table function, how to do this in python? note: the result ...
1
vote
1answer
33 views

How to plot multiple bar charts with different infill and outline?

I have the CSV's of data. I want to plot the distribution of every attribute or a single one to compare the distribution from other CSV's, Since all attributes are the same but the distribution is ...
0
votes
2answers
23 views

How to pass a list in dictionary values? [closed]

I have a list named as distribution_list ...
0
votes
0answers
7 views

How can i get all the values related to the first column in a direct or indirect relationship?

I wanna write a function using python that allows me to compare each value from the second column of my DataFrame with the first column and if they don't matche compare the second column in the same ...
0
votes
1answer
13 views

Efficient way to create matrix that shows if data exits per day [closed]

So I have a dataset containing different ID's and the time the data was created. ...
0
votes
1answer
7 views

Is there a way to make the window in df.rolling dynamic depending on which row it is calculating for?

I have a dataset of stock prices, and I want to add a column of 52 week lows for each day, however for the rows which dont have 365 days above them I just want the column to have the rolling min using ...
1
vote
0answers
9 views

2) Create data frame using following object [closed]

Create data frame using following objects. ...
1
vote
1answer
36 views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
1
vote
0answers
118 views

ValueError: Length of values does not match length of index [closed]

Why when I concatenate DataFrames my code shows this error? I trained a model to detect and classify 6 classes, and then store the coordinates and probabilities of these classes in a DataFrame, but I ...
0
votes
1answer
22 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
2
votes
1answer
43 views

How to compare 10000 data frames in Python?

I have 10000 data frames like this: Each data frame corresponds to a different card game (and so different numbers in the table) and I want to compare these data frames all together. For example, I ...
0
votes
1answer
114 views

Python: How to plot time interval from a Dataframe in Pandas

I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job: ...
-1
votes
1answer
20 views

Multiplying a dataframe by a larger one [closed]

I have two dataframes df1 and df2 with the same columns but not the same row number. I want to multiply them element-wise such ...
0
votes
2answers
26 views

Sort Pandas Dataframe per column

I have a dataset with age + another 14 variables. I have created 13 bins representing different age groups like so: ...
0
votes
0answers
21 views

How i match with SequenceMatcher ratio two dataframe

I use the SequenceMatcher ratio to match two dataframe with the best ratio. I want to check first if the score A and AA is good then check if the score between B is BB is good then if the score ...
1
vote
1answer
49 views

Replace values in dataframe with another dataframes values based on condition [closed]

I have 2 dataframes with the same columns names and rows number but the elements' values are different. I need to change the value of each element in the first dataframe to 1 if its value in the the ...
0
votes
1answer
45 views

Python: convert variables into correct format for DataFrame

I have 3 variables that I would like to use to build my dataset but since they are in a weird shape/format, I had no success so far. I'm quite new to this and really appreciate any help!! The 3 ...
0
votes
1answer
27 views

Access keys of pandas dataframe when using groupby

I have the following database: And I would like to know how many times a combination of BirthDate and Zipcode is repeated throughout the data table: Now, my question is: How can I access the keys of ...
2
votes
1answer
30 views

How to fill df with data from another df - Python [closed]

I am trying to fill df with values form a different one. The first df is: The second df is: I want to add the column to first df having values from the second one to receive output: number trans ...
0
votes
1answer
130 views

How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names. what should ...
0
votes
0answers
29 views

ValueError: Could not guess the value column! Please hand it to the function as an argument

I'm using the tsfresh library to extract features on my data but I keep getting an error: ...
0
votes
0answers
20 views

How to use pivot_table on a dataframe? How to handle exceptions.UFuncTypeError?

I tried to use the pivot_table function on a dataframe. This was working at the beginning now I get all the time errors, and don't really get why. ...
0
votes
1answer
19 views

Is it ok to fill a pandas dataframe with NaN values?

Is it correct to fill a pandas dataframe with NaN values? In specific: if I have a dataframe with a user name and his age is it ok to fill the age column with int and NaN values. ...
1
vote
1answer
53 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
1
vote
0answers
38 views

Using numpy.ndarray in machine learning sklearn.preprocessing model

I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing: I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there ...
0
votes
0answers
433 views

Excel error may be caused by pandas writing or big data? advise needed

I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do: open an xml file extract some elements create a pandas dataframe ...
2
votes
1answer
28 views

How to fill NaN based on the previous row that contains a specific value [closed]

I would like to fill the missing data using the value from the previous trading day of the same stock. In this example, the AAPL stock should be 100. I've tried with ...
0
votes
0answers
35 views

isnull().sum() returning incorrect figure

I am doing a prediction using RandomForestRegressor. Below is the code for test data: ...
1
vote
1answer
18 views

How to cluster label (in a multilabel classification problem) which mostly appear together in a class

To cluster label (in a multilabel classification problem) which mostly appear together in a dataframe? For example, I have this dataframe: ...
1
vote
1answer
24 views

Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines

I am trying to merge a "roster" where there are supposed to be several positions with one person per office. However, I have come across two instances where there are two people in one ...
0
votes
0answers
21 views

replacing last duplicate values in pandas

I have a dataframe ,that looks like this | **site** | **Active**| 0 |: deals| Active:| 1 |: deals | InActive:| 2 | : deals | Active:| 3 |:...
0
votes
1answer
18 views

replace values based on Number of duplicate rows are occured

I have a dataframe ,that looks like this site Active 0 deals Active 1 deals Active 2 deals Active 3 discount Active 4 discount Active i ...
0
votes
0answers
9 views

How should I add reference key column to output after modeling is Complete?

0 I have created a csv of data with the following columns: (1) app_key (2) churn, (3)tenure https://i.stack.imgur.com/NAlFF.png I have performed the following code in order to drop app_key and churn <...
-1
votes
1answer
31 views

How can I get all column values from a panda dataframe and save them into one string [closed]

this question has been ask couple of times but none of the answer is working for that what I want. For example I have this excel Dokument: ...
0
votes
1answer
28 views

List value in Pandas DataFrame column makes analysis harder

Should I move to database? I have a list of courses data in JSON format that looks like this: ...
0
votes
1answer
477 views
0
votes
1answer
189 views

How to convert DataFrame column to Rows in Python?

I have the following dataset in df_1 which I want to convert into the format of df_2. In df_2...
0
votes
1answer
55 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
2
votes
0answers
31 views

How to Ensemble LGBM and XGBoost Machine Learning Models?

I want to Ensemble my predictions for the StratifiedKFold of LGBM and XGBoost into another LGBM Model. I had written the following code which works when the data set has an ID_COL, but in this data ...
0
votes
0answers
8 views

Get latest Item by Date for a Recommender System

I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would ...
0
votes
0answers
749 views

How to fix IndexError: Too many levels: Index has only 1 level, not 2 in Python

I have written the following code that has a function model_data to perform a particular set of tasks. I have to pass the list of ...
0
votes
3answers
3k views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
0
votes
1answer
107 views

Comparing 2 columns from separate dataframes and copy some row values from one df to another if column value matches in pandas

Im relatively new to pandas andpython . I have a simple problem but I can not find the solution . I have 2 dataframes DF1 and DF2 as shown in image Both dataframes have ID coluumns. I wannt to check ...
0
votes
1answer
271 views

Calculate all days for each years between two dates in a pandas dataframe

I'm new in python and coding. I'm doing a university project exercise. Last question is : " For each year, compute the total amount of loans. Each loan that has planned expiration time and ...
0
votes
0answers
81 views

Numpy array value error

This is my code to train a sigmoid neuron for Iris dataset . ...
0
votes
2answers
78 views

sklearn KNN fit throws out error : value too large for dtype('float64')

I have cleaned the data from nan values and infinite values, the only feature which has a large float is the column 8 (it's a sum) I have no Idea how to fix this last error, I tried all previous ...
0
votes
1answer
82 views

Bootstrap clarification for datasets with multiple columns

How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns? For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...

1
2 3 4 5 6