Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

66 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
1answer
736 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
3
votes
1answer
266 views

Look for previous date in dataframe that has certain column category in R

I have the following data frame: ...
2
votes
0answers
38 views

How to Ensemble LGBM and XGBoost Machine Learning Models?

I want to Ensemble my predictions for the StratifiedKFold of LGBM and XGBoost into another LGBM Model. I had written the following code which works when the data set has an ID_COL, but in this data ...
2
votes
1answer
112 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
2
votes
0answers
42 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
2
votes
0answers
31 views

Calculate a mean on condtions in a dataframe with dyplr

My table looks like this: ...
2
votes
2answers
809 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
1
vote
0answers
20 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
1
vote
1answer
43 views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
1
vote
1answer
63 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
1
vote
0answers
43 views

Using numpy.ndarray in machine learning sklearn.preprocessing model

I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing: I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there ...
1
vote
1answer
19 views

How to cluster label (in a multilabel classification problem) which mostly appear together in a class

To cluster label (in a multilabel classification problem) which mostly appear together in a dataframe? For example, I have this dataframe: ...
1
vote
3answers
3k views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
1
vote
0answers
25 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
1
vote
0answers
29 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
1
vote
1answer
28 views

Plot three series on the same plot grouping data by day and month

I've a three years data set which I want to plot so to compare the data by data and month, but I'm having an hard time with the final result. I'm nearly there, I guess, but for some strange reason, ...
1
vote
1answer
1k views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
1
vote
0answers
256 views

Dask error when reading data from a large zip file

I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this: ...
1
vote
1answer
262 views

Generating normally distributed data frame with 3 columns

I am trying to generate a normally distributed data frame in Python with 3 columns. Each column requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate ...
1
vote
1answer
661 views

how to implement Excel vlookup on Python

I have a dataset A and dataset B, both data has a column in common which is the ID, i want to check if dataset A ID values can be found in dataset B ID values. Also get the records of dataset B that ...
1
vote
1answer
198 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
1
vote
1answer
468 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
1
vote
1answer
79 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
1
vote
2answers
660 views
1
vote
0answers
97 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
1
vote
0answers
32 views

Best way to find dissimilarity in a 6x2 DataFrame?

I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists. The goal is to find the most ...
1
vote
1answer
168 views

Is there any way to collect categorical features quickly in Julia DataFrames?

I'm using Julia 0.6.3 with Dataframes.jl I was wondering if there was any way to get categorial features easily in Julia? For large datasets it can be impossible to enter everything by hand. My ...
1
vote
2answers
163 views

Predicting Customer Activity Absence

Could you please assist me with to following question? I have a customer activity dataframe that looks like this: It contains at least 500.000 customers and a "timeseries" of 42 months. The ones and ...
0
votes
1answer
22 views

Pandas groupby show value counts

I have a dataframe with 3 columns: job_type (5 categories), age_group (5 categories) and answer (0, 1, 2 or 3) job_type | age_group | answer By ...
0
votes
0answers
7 views

How can i get all the values related to the first column in a direct or indirect relationship?

I wanna write a function using python that allows me to compare each value from the second column of my DataFrame with the first column and if they don't matche compare the second column in the same ...
0
votes
1answer
7 views

Is there a way to make the window in df.rolling dynamic depending on which row it is calculating for?

I have a dataset of stock prices, and I want to add a column of 52 week lows for each day, however for the rows which dont have 365 days above them I just want the column to have the rolling min using ...
0
votes
1answer
30 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
0
votes
1answer
176 views

Python: How to plot time interval from a Dataframe in Pandas

I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job: ...
0
votes
0answers
23 views

How i match with SequenceMatcher ratio two dataframe

I use the SequenceMatcher ratio to match two dataframe with the best ratio. I want to check first if the score A and AA is good then check if the score between B is BB is good then if the score ...
0
votes
1answer
46 views

Python: convert variables into correct format for DataFrame

I have 3 variables that I would like to use to build my dataset but since they are in a weird shape/format, I had no success so far. I'm quite new to this and really appreciate any help!! The 3 ...
0
votes
1answer
201 views

How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names. what should ...
0
votes
0answers
32 views

ValueError: Could not guess the value column! Please hand it to the function as an argument

I'm using the tsfresh library to extract features on my data but I keep getting an error: ...
0
votes
0answers
23 views

How to use pivot_table on a dataframe? How to handle exceptions.UFuncTypeError?

I tried to use the pivot_table function on a dataframe. This was working at the beginning now I get all the time errors, and don't really get why. ...
0
votes
0answers
512 views

Excel error may be caused by pandas writing or big data? advise needed

I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do: open an xml file extract some elements create a pandas dataframe ...
0
votes
0answers
36 views

isnull().sum() returning incorrect figure

I am doing a prediction using RandomForestRegressor. Below is the code for test data: ...
0
votes
0answers
24 views

replacing last duplicate values in pandas

I have a dataframe ,that looks like this | **site** | **Active**| 0 |: deals| Active:| 1 |: deals | InActive:| 2 | : deals | Active:| 3 |:...
0
votes
0answers
9 views

How should I add reference key column to output after modeling is Complete?

0 I have created a csv of data with the following columns: (1) app_key (2) churn, (3)tenure https://i.stack.imgur.com/NAlFF.png I have performed the following code in order to drop app_key and churn <...
0
votes
1answer
62 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
0
votes
0answers
8 views

Get latest Item by Date for a Recommender System

I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would ...
0
votes
0answers
789 views

How to fix IndexError: Too many levels: Index has only 1 level, not 2 in Python

I have written the following code that has a function model_data to perform a particular set of tasks. I have to pass the list of ...
0
votes
0answers
85 views

Numpy array value error

This is my code to train a sigmoid neuron for Iris dataset . ...
0
votes
2answers
80 views

sklearn KNN fit throws out error : value too large for dtype('float64')

I have cleaned the data from nan values and infinite values, the only feature which has a large float is the column 8 (it's a sum) I have no Idea how to fix this last error, I tried all previous ...
0
votes
1answer
38 views

RAM crashed for XML to DataFrame conversion function

I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) ...
0
votes
1answer
25 views

Time Series Plot for floating values

I have a Dataframe which looks as shown below I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as ...
0
votes
1answer
4k views

ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

I want to create a new variable for the dataframe details called lower after iterating through multiple ...