Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
0
votes
0answers
4 views

Creating a huge sparse data frame in Python without getting memory error

I have a scipy.sparse.csr_matrix, $X$, created by TfidfVectorizer and its shape is $(500000, 200000)$. I want to create a data frame from $X$ but every effort has resulted in memory error ! I have ...
0
votes
3answers
4k views

Removing duplicates and keeping the last entry in pandas

I have a table and I'm trying to remove all the duplicate and keep the the rows that has the latest datestamp. Here is the table: ...
1
vote
1answer
33 views

How to plot multiple bar charts with different infill and outline?

I have the CSV's of data. I want to plot the distribution of every attribute or a single one to compare the distribution from other CSV's, Since all attributes are the same but the distribution is ...
-1
votes
1answer
34 views

pandas count and show percentage

I have a df like this I would like to count the occurrence and also present in percentage, like below. I can easily do this in excel pivot table function, how to do this in python? note: the result ...
0
votes
2answers
8k views

How do I read a dat file, for which I don't know its structure?

Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it. How do I figure out the contents in my dat file. ...
0
votes
2answers
23 views
0
votes
3answers
2k views

Remove columns with a certain number of consecutive zeros

I'm using pandas and I am dealing with time series of sales. What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...
1
vote
1answer
72 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
2
votes
1answer
108 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
0
votes
0answers
7 views

How can i get all the values related to the first column in a direct or indirect relationship?

I wanna write a function using python that allows me to compare each value from the second column of my DataFrame with the first column and if they don't matche compare the second column in the same ...
0
votes
1answer
1k views

Comparing dataframe object with string value in django

I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL ...
1
vote
1answer
10 views
0
votes
2answers
1k views

Convert json to dataframe in python

I have a json array f below format ...
0
votes
1answer
114 views

Python: How to plot time interval from a Dataframe in Pandas

I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job: ...
0
votes
1answer
13 views

Efficient way to create matrix that shows if data exits per day [closed]

So I have a dataset containing different ID's and the time the data was created. ...
0
votes
1answer
45 views

Python: convert variables into correct format for DataFrame

I have 3 variables that I would like to use to build my dataset but since they are in a weird shape/format, I had no success so far. I'm quite new to this and really appreciate any help!! The 3 ...
0
votes
1answer
7 views

Is there a way to make the window in df.rolling dynamic depending on which row it is calculating for?

I have a dataset of stock prices, and I want to add a column of 52 week lows for each day, however for the rows which dont have 365 days above them I just want the column to have the rolling min using ...
0
votes
1answer
3k views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...
1
vote
1answer
242 views

Generating normally distributed data frame with 3 columns

I am trying to generate a normally distributed data frame in Python with 3 columns. Each column requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate ...
2
votes
2answers
796 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
1
vote
0answers
9 views

2) Create data frame using following object [closed]

Create data frame using following objects. ...
1
vote
1answer
36 views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
0
votes
1answer
22 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
1
vote
0answers
118 views

ValueError: Length of values does not match length of index [closed]

Why when I concatenate DataFrames my code shows this error? I trained a model to detect and classify 6 classes, and then store the coordinates and probabilities of these classes in a DataFrame, but I ...
0
votes
1answer
55 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
14
votes
4answers
21k views

One hot encoding alternatives for large categorical values

I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns? I found this interesting link. But they are ...
14
votes
2answers
102k views

How to plot two columns of single DataFrame on Y axis

I have two data frames (Action, Comedy). Action contains two columns (year, rating) ratings columns contains average rating with respect to year. The Comedy data frame contains the same two columns ...
2
votes
1answer
43 views

How to compare 10000 data frames in Python?

I have 10000 data frames like this: Each data frame corresponds to a different card game (and so different numbers in the table) and I want to compare these data frames all together. For example, I ...
0
votes
1answer
34 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
2
votes
0answers
31 views

How to Ensemble LGBM and XGBoost Machine Learning Models?

I want to Ensemble my predictions for the StratifiedKFold of LGBM and XGBoost into another LGBM Model. I had written the following code which works when the data set has an ID_COL, but in this data ...
1
vote
1answer
28 views

Plot three series on the same plot grouping data by day and month

I've a three years data set which I want to plot so to compare the data by data and month, but I'm having an hard time with the final result. I'm nearly there, I guess, but for some strange reason, ...
1
vote
1answer
1k views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
0
votes
2answers
1k views

How to split data frame into groups, combine rows

I have a large data set with 405 columns, many rows, and data from 15 sites. Each site has 27 columns, each one one quadrats data. Rows are species data. I would like to split the data into the 15 ...
0
votes
0answers
433 views

Excel error may be caused by pandas writing or big data? advise needed

I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do: open an xml file extract some elements create a pandas dataframe ...
1
vote
1answer
657 views

how to implement Excel vlookup on Python

I have a dataset A and dataset B, both data has a column in common which is the ID, i want to check if dataset A ID values can be found in dataset B ID values. Also get the records of dataset B that ...
-1
votes
1answer
20 views

Multiplying a dataframe by a larger one [closed]

I have two dataframes df1 and df2 with the same columns but not the same row number. I want to multiply them element-wise such ...
0
votes
1answer
130 views

How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names. what should ...
0
votes
3answers
3k views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
2
votes
1answer
305 views

Breaking a DataFrame based on Specific Column Values [closed]

Assume I have a following dataframe which has years of experience and salary. I would like to create a new dataframe from the existing one, only including observations where salary is above 45000. I ...
0
votes
2answers
26 views

Sort Pandas Dataframe per column

I have a dataset with age + another 14 variables. I have created 13 bins representing different age groups like so: ...
0
votes
0answers
21 views

How i match with SequenceMatcher ratio two dataframe

I use the SequenceMatcher ratio to match two dataframe with the best ratio. I want to check first if the score A and AA is good then check if the score between B is BB is good then if the score ...
0
votes
1answer
63 views

Better way of writing following Pandas Python code

Following code is given, we need to sum up the medals. ...
1
vote
1answer
53 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
38
votes
5answers
252k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
0
votes
1answer
25 views

Time Series Plot for floating values

I have a Dataframe which looks as shown below I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as ...
1
vote
1answer
49 views

Replace values in dataframe with another dataframes values based on condition [closed]

I have 2 dataframes with the same columns names and rows number but the elements' values are different. I need to change the value of each element in the first dataframe to 1 if its value in the the ...
3
votes
1answer
700 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
0
votes
1answer
27 views

Access keys of pandas dataframe when using groupby

I have the following database: And I would like to know how many times a combination of BirthDate and Zipcode is repeated throughout the data table: Now, my question is: How can I access the keys of ...

1
2 3 4 5 6