Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
1
vote
1answer
13 views

How to move column in rows direction in Pandas DataFrame?

For example, I want get right from left DataFrame like on picture (move column B on 2 steps down): PS. "Na" is not necessary, it can be any controlled value like null, zero or empty string.
0
votes
1answer
17 views

Replacing rows of dataframe with rows of another dataframe that have the same index

I have a dataframe that has rows with indices 0 to 128 and a smaller dataframe with indices 4, 8, 105, and 107. I made edits to the rows in the smaller dataframe and am now trying to replace rows ...
0
votes
1answer
17 views

Extra feature on test set

Suppose I convert categorical data into dummy variables with get_dummies and I get these columns in the training dataset: x_A x_B x_C 0 1 0 0 0 1 1 1 0 But in ...
0
votes
0answers
45 views

Value Error: Specifying the columns using strings is only supported for pandas Data Frames [closed]

I have built a basic Linear Regression model on the Boston house price data set and deployed it using flask. Now when I go to the home page where I have to enter the values of features and click on ...
-1
votes
1answer
31 views

How to Transform Rows into columns & Fill the values with Existing Values using Pandas as DataFrame

Here, I would like to transform rows into new columns and fill the column values with specific rows. Sample input:
0
votes
1answer
17 views

how to create new column for each unique date in pandas dataframe [closed]

Here I need to Create New Columns For Each Unique Date Value For example
1
vote
3answers
40 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
0
votes
0answers
26 views

Binning the data of one cell in a data frame using python

I have a dataframe as below: class 1 class 2 0 M NaN 1 17245 NaN 2 graduate Bach I want to use pd.cut() function and bin the data 17245 to 15000-25000. How do I do it?
1
vote
1answer
14 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
0
votes
2answers
30 views

How to convert a dataframe into a single dictionary that is not nested?

I have a dataframe as below: ...
1
vote
0answers
15 views

How to find the related columns between column?

Suppose I have a data frame with columns car_id,number_car,bike_id,number_bike. ...
1
vote
0answers
28 views

Why do I need to create a dataframe just to add a column? [closed]

So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is ...
1
vote
1answer
32 views

Pyspark.read from MariaDB fetches duplicate header rows instead of data

It's a very strange issue- instead of fetching data like expected, it returns every row as a duplicate of the header. Every cell is a literal string of the column name. If I don't use the query like ...
1
vote
1answer
27 views

Flag consecutive dates by group

Below is an example of my data (Room and Date). I would like to generate variables Goal1 , <...
0
votes
1answer
54 views

Error is occurring setting an array element with a sequence

I have the columns in my Data Frame as shown below: ...
0
votes
1answer
72 views

How to create multi-hot encoding from a list column in dataframe?

I have dataframe like this Label IDs 0 [10, 1] 1 [15] 0 [14] I want to create a multihot encoding of the feature ...
0
votes
0answers
113 views

dataframe merge on string not working

having two dataframes each having common column which is country_code. while merging it by 'inner' join, it's returns empty dataframe. but i can see there is many values are common like IND, USA.... <...
0
votes
2answers
21 views

“R” write/export dataframe preserving the class information

Short Question After formatting a dataframe in R with class information like "Date", "Factor", and etc. how to save the dataframe while preserving the class information. ...
1
vote
1answer
20 views

How to calculate the distance between two locations using Haversine Formula? [closed]

I have the columns of Latitude and Longitude of city like shown below : ...
1
vote
1answer
34 views

Keras: Provide One-Hot-Encoded input values to neural network

I have a dataframe which has two columns of interest: A and B with string values. I am trying to build a prediction model which takes in a set of values in A as input and predicts the corresponding B ...
0
votes
1answer
16 views

How to do encode this target vector containing strings

Consider a target vector like ...
1
vote
3answers
379 views

How to create a big data frame in Python

I have a sparse matrix, $X$, created by TfidfVectorizer and its size is $(500000, 200000)$. I want to convert $X$ to a data frame but I'm always getting a memory error. I tried ...
1
vote
0answers
30 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
1
vote
1answer
41 views

How to plot multiple bar charts with different infill and outline?

I have the CSV's of data. I want to plot the distribution of every attribute or a single one to compare the distribution from other CSV's, Since all attributes are the same but the distribution is ...
0
votes
0answers
8 views

How can i get all the values related to the first column in a direct or indirect relationship?

I wanna write a function using python that allows me to compare each value from the second column of my DataFrame with the first column and if they don't matche compare the second column in the same ...
1
vote
1answer
17 views
0
votes
1answer
14 views

Efficient way to create matrix that shows if data exits per day [closed]

So I have a dataset containing different ID's and the time the data was created. ...
0
votes
1answer
11 views

Is there a way to make the window in df.rolling dynamic depending on which row it is calculating for?

I have a dataset of stock prices, and I want to add a column of 52 week lows for each day, however for the rows which dont have 365 days above them I just want the column to have the rolling min using ...
1
vote
0answers
12 views

2) Create data frame using following object [closed]

Create data frame using following objects. ...
1
vote
1answer
342 views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
1
vote
0answers
1k views

ValueError: Length of values does not match length of index [closed]

Why when I concatenate DataFrames my code shows this error? I trained a model to detect and classify 6 classes, and then store the coordinates and probabilities of these classes in a DataFrame, but I ...
0
votes
1answer
126 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
2
votes
1answer
50 views

How to compare 10000 data frames in Python?

I have 10000 data frames like this: Each data frame corresponds to a different card game (and so different numbers in the table) and I want to compare these data frames all together. For example, I ...
0
votes
1answer
746 views

Python: How to plot time interval from a Dataframe in Pandas

I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job: ...
-1
votes
1answer
92 views

Multiplying a dataframe by a larger one [closed]

I have two dataframes df1 and df2 with the same columns but not the same row number. I want to multiply them element-wise such ...
1
vote
2answers
33 views

Sort Pandas Dataframe per column [closed]

I have a dataset with age + another 14 variables. I have created 13 bins representing different age groups like so: ...
0
votes
0answers
64 views

How i match with SequenceMatcher ratio two dataframe

I use the SequenceMatcher ratio to match two dataframe with the best ratio. I want to check first if the score A and AA is good then check if the score between B is BB is good then if the score ...
1
vote
1answer
844 views

Replace values in dataframe with another dataframes values based on condition [closed]

I have 2 dataframes with the same columns names and rows number but the elements' values are different. I need to change the value of each element in the first dataframe to 1 if its value in the the ...
0
votes
1answer
53 views

Python: convert variables into correct format for DataFrame

I have 3 variables that I would like to use to build my dataset but since they are in a weird shape/format, I had no success so far. I'm quite new to this and really appreciate any help!! The 3 ...
0
votes
1answer
31 views

Access keys of pandas dataframe when using groupby

I have the following database: And I would like to know how many times a combination of BirthDate and Zipcode is repeated throughout the data table: Now, my question is: How can I access the keys of ...
2
votes
1answer
34 views

How to fill df with data from another df - Python [closed]

I am trying to fill df with values form a different one. The first df is: The second df is: I want to add the column to first df having values from the second one to receive output: number trans ...
0
votes
1answer
644 views

How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names. what should ...
0
votes
0answers
60 views

ValueError: Could not guess the value column! Please hand it to the function as an argument

I'm using the tsfresh library to extract features on my data but I keep getting an error: ...
0
votes
0answers
24 views

How to use pivot_table on a dataframe? How to handle exceptions.UFuncTypeError?

I tried to use the pivot_table function on a dataframe. This was working at the beginning now I get all the time errors, and don't really get why. ...
0
votes
1answer
22 views

Is it ok to fill a pandas dataframe with NaN values?

Is it correct to fill a pandas dataframe with NaN values? In specific: if I have a dataframe with a user name and his age is it ok to fill the age column with int and NaN values. ...
1
vote
1answer
158 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
1
vote
0answers
52 views

Using numpy.ndarray in machine learning sklearn.preprocessing model

I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing: I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there ...
0
votes
0answers
927 views

Excel error may be caused by pandas writing or big data? advise needed

I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do: open an xml file extract some elements create a pandas dataframe ...
2
votes
1answer
52 views

How to fill NaN based on the previous row that contains a specific value [closed]

I would like to fill the missing data using the value from the previous trading day of the same stock. In this example, the AAPL stock should be 100. I've tried with ...
0
votes
0answers
44 views

isnull().sum() returning incorrect figure

I am doing a prediction using RandomForestRegressor. Below is the code for test data: ...

1
2 3 4 5 6