Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
2answers
9 views

How do I remove float object in dataframe? Ex: "winter 2021" to "winter"

I have the premiered column where I want to remove the year and only keep the season. Example: "winter 2021" to become "winter" .
0
votes
0answers
11 views

Scaling columns pandas DataFrame

For a dataset having different numeric columns, they usually have different range and distributions. As an example, I have used the Iris dataset. The distributions of it's 4 columns are shown: My ...
0
votes
1answer
24 views

How to convert duration column from 1 hr. 17 min to 77 min in pandas?

I am trying to convert all the hr., min, and sec into just mins. For example, 1 hr. 17 min to 77 min, and 34 sec to 0.56 min, not 0.34 min. So I have used this code: ...
0
votes
1answer
32 views

Pandas series: how to compute daily cumulative sum of a column

Consider a pandas dataframe where each index is represented as a datetime object in the form like 2009-01-30 23:01:45 . In order to calculate the total value of a ...
-1
votes
0answers
30 views

how to fill nan values based on conditions in pandas?

I am working on the Titanic dataset and there are null values in the Age column. I would like to fill them in based on there Pclass. For example, if class is 1, I ...
0
votes
1answer
16 views

Kernel dies or proses stuck when making LR prediction on dataframe using apply

I'm trying to making predictions with a simple model. model=LogisticRegression() model.fit(X_train,y_train) After fitting, i try to make predictions. A sample ...
1
vote
1answer
85 views

How to detect anomalies in each feature - time series

I have a dataset with 5 features corresponding to 5 sensors that measure each three seconds the state of an accelerator. It is structured as well: ...
0
votes
0answers
30 views

Data science/Dataframe question concerning an approach to a problem

I want to answer which kind of school type (mixed gender, only for boys or only for girls) is more efficient in training its students based on the result of a normalized exam score. And my model ...
2
votes
1answer
25 views

Change the ids of two linked pandas dataframes [closed]

Is it possible to change IDs columns into two linked pandas dataframes? Example: I have these two dataframes: Users and Ratings I would like the Users IDs to become as follows: This is my original ...
1
vote
1answer
54 views

Numpy array vs Pandas DataFrame when training [closed]

https://towardsdatascience.com/speed-testing-pandas-vs-numpy-ffbf80070ee7 (You can open the link in incognito if its locked). Numpy arrays are faster than DataFrame on normal mathematical operations. ...
0
votes
2answers
27 views

How to group by IDs and count the number of groups with occurrence of a variable after first point?

Language: Python 3.8 I have a dataframe that consists of a series of people (each appearing multiple times in the dataframe), dates, and binary variables. I am trying to figure out how many people ...
1
vote
1answer
38 views

convert a column into rows [closed]

I used a webscraper and i got data only in one column like ABC123456 I need the data in this form A B C1 2 34 5 6 Is there any way?
0
votes
1answer
22 views

Having issues with DataSet [closed]

I just started learning data science and am having a problem when generating a dataset. Dataset: ...
1
vote
0answers
26 views

How to make the values in different columns in the correct order based on another data frame (mapping) in Python Pandas

I am pretty new to Python and Pandas and I struggle with combining a messy dataframe from excel with a mapping. I have tried to find some solutions on the Internet, however with no success. My first ...
1
vote
0answers
33 views

Numpy-like reduction of particular dimensions in Pandas DataFrame?

With a numpy multidimensional arrays, it is easy to reduce particular axes with functions such as np.mean or np.sum. For example,...
0
votes
0answers
11 views

Find a column by name in a row in scala spark

I have a Seq[Row].Each row is an Array of Struct.Struct has four fields: a,b,c and d all of which are String.The data in a particular row is something like this: [{"a":"ahahk",&...
1
vote
1answer
20 views

Removing features that correlate with the target label

I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
1
vote
1answer
19 views

putting a statement into a python function doesn't work [closed]

I tried to change one column's value using another column in pandas dataframe. e.g. when the target column is null or has value of "No", and the source column is not null, the value in ...
0
votes
0answers
26 views

Is there row limit when saving pandas dataframe as hdf5?

I am making a dataframe with more than 100 million rows while preprocessing the data. csv file knows that there is a limit on the number of rows, so I want to save it in a different format. I'm trying ...
0
votes
1answer
207 views

Create a tooltip for each dot on a scatter plot created for a Pandas dataframe

I have a pandas based scatter plot that is created from a dataframe, with the following code: ...
1
vote
1answer
18 views

How to move column in rows direction in Pandas DataFrame?

For example, I want get right from left DataFrame like on picture (move column B on 2 steps down): PS. "Na" is not necessary, it can be any controlled value like null, zero or empty string.
0
votes
1answer
261 views

Replacing rows of dataframe with rows of another dataframe that have the same index

I have a dataframe that has rows with indices 0 to 128 and a smaller dataframe with indices 4, 8, 105, and 107. I made edits to the rows in the smaller dataframe and am now trying to replace rows ...
0
votes
1answer
21 views

Extra feature on test set

Suppose I convert categorical data into dummy variables with get_dummies and I get these columns in the training dataset: x_A x_B x_C 0 1 0 0 0 1 1 1 0 But in ...
1
vote
3answers
99 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
0
votes
0answers
29 views

Binning the data of one cell in a data frame using python

I have a dataframe as below: class 1 class 2 0 M NaN 1 17245 NaN 2 graduate Bach I want to use pd.cut() function and bin the data 17245 to 15000-25000. How do I do it?
1
vote
1answer
42 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
1
vote
2answers
35 views

How to convert a dataframe into a single dictionary that is not nested?

I have a dataframe as below: ...
1
vote
0answers
15 views

How to find the related columns between column?

Suppose I have a data frame with columns car_id,number_car,bike_id,number_bike. ...
1
vote
0answers
35 views

Why do I need to create a dataframe just to add a column? [closed]

So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is ...
1
vote
1answer
44 views

Pyspark.read from MariaDB fetches duplicate header rows instead of data

It's a very strange issue- instead of fetching data like expected, it returns every row as a duplicate of the header. Every cell is a literal string of the column name. If I don't use the query like ...
1
vote
1answer
33 views

Flag consecutive dates by group

Below is an example of my data (Room and Date). I would like to generate variables Goal1 , <...
0
votes
1answer
59 views

Error is occurring setting an array element with a sequence

I have the columns in my Data Frame as shown below: ...
0
votes
1answer
287 views

How to create multi-hot encoding from a list column in dataframe?

I have dataframe like this Label IDs 0 [10, 1] 1 [15] 0 [14] I want to create a multihot encoding of the feature ...
0
votes
0answers
275 views

dataframe merge on string not working

having two dataframes each having common column which is country_code. while merging it by 'inner' join, it's returns empty dataframe. but i can see there is many values are common like IND, USA.... <...
0
votes
2answers
36 views

"R" write/export dataframe preserving the class information

Short Question After formatting a dataframe in R with class information like "Date", "Factor", and etc. how to save the dataframe while preserving the class information. ...
1
vote
1answer
85 views

How to calculate the distance between two locations using Haversine Formula? [closed]

I have the columns of Latitude and Longitude of city like shown below : ...
1
vote
1answer
54 views

Keras: Provide One-Hot-Encoded input values to neural network

I have a dataframe which has two columns of interest: A and B with string values. I am trying to build a prediction model which takes in a set of values in A as input and predicts the corresponding B ...
0
votes
1answer
22 views

How to do encode this target vector containing strings

Consider a target vector like ...
1
vote
3answers
482 views

How to create a big data frame in Python

I have a sparse matrix, $X$, created by TfidfVectorizer and its size is $(500000, 200000)$. I want to convert $X$ to a data frame but I'm always getting a memory error. I tried ...
1
vote
0answers
62 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
1
vote
1answer
44 views

How to plot multiple bar charts with different infill and outline?

I have the CSV's of data. I want to plot the distribution of every attribute or a single one to compare the distribution from other CSV's, Since all attributes are the same but the distribution is ...
0
votes
0answers
10 views

How can i get all the values related to the first column in a direct or indirect relationship?

I wanna write a function using python that allows me to compare each value from the second column of my DataFrame with the first column and if they don't matche compare the second column in the same ...
0
votes
1answer
17 views

Efficient way to create matrix that shows if data exits per day [closed]

So I have a dataset containing different ID's and the time the data was created. ...
0
votes
1answer
27 views

Is there a way to make the window in df.rolling dynamic depending on which row it is calculating for?

I have a dataset of stock prices, and I want to add a column of 52 week lows for each day, however for the rows which dont have 365 days above them I just want the column to have the rolling min using ...
1
vote
0answers
15 views

2) Create data frame using following object [closed]

Create data frame using following objects. ...
1
vote
1answer
2k views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
1
vote
0answers
2k views

ValueError: Length of values does not match length of index [closed]

Why when I concatenate DataFrames my code shows this error? I trained a model to detect and classify 6 classes, and then store the coordinates and probabilities of these classes in a DataFrame, but I ...
0
votes
1answer
244 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
2
votes
1answer
55 views

How to compare 10000 data frames in Python?

I have 10000 data frames like this: Each data frame corresponds to a different card game (and so different numbers in the table) and I want to compare these data frames all together. For example, I ...

1
2 3 4 5 6