Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
1answer
35 views

find the difference between two columns in specific rows

I have the following columns from one data frame ...
0
votes
1answer
12 views

Map items from list to a column in data frame and append it the data frame [closed]

I have a list as below: ...
0
votes
0answers
8 views

Dask error when reading data from a large zip file

I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this: ...
0
votes
1answer
13 views

How to change random number of random elements in a column by group in R?

I have the following data frame: >df Name 1 Ed 2 Ed 3 Bill 4 Tom 5 Tom 6 Tom 7 Ed 8 Bill 9 Bill 10 Bill My goal is that from each group by "Name" ...
4
votes
2answers
101 views
+50

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
0
votes
0answers
15 views

conversion of any date to datetime in pandas dataframe

I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I want to convert those fields ...
0
votes
2answers
20 views

Updating 1st dataframe columns from 2nd data frame coulmns

I have two data frames df1 and df2 which look something like this. ...
1
vote
1answer
49 views

Python dataframe iterate over rows

Dataset ...
-1
votes
1answer
36 views

Sampling randomly from pd.DataFrame, but ignoring NaN values

I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column. My sampling function at the moment looks like this: ...
-1
votes
1answer
32 views

Custom Pandas function for formatting

My data consists of a lot of Dataframes having the format as below: ...
0
votes
1answer
29 views

Reshaping Pandas DataFrame

I am trying to do the following as shown below. ...
0
votes
0answers
6 views

Handling numpy data in highly frequently updated environment

Having the following environment: Where the input rows (150 million per day) are getting directed into 500 independent DataFrames (300k rows per DataFrame). In ...
0
votes
0answers
20 views

How to combine three lists of data.frames?

I have been reading over similar questions asked here but the solutions aren't working at all and I cannot work out how to make their examples applicable to me. Any help would be appreciated. I ...
0
votes
0answers
12 views

Same Output for every row when doing prediction. Thus I am having wrong predictions

I have implemented an emotion analysis using lstm method, I have trained my model in the first part of the codes and i am now am doing the prediction part. When I run my system, I am having same ...
2
votes
1answer
71 views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
0
votes
1answer
48 views

Plot dataframe with two columns on the x axis

how would I plot the data below with the x axis as Year & Month? Each Year-month combination has a unique monthly count (Y). I am unsure how to proceed, given than they are in different columns, ...
0
votes
0answers
12 views

count movements of values from 1 quarter to the other using pandas

i want to count all 10's that moved to 20 from 1 Quarter to the other and total of all 10's in each Quarter Below is the result i want ...
1
vote
1answer
36 views

Convert CSV from an api to a dataframe format in python

I am new to python, I have extracted some reviews from a website and I used the api of the webscrapping tool to import my data in python and the format is in csv. I want to convert this csv to a ...
0
votes
0answers
14 views

PyOD algorithms that support datasets with empty cells

The idea is to find outliers in large datasets. What am I going to use to detect these outliers? I am going to use a library called PyOD for the detection of anomalies which was developed by Yue ...
1
vote
2answers
33 views

Can an entire data frame be used as a prediction variable?

I am attempting to use XGBoost in R to train a model that predicts a fixed number of target variables using all data from previous dates, as well as the two categorical variables (...
1
vote
2answers
355 views

Export pandas dataframe to a nested dictionary from multiple columns

Its a similar question to Export pandas to dictionary by combining multiple row values But in this case I want something different. ...
0
votes
0answers
42 views

How do I read a dat file, for which I don't know its structure?

Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it. How do I figure out the contents in my dat file. ...
2
votes
2answers
258 views

Passing data to SMOTE after applying train/test split

I'm trying to resample my dataset after splitting it into train and test partitions using SMOTE. Here's my code: ...
1
vote
0answers
19 views

k-means in pandas dataframe, optimal k (number of clusters)? [closed]

I have a dataframe like this. At first, i would like to calculate the optimal number of clusters and then I would like to use k-means without the last column (id). After that i would like to know ...
0
votes
0answers
59 views
-1
votes
2answers
42 views

Creating Pandas DataFrames from a dictionary containing various CSV files

I want to create 3 dataframes (i.e., gs, test and train) from the following dictionary: ...
-1
votes
1answer
69 views

Renaming Column Elements of DataFrames

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
3
votes
1answer
104 views

How to properly store a list of pandas dataframes, so that I can import them in a new python script?

I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
3
votes
4answers
272 views

Clustering algorithm which does not require to tell the number of clusters

I have a dataframe with 2 columns of numerical values. I want to apply a clustering algorithm to put all the entries into the same group, which have a relatively small distance to the other entries. ...
1
vote
0answers
62 views

passing a dataframe as a argument while applying a lambda function on a dataframe [closed]

I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
1
vote
0answers
22 views

Training on small, sparse vectors included in one big, denormalized table/dataframe?

So I have a table like this: ...
0
votes
1answer
20 views

Python - Appending a new Dataframe column that is a function of two separate numerical columns

I have a dataset that gives me the geographic coordinates of residential properties. My goal is to append a new column to this dataframe that shows the properties' distances to various prominent ...
1
vote
1answer
23 views

Help getting corresponding dataframe values

I have two dataframes: ...
0
votes
1answer
79 views

Problem with sort by and group by in pandas

I have a DataFrame with duplicate id's but have different dates. For Example: ...
0
votes
1answer
44 views

How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]

I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are: A <...
0
votes
2answers
264 views

Create a new column in Pandas Dataframe based on the 'NaN' values in another column [closed]

What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large) ...
2
votes
2answers
35 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
1
vote
2answers
944 views

Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode

Hopefully, you don't mind me posting a question here instead of the regular stack exchange forum. I'm learning ML basics and practising pandas. assuming the data frame is called df, column name = ...
1
vote
3answers
187 views

creating a csv file from two csv files

I have two csv files: sub_compiler.to_csv('sub_compiler.csv') sub_compiler.head() and ...
0
votes
1answer
30 views

Merging two pandas dataframes [duplicate]

I have two csv files,and if I do: df_compiler = pd.read_csv('sub_compiler.csv') I obtain: and for the other csv file I do: ...
1
vote
1answer
25 views

Changing the value of a column based on the condition on another column

I have a table whose header looks like this: complaint_type borough street_name incident_zip latitude longitude I want to check if the "incident_zip" column of each row is in a specific ...
2
votes
1answer
55 views

Get elements from lists in pandas dataframe [closed]

I have the following column of a data frame: I get it by doing dataFrame['opcodes]. and I would like to consider only the first 20 and the last 20 elements of ...
0
votes
2answers
155 views

use .apply() function to change values to a column of the dataframe

I have a dataframe which is the following: and I would like to consider only the column of instructions and keep just the values push, test, mov, test ,....., so just the first word of each string ...
-1
votes
2answers
1k views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
0
votes
0answers
30 views

Convert column of float valus to time format using panda

I have a data frame with a column of float data (e.g. 1.15 or 12.50).The first number represent hour and the digit represent minutes. I want to format it in order to have 4 numbers by adding 0 to ...
-1
votes
1answer
856 views
2
votes
0answers
34 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
0
votes
0answers
204 views

Empty Dataframe, Columns [], Index [], not sure how to resolve

The goal for this application is to read stock tickers from excel to python, The information reads and prints great, however when I run the program that reads this 'Tickers' data frame, I obtain this: ...
0
votes
1answer
138 views

Join the List and the Data frame for matching list elements in Python

I have a list =[a,b] and Dataframe as below If the text in the 'Text column' matches with List, then need to add additional column matchedItem So output should be Please suggest on this..How ...
0
votes
1answer
27 views

generate normally distributed dataframe with 3 columns

I am trying to generate a normally distributed data frame in python with 3 columns. Each columns requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate ...