Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0 votes
1 answer
36 views

Frameworks vs models

My second question (Models vs algorithms) is what is the difference between models and frameworks?
quanity's user avatar
  • 101
0 votes
1 answer
64 views

Dropping dataframe columns with missing values

When I looked up different sources that talk about dropping the columns of a dataframe that contains missing values, I got answers that are starkly different from one another. Some sources say, ...
Apoorva's user avatar
  • 275
0 votes
1 answer
30 views

Finding the labels of a shuffled pandas dataframe

I have a dataframe (namely 'original') with over a thousand numerical columns as features and one label column. I also have a shuffled version of the same dataframe (namely 'shuffled') with label ...
asmani's user avatar
  • 101
0 votes
0 answers
19 views

How to parse the data from the json file describing via dataframe

This is the link to the json file I want to parse the data (resolve [a sentence] into its components and describe their syntactic roles.) Form this file into a dataframe describing: keys as columns ...
Charlie Max's user avatar
0 votes
1 answer
188 views

What is the use of using Vector databases over Pandas Dataframes?

My question is why can't we use a pandas dataframe to do all the data retrieval? Can someone give reasons apart from the fact that databases has the persistence of Data?
Denuwan Weerarathne's user avatar
0 votes
0 answers
59 views

content wise operation in polars series?

i have a dataframe with 2 series, i want to create new column that has 1 if if start word of array one series is equal to start word of another series else 0. How do I do it? i am using polars, it ...
tikendraw's user avatar
  • 135
0 votes
2 answers
52 views

How to filter out redundant id rows with deviant values

First post, so thank you in advance for any pointers! I have a dataframe with an issue similar to the following: id value 10 25.03 11 26.00 11 0.95 12 32.95 13 34.43 13 1.02 One can see ...
OldDogNewTrick's user avatar
0 votes
0 answers
84 views

what is the best practice to handle carriage returns when exporting data from pandas dataframe to CSV?

While trying to export data from a pandas dataframe to a CSV, we noticed that carriage returns (inserted at the middle of a line or at the end of a line) produce outputs that are different from ...
Daniel S.'s user avatar
  • 101
2 votes
2 answers
84 views

Why does pandas.dataframe.info() not update when I delete the first row of a dataset?

I am working with a dataset that comes in with nonsense field names in the first row, with the actual field names in the second row. Currently I'm using this script: ...
SchrodingersStat's user avatar
0 votes
1 answer
30 views

how to prepare pandas dataframe from Solar Flare Data Set?

I am looking to download and use Solar Flare Data Set (https://archive.ics.uci.edu/ml/datasets/Solar+Flare) but unable to figure out how to get the pandas dataframe from the file that is being ...
Sachin's user avatar
  • 101
0 votes
0 answers
20 views

Running all cells again adding new rows instead of return new dataframe

i am creating a class just like lazypredict library but with k fold support and return a data frame withh the mean score of every model every thing works and i can ...
Burhan Khanzada's user avatar
0 votes
1 answer
36 views

Build a dataframe with multiple CSV

I am collecting metrics on 6 REST services from a microservices architecture. For each instant collected, I extract two CSV from each service. One CSV contains three latency metrics (99th quantile, ...
user2989745's user avatar
0 votes
1 answer
512 views

XGB DataFrame.dtypes error

Here is the code to assign the variables ...
Nata's user avatar
  • 3
0 votes
0 answers
22 views

How to structure dataframe combinations for regression, without corruption/loss?

I have a data set, redacted sample below. My goal is linear regression. My question is: Have I created unintended results, due to how I structured the df, using concat and/or div? For example, ...
Megan Perry's user avatar
0 votes
1 answer
451 views

Append prediction of tensorflow to a pandas dataframe

I built a tensorflow model to make text classification in four category, after testing and evaluating it, I need to apply it to actual data to predict the class of them, I create a predict function ...
Ali A. Jalil's user avatar
0 votes
1 answer
19 views

Combination of groupby and mean methods

I am looking at the below csv file : We have the question : Display the mean of the variable gre by group of admitted/not admitted students, using the combination of groupby and mean methods. I would ...
evinda's user avatar
  • 101
0 votes
0 answers
11 views

How to storge large data values in data frame or what would be suitable way?

My output has two columns, basically one is a token name and the second column is the embedding which is the array consisting of 300 values, if I store these tables in CSV files, most of the array ...
Abdul Basit Niazi's user avatar
0 votes
0 answers
14 views

'molten' versus truly 'tidy' dataframe

I am currently 'tidying' up a dataset related to a city's migration statistics across three characteristics: year, nationality and gender, i.e. how many nationals versus non-nationals by gender have ...
Max's user avatar
  • 1
0 votes
2 answers
176 views

Average 5 DataFrames with same index and columns?

I have 5 Dataframe with same index and columns . but the numbers is them is different , I want to average it out . Is there any way to do that ?
tikendraw's user avatar
  • 135
0 votes
0 answers
23 views

How to reference an intermediary object in scala?

I am using spark/scala to do some transformations on a spark dataframe. I have the following code: ...
ptushev's user avatar
  • 11
0 votes
0 answers
19 views

Extracting variables from different properties with similar timestamps -- PySpark

I have a list of variables I need to extract. I am using the PySpark API. Almost all variables are already in the same property, but just one variable I also need was logged in another property. For ...
Guarneer's user avatar
  • 101
0 votes
1 answer
3k views

select rows containing max value basing on another duplicated rows of a group

I want to select rows by the maximum values of another column which would be the duplicated rows containing duplicated maximum values of a group. This should contain three steps: (1) group dataframe ...
Jiao's user avatar
  • 103
0 votes
1 answer
12k views

ValueError: Could not interpret value `female_height` for parameter `x` when trying to plot certain rows

I have a big dataframe that consists of male and female shoe sizes. I am currently trying to plot size versus height for male and female separately. I have tried to slice the data and stored it in ...
royalTiger's user avatar
0 votes
1 answer
426 views

Using Simple imputer replace NaN values with mean error

I am trying to replace 2 missing NaN values in data using the SimpleImputer. I load my data as follow; ...
royalTiger's user avatar
0 votes
0 answers
398 views

Creating a scatter plot with matplotlib with large dataset

I am trying to create a scatter plot from instrument peak area data recorded every tenth of a second or so. For some reason, when I try to plot it with Matplotlib, I get a straight line, which is not ...
Peter Bright's user avatar
0 votes
1 answer
779 views

Append Existing Columns to another Column in Pandas Dataframe

I have a data that looks like this: The T2M indicates the temperature, and the followed row is the year. I want to append all the similar parameters columns under a single column having all the years,...
Mohamed Abduljawad's user avatar
0 votes
1 answer
25 views

Transforming pandas dataframe from multiple series to categorical label

I have a dataframe in set up in this way: depth val1 val2 1 0.5 0.6 2 0.4 0.7 3 0.3 0.8 and I want to get it into this layout: depth y category 1 0.5 val1 1 0.6 val2 2 0.4 val1 2 0.7 val2 3 ...
artificialnull's user avatar
0 votes
1 answer
307 views

Cumsum starting value not 0 in Pandas

I have the following dataframe df of athletes (indexed by Athlete_ID) and their Rank (indexed by ...
borcherds's user avatar
0 votes
1 answer
1k views

Set column names to column value in pandas forloop, add prefix to column names

I have a dataframe with repeat column names, and I am trying to add a prefix label to column names with an appropriate identifier that comes from a column within the dataframe. I have this data frame: ...
Peter Bright's user avatar
0 votes
0 answers
2k views

How to correctly create a PyTorch Tensor from a Pandas DataFrame?

I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data. Obviously, This new tensor do ...
EvilRoach's user avatar
  • 101
0 votes
2 answers
602 views

How to Insert a Row before Dataframe in Pandas

I have a csv file that is used as a pandas dataframe, now I only need to insert a dummy row before the dataframe starts like in the screenshot denoted as "Label". How can I do that?
Mohamed Abduljawad's user avatar
0 votes
3 answers
3k views

How to upload a Pandas DataFrame into a Postgres table in Airflow?

I need to create a task in Airflow which downloads data from one Postgres database and uploads it to another. My code looks like this: ...
frogseer's user avatar
1 vote
1 answer
73 views

do inner functions have a substantial impact when used in series.apply() in Pandas

In pandas, if I use series.apply() to apply a function with an inner function definition, for example: ...
Alex's user avatar
  • 13
0 votes
2 answers
2k views

How to plot categorical variables with a pie chart

I am concerned with a single column (fruit) from my df: ...
Tom Bomer's user avatar
2 votes
2 answers
2k views

nested xml to dataframe [closed]

I am trying to convert the below mentioned sample xml file to a pandas dataframe. I have multiple xml files which I will loop over to add all xml data into a single dataframe once i succeed with this ...
Haider's user avatar
  • 31
1 vote
3 answers
79 views

Pandas Dataframe grouping and summarizing

I have the following pandas dataframe: df_1: ...
Ali A. Jalil's user avatar
0 votes
1 answer
1k views

Str.contains and isin function do not return all correct rows of dataframe [closed]

Given a list of strings L1 L1 = ['a', 'b', 'c'], I need to extract the rows which contain the values given in list L1. I used the isin function: ...
kanha's user avatar
  • 23
-1 votes
1 answer
136 views

Removing rows with total number of ID occurred not equal to a specific number in Pandas Python [closed]

I have the following data frame in Pandas: ...
Ishigami's user avatar
  • 101
0 votes
1 answer
2k views

How to fix ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am using pytest to test a function that reads a csv into a dictionary of lists. In the test function it writes a sample dictionary to a csv then calls the function to read it back out. To accomplish ...
Michael Wallentine's user avatar
0 votes
1 answer
14 views

Conditional multiplicaiton of two dataFrames [closed]

I have a first dataFrame which looks like this ...
Meth's user avatar
  • 101
1 vote
2 answers
473 views

python compare 2 dataframes [closed]

in reference to this post, python-pandas-compare-columns-in-separate-dataframes-then-delete-non-matching, I can't get the isin function to work with my 2 example dataframes : df 1 : ...
iulianpn's user avatar
0 votes
1 answer
293 views

Seaborn Heatmap with month & hour of database entry

I have a dataframe with X rows. For each row, I have the information of the month (value from 1 to 12) and hour (value from 1 to 24) in separate columns I need to create a heatmap with seaborn in ...
vib's user avatar
  • 103
0 votes
1 answer
822 views

Turning multiple binary columns into categorical (with less columns) with Python Pandas

I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
Legna's user avatar
  • 21
2 votes
1 answer
433 views

Dataframe Python - Conditional Column based on multiple criteria

I want someone to correct my code in python. My goal is to create a code that will add a new column that is based on the conditional function of two columns in my dataframe. I want to add a fees ...
Mostafa Fakhra's user avatar
0 votes
2 answers
2k views

Create new rows based on a value in a column

My dateset is generated like the example df = {'event':['A','B','C','D'], 'budget':['123','433','1000','1299'], 'duration_days':['6','3','4','2']} I need ...
conradoov's user avatar
0 votes
1 answer
1k views

Plot multiple time series from single dataframe

I have a dataframe with multiple time series and columns with labels. My goal is to plot all time series in a single plot, where the labels should be used in the legend of the plot. The important ...
Davgrabner's user avatar
0 votes
0 answers
13 views

Identify the existence of a wall like structure in a given int array

The idea is that I will query an API endpoint which will return me an array consisting of a price value and a quantity value [price, quantity]. In this dataset ...
deniz's user avatar
  • 1
0 votes
1 answer
3k views

For each unique value in a column, count respective unique values in another column

I have a set of tabular data (e.g. csv) representing accesses to a server through a specific protocol . The data follows this format: ...
Mafalda Hopkirk's user avatar
0 votes
1 answer
1k views

How to set a new numeric index

I have read data from a json file, and the dataframe looks like: Col1 Col2 Col3 A: 1 Hi 7 B: 22 CC 1 I think it is because the file was a ...
Karl 17302's user avatar
1 vote
0 answers
102 views

How to solve this IndexError? [closed]

I have created a training dataframe Traindata as following: dataFile='/content/drive/Colab Notebooks/.../Normal_Anomalous_8Digits.csv' data8=pd.read_csv(dataFile) ...
TariqS's user avatar
  • 21

1
2 3 4 5
8