Skip to main content

Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

Doing a join between excel tables/sheets in Python Pandas, why do all of the entries in one particular column have the same values after the join?

I have two excel docs: invasive_plants_in_and_around_park_with_compared_and_with_habitat_data_hand_edited.xlsx -with 3 sheets "Species_only_at_park", "Species_only_outside_park", ...
Henry 's user avatar
0 votes
0 answers
13 views

How to visualize correlation values coming from two data frame for comparison

I am working on a project wherein we are comparing two methods used for modeling gene expression. One method is using elastic net and other is using lasso regression. In one method: we see that ...
Rhea Bedi's user avatar
0 votes
0 answers
14 views

pyspark combined standard deviation?

I have a data frame which consists of multiple standard deviations with mean and count of each. my df is as follows ...
userDuR's user avatar
  • 101
1 vote
2 answers
102 views

Merging Turbine and Forecast Data Diagonally

I am trying to combine my main dataset, which is hourly data collected from a turbine, with forecast data. The forecast data is collected by calling an API every hour to get a new .csv file which is ...
Eli Orians's user avatar
1 vote
1 answer
65 views

How to filter a dataframe and get a sub data frame based on frequency?

I have a data-frame. Lets call it df. My data frame has a column with categorical values e.g. a column named Place. But the problem is that there are many different (string) values in that column and ...
Ali.A's user avatar
  • 71
0 votes
0 answers
10 views

How to make duplicates unique in a DataFrame robustly?

I have the following table. I was able to parse it and display it as a DataFrame with pandas in python I need to change the Ref for every value that is duplicate such that it becomes unique. For ...
bem22's user avatar
  • 101
0 votes
1 answer
65 views

Frameworks vs models

My second question (Models vs algorithms) is what is the difference between models and frameworks?
quanity's user avatar
  • 101
0 votes
1 answer
920 views

Dropping dataframe columns with missing values

When I looked up different sources that talk about dropping the columns of a dataframe that contains missing values, I got answers that are starkly different from one another. Some sources say, ...
Apoorva's user avatar
  • 297
0 votes
1 answer
31 views

Finding the labels of a shuffled pandas dataframe

I have a dataframe (namely 'original') with over a thousand numerical columns as features and one label column. I also have a shuffled version of the same dataframe (namely 'shuffled') with label ...
asmani's user avatar
  • 101
0 votes
0 answers
24 views

How to parse the data from the json file describing via dataframe

This is the link to the json file I want to parse the data (resolve [a sentence] into its components and describe their syntactic roles.) Form this file into a dataframe describing: keys as columns ...
Charlie Max's user avatar
0 votes
1 answer
571 views

What is the use of using Vector databases over Pandas Dataframes?

My question is why can't we use a pandas dataframe to do all the data retrieval? Can someone give reasons apart from the fact that databases has the persistence of Data?
Denuwan Weerarathne's user avatar
0 votes
0 answers
73 views

content wise operation in polars series?

i have a dataframe with 2 series, i want to create new column that has 1 if if start word of array one series is equal to start word of another series else 0. How do I do it? i am using polars, it ...
tikendraw's user avatar
  • 135
0 votes
2 answers
122 views

How to filter out redundant id rows with deviant values

First post, so thank you in advance for any pointers! I have a dataframe with an issue similar to the following: id value 10 25.03 11 26.00 11 0.95 12 32.95 13 34.43 13 1.02 One can see ...
OldDogNewTrick's user avatar
0 votes
0 answers
260 views

what is the best practice to handle carriage returns when exporting data from pandas dataframe to CSV?

While trying to export data from a pandas dataframe to a CSV, we noticed that carriage returns (inserted at the middle of a line or at the end of a line) produce outputs that are different from ...
Daniel S.'s user avatar
  • 101
2 votes
2 answers
105 views

Why does pandas.dataframe.info() not update when I delete the first row of a dataset?

I am working with a dataset that comes in with nonsense field names in the first row, with the actual field names in the second row. Currently I'm using this script: ...
SchrodingersStat's user avatar
0 votes
1 answer
37 views

how to prepare pandas dataframe from Solar Flare Data Set?

I am looking to download and use Solar Flare Data Set (https://archive.ics.uci.edu/ml/datasets/Solar+Flare) but unable to figure out how to get the pandas dataframe from the file that is being ...
Sachin's user avatar
  • 101
0 votes
1 answer
36 views

Build a dataframe with multiple CSV

I am collecting metrics on 6 REST services from a microservices architecture. For each instant collected, I extract two CSV from each service. One CSV contains three latency metrics (99th quantile, ...
user2989745's user avatar
0 votes
1 answer
687 views

XGB DataFrame.dtypes error

Here is the code to assign the variables ...
Nata's user avatar
  • 3
0 votes
1 answer
597 views

Append prediction of tensorflow to a pandas dataframe

I built a tensorflow model to make text classification in four category, after testing and evaluating it, I need to apply it to actual data to predict the class of them, I create a predict function ...
Ali A. Jalil's user avatar
0 votes
1 answer
20 views

Combination of groupby and mean methods

I am looking at the below csv file : We have the question : Display the mean of the variable gre by group of admitted/not admitted students, using the combination of groupby and mean methods. I would ...
evinda's user avatar
  • 101
0 votes
2 answers
258 views

Average 5 DataFrames with same index and columns?

I have 5 Dataframe with same index and columns . but the numbers is them is different , I want to average it out . Is there any way to do that ?
tikendraw's user avatar
  • 135
0 votes
1 answer
4k views

select rows containing max value basing on another duplicated rows of a group

I want to select rows by the maximum values of another column which would be the duplicated rows containing duplicated maximum values of a group. This should contain three steps: (1) group dataframe ...
Jiao's user avatar
  • 103
0 votes
1 answer
14k views

ValueError: Could not interpret value `female_height` for parameter `x` when trying to plot certain rows

I have a big dataframe that consists of male and female shoe sizes. I am currently trying to plot size versus height for male and female separately. I have tried to slice the data and stored it in ...
royalTiger's user avatar
0 votes
1 answer
633 views

Using Simple imputer replace NaN values with mean error

I am trying to replace 2 missing NaN values in data using the SimpleImputer. I load my data as follow; ...
royalTiger's user avatar
0 votes
0 answers
511 views

Creating a scatter plot with matplotlib with large dataset

I am trying to create a scatter plot from instrument peak area data recorded every tenth of a second or so. For some reason, when I try to plot it with Matplotlib, I get a straight line, which is not ...
Peter Bright's user avatar
0 votes
1 answer
1k views

Append Existing Columns to another Column in Pandas Dataframe

I have a data that looks like this: The T2M indicates the temperature, and the followed row is the year. I want to append all the similar parameters columns under a single column having all the years,...
Mohamed Abduljawad's user avatar
0 votes
1 answer
30 views

Transforming pandas dataframe from multiple series to categorical label

I have a dataframe in set up in this way: depth val1 val2 1 0.5 0.6 2 0.4 0.7 3 0.3 0.8 and I want to get it into this layout: depth y category 1 0.5 val1 1 0.6 val2 2 0.4 val1 2 0.7 val2 3 ...
artificialnull's user avatar
0 votes
1 answer
345 views

Cumsum starting value not 0 in Pandas

I have the following dataframe df of athletes (indexed by Athlete_ID) and their Rank (indexed by ...
borcherds's user avatar
0 votes
1 answer
2k views

Set column names to column value in pandas forloop, add prefix to column names

I have a dataframe with repeat column names, and I am trying to add a prefix label to column names with an appropriate identifier that comes from a column within the dataframe. I have this data frame: ...
Peter Bright's user avatar
0 votes
0 answers
2k views

How to correctly create a PyTorch Tensor from a Pandas DataFrame?

I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data. Obviously, This new tensor do ...
EvilRoach's user avatar
  • 101
0 votes
2 answers
840 views

How to Insert a Row before Dataframe in Pandas

I have a csv file that is used as a pandas dataframe, now I only need to insert a dummy row before the dataframe starts like in the screenshot denoted as "Label". How can I do that?
Mohamed Abduljawad's user avatar
0 votes
3 answers
5k views

How to upload a Pandas DataFrame into a Postgres table in Airflow?

I need to create a task in Airflow which downloads data from one Postgres database and uploads it to another. My code looks like this: ...
frogseer's user avatar
1 vote
1 answer
78 views

do inner functions have a substantial impact when used in series.apply() in Pandas

In pandas, if I use series.apply() to apply a function with an inner function definition, for example: ...
Alex's user avatar
  • 13
0 votes
2 answers
2k views

How to plot categorical variables with a pie chart

I am concerned with a single column (fruit) from my df: ...
Tom Bomer's user avatar
2 votes
2 answers
2k views

nested xml to dataframe [closed]

I am trying to convert the below mentioned sample xml file to a pandas dataframe. I have multiple xml files which I will loop over to add all xml data into a single dataframe once i succeed with this ...
Haider's user avatar
  • 31
1 vote
3 answers
102 views

Pandas Dataframe grouping and summarizing

I have the following pandas dataframe: df_1: ...
Ali A. Jalil's user avatar
0 votes
1 answer
2k views

Str.contains and isin function do not return all correct rows of dataframe [closed]

Given a list of strings L1 L1 = ['a', 'b', 'c'], I need to extract the rows which contain the values given in list L1. I used the isin function: ...
kanha's user avatar
  • 23
-1 votes
1 answer
146 views

Removing rows with total number of ID occurred not equal to a specific number in Pandas Python [closed]

I have the following data frame in Pandas: ...
Ishigami's user avatar
  • 101
0 votes
1 answer
2k views

How to fix ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am using pytest to test a function that reads a csv into a dictionary of lists. In the test function it writes a sample dictionary to a csv then calls the function to read it back out. To accomplish ...
Michael Wallentine's user avatar
0 votes
1 answer
14 views

Conditional multiplicaiton of two dataFrames [closed]

I have a first dataFrame which looks like this ...
Meth's user avatar
  • 101
1 vote
2 answers
515 views

python compare 2 dataframes [closed]

in reference to this post, python-pandas-compare-columns-in-separate-dataframes-then-delete-non-matching, I can't get the isin function to work with my 2 example dataframes : df 1 : ...
iulianpn's user avatar
0 votes
1 answer
359 views

Seaborn Heatmap with month & hour of database entry

I have a dataframe with X rows. For each row, I have the information of the month (value from 1 to 12) and hour (value from 1 to 24) in separate columns I need to create a heatmap with seaborn in ...
vib's user avatar
  • 103
0 votes
1 answer
911 views

Turning multiple binary columns into categorical (with less columns) with Python Pandas

I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
Legna's user avatar
  • 21
2 votes
1 answer
469 views

Dataframe Python - Conditional Column based on multiple criteria

I want someone to correct my code in python. My goal is to create a code that will add a new column that is based on the conditional function of two columns in my dataframe. I want to add a fees ...
Mostafa Fakhra's user avatar
0 votes
2 answers
2k views

Create new rows based on a value in a column

My dateset is generated like the example df = {'event':['A','B','C','D'], 'budget':['123','433','1000','1299'], 'duration_days':['6','3','4','2']} I need ...
conradoov's user avatar
0 votes
1 answer
2k views

Plot multiple time series from single dataframe

I have a dataframe with multiple time series and columns with labels. My goal is to plot all time series in a single plot, where the labels should be used in the legend of the plot. The important ...
Davgrabner's user avatar
0 votes
0 answers
13 views

Identify the existence of a wall like structure in a given int array

The idea is that I will query an API endpoint which will return me an array consisting of a price value and a quantity value [price, quantity]. In this dataset ...
deniz's user avatar
  • 1
0 votes
1 answer
4k views

For each unique value in a column, count respective unique values in another column

I have a set of tabular data (e.g. csv) representing accesses to a server through a specific protocol . The data follows this format: ...
Mafalda Hopkirk's user avatar
0 votes
1 answer
2k views

How to set a new numeric index

I have read data from a json file, and the dataframe looks like: Col1 Col2 Col3 A: 1 Hi 7 B: 22 CC 1 I think it is because the file was a ...
Karl 17302's user avatar
1 vote
0 answers
131 views

How to solve this IndexError? [closed]

I have created a training dataframe Traindata as following: dataFile='/content/drive/Colab Notebooks/.../Normal_Anomalous_8Digits.csv' data8=pd.read_csv(dataFile) ...
TariqS's user avatar
  • 21

1
2 3 4 5
8