Questions tagged [dataframe]
A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.
353
questions
0
votes
1
answer
36
views
Frameworks vs models
My second question (Models vs algorithms) is what is the difference between models and frameworks?
- 101
0
votes
1
answer
64
views
Dropping dataframe columns with missing values
When I looked up different sources that talk about dropping the columns of a dataframe that contains missing values, I got answers that are starkly different from one another.
Some sources say, ...
- 275
0
votes
1
answer
30
views
Finding the labels of a shuffled pandas dataframe
I have a dataframe (namely 'original') with over a thousand numerical columns as features and one label column. I also have a shuffled version of the same dataframe (namely 'shuffled') with label ...
- 101
0
votes
0
answers
19
views
How to parse the data from the json file describing via dataframe
This is the link to the json file I want to parse the data (resolve [a sentence] into its components and describe their syntactic roles.)
Form this file into a dataframe describing:
keys as columns
...
0
votes
1
answer
190
views
What is the use of using Vector databases over Pandas Dataframes?
My question is why can't we use a pandas dataframe to do all the data retrieval? Can someone give reasons apart from the fact that databases has the persistence of Data?
0
votes
0
answers
59
views
content wise operation in polars series?
i have a dataframe with 2 series, i want to create new column that has 1 if if start word of array one series is equal to start word of another series else 0.
How do I do it?
i am using polars, it ...
- 135
0
votes
2
answers
53
views
How to filter out redundant id rows with deviant values
First post, so thank you in advance for any pointers! I have a dataframe with an issue similar to the following:
id value
10 25.03
11 26.00
11 0.95
12 32.95
13 34.43
13 1.02
One can see ...
0
votes
0
answers
84
views
what is the best practice to handle carriage returns when exporting data from pandas dataframe to CSV?
While trying to export data from a pandas dataframe to a CSV, we noticed that carriage returns (inserted at the middle of a line or at the end of a line) produce outputs that are different from ...
- 101
2
votes
2
answers
84
views
Why does pandas.dataframe.info() not update when I delete the first row of a dataset?
I am working with a dataset that comes in with nonsense field names in the first row, with the actual field names in the second row.
Currently I'm using this script:
...
- 165
0
votes
1
answer
30
views
how to prepare pandas dataframe from Solar Flare Data Set?
I am looking to download and use Solar Flare Data Set (https://archive.ics.uci.edu/ml/datasets/Solar+Flare) but unable to figure out how to get the pandas dataframe from the file that is being ...
- 101
0
votes
0
answers
20
views
Running all cells again adding new rows instead of return new dataframe
i am creating a class just like lazypredict library but with k fold support and return a data frame withh the mean score of every model every thing works and i can ...
- 101
0
votes
1
answer
36
views
Build a dataframe with multiple CSV
I am collecting metrics on 6 REST services from a microservices architecture. For each instant collected, I extract two CSV from each service. One CSV contains three latency metrics (99th quantile, ...
- 101
0
votes
1
answer
516
views
0
votes
0
answers
22
views
How to structure dataframe combinations for regression, without corruption/loss?
I have a data set, redacted sample below. My goal is linear regression. My question is: Have I created unintended results, due to how I structured the df, using concat and/or div?
For example, ...
- 11
0
votes
1
answer
454
views
Append prediction of tensorflow to a pandas dataframe
I built a tensorflow model to make text classification in four category, after testing and evaluating it, I need to apply it to actual data to predict the class of them, I create a predict function ...
- 121
0
votes
1
answer
19
views
Combination of groupby and mean methods
I am looking at the below csv file :
We have the question :
Display the mean of the variable gre by group of admitted/not admitted students, using the combination of groupby and mean methods.
I would ...
- 101
0
votes
0
answers
11
views
How to storge large data values in data frame or what would be suitable way?
My output has two columns, basically one is a token name and the second column is the embedding which is the array consisting of 300 values, if I store these tables in CSV files, most of the array ...
0
votes
0
answers
14
views
'molten' versus truly 'tidy' dataframe
I am currently 'tidying' up a dataset related to a city's migration statistics across three characteristics: year, nationality and gender, i.e. how many nationals versus non-nationals by gender have ...
- 1
0
votes
2
answers
177
views
Average 5 DataFrames with same index and columns?
I have 5 Dataframe with same index and columns . but the numbers is them is different , I want to average it out . Is there any way to do that ?
- 135
0
votes
0
answers
23
views
How to reference an intermediary object in scala?
I am using spark/scala to do some transformations on a spark dataframe. I have the following code:
...
- 11
0
votes
0
answers
19
views
Extracting variables from different properties with similar timestamps -- PySpark
I have a list of variables I need to extract. I am using the PySpark API. Almost all variables are already in the same property, but just one variable I also need was logged in another property. For ...
- 101
0
votes
1
answer
3k
views
select rows containing max value basing on another duplicated rows of a group
I want to select rows by the maximum values of another column which would be the duplicated rows containing duplicated maximum values of a group.
This should contain three steps:
(1) group dataframe ...
- 103
0
votes
1
answer
12k
views
ValueError: Could not interpret value `female_height` for parameter `x` when trying to plot certain rows
I have a big dataframe that consists of male and female shoe sizes. I am currently trying to plot size versus height for male and female separately. I have tried to slice the data and stored it in ...
0
votes
1
answer
427
views
Using Simple imputer replace NaN values with mean error
I am trying to replace 2 missing NaN values in data using the SimpleImputer.
I load my data as follow;
...
0
votes
0
answers
398
views
Creating a scatter plot with matplotlib with large dataset
I am trying to create a scatter plot from instrument peak area data recorded every tenth of a second or so. For some reason, when I try to plot it with Matplotlib, I get a straight line, which is not ...
- 11
0
votes
1
answer
780
views
Append Existing Columns to another Column in Pandas Dataframe
I have a data that looks like this:
The T2M indicates the temperature, and the followed row is the year. I want to append all the similar parameters columns under a single column having all the years,...
0
votes
1
answer
25
views
Transforming pandas dataframe from multiple series to categorical label
I have a dataframe in set up in this way:
depth
val1
val2
1
0.5
0.6
2
0.4
0.7
3
0.3
0.8
and I want to get it into this layout:
depth
y
category
1
0.5
val1
1
0.6
val2
2
0.4
val1
2
0.7
val2
3
...
- 101
0
votes
1
answer
307
views
Cumsum starting value not 0 in Pandas
I have the following dataframe df of athletes (indexed by Athlete_ID) and their Rank (indexed by ...
0
votes
1
answer
1k
views
Set column names to column value in pandas forloop, add prefix to column names
I have a dataframe with repeat column names, and I am trying to add a prefix label to column names with an appropriate identifier that comes from a column within the dataframe. I have this data frame:
...
- 11
0
votes
0
answers
2k
views
How to correctly create a PyTorch Tensor from a Pandas DataFrame?
I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data.
Obviously, This new tensor do ...
- 101
0
votes
2
answers
603
views
How to Insert a Row before Dataframe in Pandas
I have a csv file that is used as a pandas dataframe, now I only need to insert a dummy row before the dataframe starts like in the screenshot denoted as "Label". How can I do that?
0
votes
3
answers
3k
views
How to upload a Pandas DataFrame into a Postgres table in Airflow?
I need to create a task in Airflow which downloads data from one Postgres database and uploads it to another.
My code looks like this:
...
- 1
1
vote
1
answer
73
views
do inner functions have a substantial impact when used in series.apply() in Pandas
In pandas, if I use series.apply() to apply a function with an inner function definition, for example:
...
- 13
0
votes
2
answers
2k
views
How to plot categorical variables with a pie chart
I am concerned with a single column (fruit) from my df:
...
- 13
2
votes
2
answers
2k
views
nested xml to dataframe [closed]
I am trying to convert the below mentioned sample xml file to a pandas dataframe. I have multiple xml files which I will loop over to add all xml data into a single dataframe once i succeed with this ...
- 31
1
vote
3
answers
79
views
0
votes
1
answer
1k
views
Str.contains and isin function do not return all correct rows of dataframe [closed]
Given a list of strings L1 L1 = ['a', 'b', 'c'], I need to extract the rows which contain the values given in list L1. I used the isin function: ...
- 23
-1
votes
1
answer
136
views
Removing rows with total number of ID occurred not equal to a specific number in Pandas Python [closed]
I have the following data frame in Pandas:
...
- 101
0
votes
1
answer
2k
views
How to fix ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
I am using pytest to test a function that reads a csv into a dictionary of lists. In the test function it writes a sample dictionary to a csv then calls the function to read it back out. To accomplish ...
0
votes
1
answer
14
views
Conditional multiplicaiton of two dataFrames [closed]
I have a first dataFrame which looks like this
...
- 101
1
vote
2
answers
473
views
python compare 2 dataframes [closed]
in reference to this post, python-pandas-compare-columns-in-separate-dataframes-then-delete-non-matching, I can't get the isin function to work with my 2 example dataframes :
df 1 :
...
- 11
0
votes
1
answer
293
views
Seaborn Heatmap with month & hour of database entry
I have a dataframe with X rows.
For each row, I have the information of the month (value from 1 to 12) and hour (value from 1 to 24) in separate columns
I need to create a heatmap with seaborn in ...
- 103
0
votes
1
answer
822
views
Turning multiple binary columns into categorical (with less columns) with Python Pandas
I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
- 21
2
votes
1
answer
433
views
Dataframe Python - Conditional Column based on multiple criteria
I want someone to correct my code in python. My goal is to create a code that will add a new column that is based on the conditional function of two columns in my dataframe. I want to add a fees ...
0
votes
2
answers
2k
views
Create new rows based on a value in a column
My dateset is generated like the example
df = {'event':['A','B','C','D'],
'budget':['123','433','1000','1299'],
'duration_days':['6','3','4','2']}
I need ...
0
votes
1
answer
1k
views
Plot multiple time series from single dataframe
I have a dataframe with multiple time series and columns with labels. My goal is to plot all time series in a single plot, where the labels should be used in the legend of the plot. The important ...
0
votes
0
answers
13
views
Identify the existence of a wall like structure in a given int array
The idea is that I will query an API endpoint which will return me an array consisting of a price value and a quantity value [price, quantity].
In this dataset ...
- 1
0
votes
1
answer
3k
views
For each unique value in a column, count respective unique values in another column
I have a set of tabular data (e.g. csv) representing accesses to a server through a specific protocol . The data follows this format:
...
0
votes
1
answer
1k
views
How to set a new numeric index
I have read data from a json file, and the dataframe looks like:
Col1 Col2 Col3
A: 1 Hi 7
B: 22 CC 1
I think it is because the file was a ...
1
vote
0
answers
102
views
How to solve this IndexError? [closed]
I have created a training dataframe Traindata as following:
dataFile='/content/drive/Colab Notebooks/.../Normal_Anomalous_8Digits.csv'
data8=pd.read_csv(dataFile)
...
- 21