Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
1
vote
2answers
20 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
0
votes
0answers
12 views

Deriving columns and filling them with existing data of the column values

hello guys need help in solving the problem, in the given image I have a column named as the filename which has more than 10000 rows basically there are CSV file names. I want to create columns based ...
1
vote
2answers
29 views

Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode

Hopefully, you don't mind me posting a question here instead of the regular stack exchange forum. I'm learning ML basics and practising pandas. assuming the data frame is called df, column name = ...
1
vote
3answers
108 views

creating a csv file from two csv files

I have two csv files: sub_compiler.to_csv('sub_compiler.csv') sub_compiler.head() and ...
0
votes
1answer
26 views

Merging two pandas dataframes [duplicate]

I have two csv files,and if I do: df_compiler = pd.read_csv('sub_compiler.csv') I obtain: and for the other csv file I do: ...
0
votes
0answers
5 views

Error condition has length > 1 and only the first element will be used [migrated]

I am writing a function so that I can count the number of NAs in the columns I am interested in. Below, pts_au is a balanced panel data set of country-years, ...
1
vote
1answer
22 views

Changing the value of a column based on the condition on another column

I have a table whose header looks like this: complaint_type borough street_name incident_zip latitude longitude I want to check if the "incident_zip" column of each row is in a specific ...
2
votes
1answer
30 views

Get elements from lists in pandas dataframe [closed]

I have the following column of a data frame: I get it by doing dataFrame['opcodes]. and I would like to consider only the first 20 and the last 20 elements of ...
0
votes
2answers
34 views

use .apply() function to change values to a column of the dataframe

I have a dataframe which is the following: and I would like to consider only the column of instructions and keep just the values push, test, mov, test ,....., so just the first word of each string ...
0
votes
2answers
70 views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
0
votes
0answers
28 views

Convert column of float valus to time format using panda

I have a data frame with a column of float data (e.g. 1.15 or 12.50).The first number represent hour and the digit represent minutes. I want to format it in order to have 4 numbers by adding 0 to ...
-1
votes
1answer
49 views
2
votes
0answers
26 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
0
votes
0answers
30 views

Empty Dataframe, Columns [], Index [], not sure how to resolve

The goal for this application is to read stock tickers from excel to python, The information reads and prints great, however when I run the program that reads this 'Tickers' data frame, I obtain this: ...
0
votes
1answer
31 views

Join the List and the Data frame for matching list elements in Python

I have a list =[a,b] and Dataframe as below If the text in the 'Text column' matches with List, then need to add additional column matchedItem So output should be Please suggest on this..How ...
0
votes
1answer
21 views

generate normally distributed dataframe with 3 columns

I am trying to generate a normally distributed data frame in python with 3 columns. Each columns requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate ...
-1
votes
1answer
133 views

Finding number of common elements between different columns of a DataFrame

I have a dataframe which has almost 70-80 columns. Each column consists of 100-150 rows in which values are stored as strings. I would like to find, for each column, what is the number of common ...
0
votes
3answers
50 views

Compare dataframes columns

...
1
vote
3answers
39 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
1answer
25 views

how to create multiple plot from a panda Dataframe

I want to plot multiple plots. The data is stored in a pandas dataframe and each row should be a seperate plot. Each row has an ID (ZRD_ID) which doenst matter and a date (TAG) and 24 values to be ...
0
votes
2answers
360 views

Delete/Drop only the rows which has all values as NaN in pandas [closed]

I have a Dataframe, i need to drop the rows which has all the values as NaN. ID Age Gender 601 21 M 501 NaN F NaN NaN NaN The ...
1
vote
0answers
30 views

visualize images inside the Dataframe [closed]

I use python 3.7.3 and I run this code on Jupyter Notebook . I have a column named 'image' in my Dataframe named df that contains Url of images I wanna visualize inside the df. I saw some answers ...
0
votes
1answer
1k views

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

Firstly I have a pandas series of recommended product (recmd_prdt_list). In this series there is a possibility of presence of deleted products. So as to remove deleted products from the recommended ...
0
votes
1answer
72 views

How to change a Pandas dataframe into feature vector?

I have a Pandas dataframe with 10 columns, 9 of which are features to be used to predict the 10th column. How is it ossible to convert this Pandas dataframe into X and y vectors to use in a linear ...
2
votes
1answer
133 views

How to shuffle only a fraction of a column in a Pandas dataframe?

I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe. How would you do it? Is there a simple idiomatic way to do that, maybe using ...
1
vote
1answer
28 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
0
votes
1answer
37 views

Make column as dictionary key and row as value in pandas dataframe

I have Dataframe as below. name value id A 123 1 B 345 5 C 567 4 D 789 2 I need to create a ...
1
vote
1answer
27 views

very important features, but rarely input

The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would ...
0
votes
1answer
23 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
0
votes
0answers
34 views

sorting pyspark dataframe

I'm sorting a pyspark DataFrame in databricks by df.sort(desc('x')). However, when I look at the DataFrame by display(df), I don't see the DataFrame being sorted in 'x'. What could be the reason?
0
votes
0answers
33 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
0
votes
1answer
39 views

How can I arrange data in different columns (for different constituents) from one?

I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want ...
1
vote
1answer
61 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
0
votes
2answers
25 views

Find columns with numeric values, but stored as string

I need to find the columns in data frame, which has numeric values and are stored as string. ...
0
votes
1answer
53 views

Why we add a constant value column in our DataFrame sometimes?

Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1. I need to ...
1
vote
1answer
265 views

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
1
vote
1answer
41 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
0
votes
0answers
18 views

Dataframe header colouring doesn't work

I would like to construct dataframe whos header has specified colour. I tried the following code but for some unknown reason it doesn't work. Any recommendations to solve this task? ...
0
votes
1answer
195 views

Creating new column in dataframe based on conditions in 2 other columns [closed]

I would like to create a new column in my dataframe based on values from both the gender and experimental_grouping columns. As I have it written below, the column ...
0
votes
1answer
295 views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...
0
votes
0answers
23 views

How to compare genetic profiles or vcf files in Python?

I have hundreds of vcf file where each vcf file contains genome profile for a tissue. A portion of the vcf file is as follows: I can read each vcf file into a dataframe. So it would be hundreds of ...
1
vote
2answers
435 views

Binary encoding and its interpretation in Python

I have a column named Street that has 2 values: Paved and Gravel. Here is what print(train[binary_columns[0]].unique().tolist())...
2
votes
1answer
153 views

Change values of a particular column to value_count()

Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a ...
0
votes
2answers
578 views

Removing duplicates and keeping the last entry in pandas

I have a table and I'm trying to remove all the duplicate and keep the the rows that has the latest datestamp. Here is the table: ...
0
votes
0answers
201 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
0answers
292 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
0
votes
1answer
290 views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
2
votes
0answers
25 views

Calculate a mean on condtions in a dataframe with dyplr

My table looks like this: ...
2
votes
1answer
67 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
0
votes
1answer
224 views

Remove columns with a certain number of consecutive zeros

I'm using pandas and I am dealing with time series of sales. What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...