Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
1answer
30 views

Problem with sort by and group by in pandas

I have a DataFrame with duplicate id's but have different dates. For Example: ...
0
votes
1answer
23 views

How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]

I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are: A <...
0
votes
2answers
44 views

Create a new column in Pandas Dataframe based on the 'NaN' values in another column [closed]

What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large) ...
1
vote
2answers
23 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
0
votes
0answers
16 views

Deriving columns and filling them with existing data of the column values [closed]

hello guys need help in solving the problem, in the given image I have a column named as the filename which has more than 10000 rows basically there are CSV file names. I want to create columns based ...
1
vote
2answers
72 views

Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode

Hopefully, you don't mind me posting a question here instead of the regular stack exchange forum. I'm learning ML basics and practising pandas. assuming the data frame is called df, column name = ...
1
vote
3answers
116 views

creating a csv file from two csv files

I have two csv files: sub_compiler.to_csv('sub_compiler.csv') sub_compiler.head() and ...
0
votes
1answer
27 views

Merging two pandas dataframes [duplicate]

I have two csv files,and if I do: df_compiler = pd.read_csv('sub_compiler.csv') I obtain: and for the other csv file I do: ...
0
votes
0answers
5 views

Error condition has length > 1 and only the first element will be used [migrated]

I am writing a function so that I can count the number of NAs in the columns I am interested in. Below, pts_au is a balanced panel data set of country-years, ...
1
vote
1answer
22 views

Changing the value of a column based on the condition on another column

I have a table whose header looks like this: complaint_type borough street_name incident_zip latitude longitude I want to check if the "incident_zip" column of each row is in a specific ...
2
votes
1answer
30 views

Get elements from lists in pandas dataframe [closed]

I have the following column of a data frame: I get it by doing dataFrame['opcodes]. and I would like to consider only the first 20 and the last 20 elements of ...
0
votes
2answers
36 views

use .apply() function to change values to a column of the dataframe

I have a dataframe which is the following: and I would like to consider only the column of instructions and keep just the values push, test, mov, test ,....., so just the first word of each string ...
-1
votes
2answers
154 views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
0
votes
0answers
28 views

Convert column of float valus to time format using panda

I have a data frame with a column of float data (e.g. 1.15 or 12.50).The first number represent hour and the digit represent minutes. I want to format it in order to have 4 numbers by adding 0 to ...
-1
votes
1answer
71 views
2
votes
0answers
28 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
0
votes
0answers
34 views

Empty Dataframe, Columns [], Index [], not sure how to resolve

The goal for this application is to read stock tickers from excel to python, The information reads and prints great, however when I run the program that reads this 'Tickers' data frame, I obtain this: ...
0
votes
1answer
32 views

Join the List and the Data frame for matching list elements in Python

I have a list =[a,b] and Dataframe as below If the text in the 'Text column' matches with List, then need to add additional column matchedItem So output should be Please suggest on this..How ...
0
votes
1answer
21 views

generate normally distributed dataframe with 3 columns

I am trying to generate a normally distributed data frame in python with 3 columns. Each columns requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate ...
-1
votes
1answer
183 views

Finding number of common elements between different columns of a DataFrame

I have a dataframe which has almost 70-80 columns. Each column consists of 100-150 rows in which values are stored as strings. I would like to find, for each column, what is the number of common ...
0
votes
3answers
58 views

Compare dataframes columns

...
1
vote
3answers
60 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
1answer
25 views

how to create multiple plot from a panda Dataframe

I want to plot multiple plots. The data is stored in a pandas dataframe and each row should be a seperate plot. Each row has an ID (ZRD_ID) which doenst matter and a date (TAG) and 24 values to be ...
0
votes
2answers
529 views

Delete/Drop only the rows which has all values as NaN in pandas [closed]

I have a Dataframe, i need to drop the rows which has all the values as NaN. ID Age Gender 601 21 M 501 NaN F NaN NaN NaN The ...
1
vote
0answers
31 views

visualize images inside the Dataframe [closed]

I use python 3.7.3 and I run this code on Jupyter Notebook . I have a column named 'image' in my Dataframe named df that contains Url of images I wanna visualize inside the df. I saw some answers ...
0
votes
1answer
2k views

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

Firstly I have a pandas series of recommended product (recmd_prdt_list). In this series there is a possibility of presence of deleted products. So as to remove deleted products from the recommended ...
0
votes
1answer
86 views

How to change a Pandas dataframe into feature vector?

I have a Pandas dataframe with 10 columns, 9 of which are features to be used to predict the 10th column. How is it ossible to convert this Pandas dataframe into X and y vectors to use in a linear ...
2
votes
1answer
180 views

How to shuffle only a fraction of a column in a Pandas dataframe?

I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe. How would you do it? Is there a simple idiomatic way to do that, maybe using ...
1
vote
1answer
34 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
0
votes
1answer
40 views

Make column as dictionary key and row as value in pandas dataframe

I have Dataframe as below. name value id A 123 1 B 345 5 C 567 4 D 789 2 I need to create a ...
1
vote
1answer
27 views

very important features, but rarely input

The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would ...
0
votes
1answer
23 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
0
votes
0answers
35 views

sorting pyspark dataframe

I'm sorting a pyspark DataFrame in databricks by df.sort(desc('x')). However, when I look at the DataFrame by display(df), I don't see the DataFrame being sorted in 'x'. What could be the reason?
0
votes
0answers
33 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
0
votes
1answer
40 views

How can I arrange data in different columns (for different constituents) from one?

I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want ...
1
vote
1answer
65 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
0
votes
2answers
28 views

Find columns with numeric values, but stored as string

I need to find the columns in data frame, which has numeric values and are stored as string. ...
0
votes
1answer
69 views

Why we add a constant value column in our DataFrame sometimes?

Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1. I need to ...
1
vote
1answer
336 views

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
1
vote
1answer
42 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
0
votes
0answers
18 views

Dataframe header colouring doesn't work

I would like to construct dataframe whos header has specified colour. I tried the following code but for some unknown reason it doesn't work. Any recommendations to solve this task? ...
0
votes
1answer
313 views

Creating new column in dataframe based on conditions in 2 other columns [closed]

I would like to create a new column in my dataframe based on values from both the gender and experimental_grouping columns. As I have it written below, the column ...
0
votes
1answer
383 views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...
0
votes
0answers
26 views

How to compare genetic profiles or vcf files in Python?

I have hundreds of vcf file where each vcf file contains genome profile for a tissue. A portion of the vcf file is as follows: I can read each vcf file into a dataframe. So it would be hundreds of ...
1
vote
2answers
524 views

Binary encoding and its interpretation in Python

I have a column named Street that has 2 values: Paved and Gravel. Here is what print(train[binary_columns[0]].unique().tolist())...
2
votes
1answer
193 views

Change values of a particular column to value_count()

Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a ...
0
votes
2answers
682 views

Removing duplicates and keeping the last entry in pandas

I have a table and I'm trying to remove all the duplicate and keep the the rows that has the latest datestamp. Here is the table: ...
0
votes
0answers
229 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
0answers
306 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
0
votes
1answer
312 views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...