Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
1
vote
3answers
17 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
1answer
18 views

how to create multiple plot from a panda Dataframe

I want to plot multiple plots. The data is stored in a pandas dataframe and each row should be a seperate plot. Each row has an ID (ZRD_ID) which doenst matter and a date (TAG) and 24 values to be ...
0
votes
2answers
53 views

Delete/Drop only the rows which has all values as NaN in pandas [closed]

I have a Dataframe, i need to drop the rows which has all the values as NaN. ID Age Gender 601 21 M 501 NaN F NaN NaN NaN The ...
1
vote
0answers
28 views

visualize images inside the Dataframe [closed]

I use python 3.7.3 and I run this code on Jupyter Notebook . I have a column named 'image' in my Dataframe named df that contains Url of images I wanna visualize inside the df. I saw some answers ...
0
votes
1answer
61 views

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

Firstly I have a pandas series of recommended product (recmd_prdt_list). In this series there is a possibility of presence of deleted products. So as to remove deleted products from the recommended ...
0
votes
1answer
18 views

How to change a Pandas dataframe into feature vector?

I have a Pandas dataframe with 10 columns, 9 of which are features to be used to predict the 10th column. How is it ossible to convert this Pandas dataframe into X and y vectors to use in a linear ...
0
votes
0answers
8 views

ValueError: can not merge DataFrame with instance of type <class 'str'> [migrated]

I have two dfs: df_main_lat_lon and df_lat_lat_lon. They are as follows: When I merged these two dfs as: merged_inner = pd.merge('df_tp_lat_lon', 'df_main_lat_lon', on=['LATITUDE', 'LONGITUDE'], ...
2
votes
1answer
37 views

How to shuffle only a fraction of a column in a Pandas dataframe?

I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe. How would you do it? Is there a simple idiomatic way to do that, maybe using ...
1
vote
1answer
19 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
0
votes
1answer
25 views

Make column as dictionary key and row as value in pandas dataframe

I have Dataframe as below. name value id A 123 1 B 345 5 C 567 4 D 789 2 I need to create a ...
1
vote
1answer
27 views

very important features, but rarely input

The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would ...
0
votes
1answer
20 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
0
votes
0answers
21 views

sorting pyspark dataframe

I'm sorting a pyspark DataFrame in databricks by df.sort(desc('x')). However, when I look at the DataFrame by display(df), I don't see the DataFrame being sorted in 'x'. What could be the reason?
0
votes
0answers
30 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
0
votes
1answer
36 views

How can I arrange data in different columns (for different constituents) from one?

I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want ...
1
vote
1answer
35 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
0
votes
2answers
21 views

Find columns with numeric values, but stored as string

I need to find the columns in data frame, which has numeric values and are stored as string. ...
0
votes
1answer
35 views

Why we add a constant value column in our DataFrame sometimes?

Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1. I need to ...
0
votes
1answer
60 views

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
1
vote
1answer
38 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
0
votes
0answers
16 views

Dataframe header colouring doesn't work

I would like to construct dataframe whos header has specified colour. I tried the following code but for some unknown reason it doesn't work. Any recommendations to solve this task? ...
0
votes
1answer
56 views

Creating new column in dataframe based on conditions in 2 other columns [closed]

I would like to create a new column in my dataframe based on values from both the gender and experimental_grouping columns. As I have it written below, the column ...
0
votes
1answer
101 views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...
0
votes
0answers
15 views

How to compare genetic profiles or vcf files in Python?

I have hundreds of vcf file where each vcf file contains genome profile for a tissue. A portion of the vcf file is as follows: I can read each vcf file into a dataframe. So it would be hundreds of ...
1
vote
2answers
210 views

Binary encoding and its interpretation in Python

I have a column named Street that has 2 values: Paved and Gravel. Here is what print(train[binary_columns[0]].unique().tolist())...
2
votes
1answer
58 views

Change values of a particular column to value_count()

Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a ...
0
votes
2answers
192 views

Removing duplicates and keeping the last entry in pandas

I have a table and I'm trying to remove all the duplicate and keep the the rows that has the latest datestamp. Here is the table: ...
0
votes
0answers
103 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
0answers
172 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
0
votes
1answer
167 views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
2
votes
0answers
22 views

Calculate a mean on condtions in a dataframe with dyplr

My table looks like this: ...
2
votes
1answer
58 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
0
votes
1answer
88 views

Remove columns with a certain number of consecutive zeros

I'm using pandas and I am dealing with time series of sales. What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...
0
votes
0answers
38 views

Add Header name without using pandas

I want to add header name without using pandas. In my program I have given two columns in my txt file where header name is missing: ...
2
votes
1answer
58 views

Pandas dataframe with multiple hierarchical indices

I have a data frame which looks like this ...
0
votes
1answer
43 views

Segmenting data in a Dataframe and assigning order numbers (Python using Pandas)

I have a large Excel data set which I load as Pandas. In one column I have repeating strings/numbers (the information can be a text and or integers), in another column I have a unique identifier of ...
0
votes
0answers
27 views

How to graphically present the statistical interpretations when using Kolmogorov–Smirnov test (K-S) test?

I am referring to my previous question (Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?) related to visualising the Pandas dataframes to show that they have ...
1
vote
1answer
82 views

Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?

I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same ...
5
votes
1answer
12k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
0
votes
1answer
88 views
0
votes
1answer
53 views

How to transform entire pandas data frame in one hot representation?

I want all the columns one hot encoded without the need of listing out the columns or apply one hot encode one by one. I know how to do it one column then another.
0
votes
1answer
353 views

How to set limits of Y-axes in countplot?

df in my program happens to be a dataframe with these columns : ...
4
votes
1answer
3k views

Pandas merge column duplicate and sum value

How to merge duplicate column and sum their value? What I have A 30 A 40 B 50 What I need A 70 B 50 DF for this ...
0
votes
2answers
864 views

How to fill an missing values in a column based on another column

I have a dataframe called shoes ...
0
votes
1answer
176 views
3
votes
1answer
2k views

Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

I want to store JSON Data into MySQL Database using Python. I used dataframe of pandas. I found that ...
1
vote
2answers
747 views

Is there a way to drop duplicated rows based on an unhashable column?

i have a pandas dataframe df with one column z filled with set values i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values (...
3
votes
3answers
5k views

How to Write Multiple Data Frames in an Excel Sheet

I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. ...
0
votes
1answer
4k views

dataframe.columns.difference() use

I am trying to find the working of dataframe.columns.difference() but couldn't find a satisfactory explanation about it. Can anyone explain the working of this ...
2
votes
2answers
118 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...