Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
1
vote
2answers
2k views

Binary encoding and its interpretation in Python

I have a column named Street that has 2 values: Paved and Gravel. Here is what print(train[binary_columns[0]].unique().tolist())...
2
votes
1answer
762 views

Change values of a particular column to value_count()

Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a ...
0
votes
2answers
4k views

Removing duplicates and keeping the last entry in pandas

I have a table and I'm trying to remove all the duplicate and keep the the rows that has the latest datestamp. Here is the table: ...
0
votes
0answers
479 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
0answers
414 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
1
vote
2answers
1k views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
2
votes
0answers
31 views

Calculate a mean on condtions in a dataframe with dyplr

My table looks like this: ...
3
votes
1answer
181 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
0
votes
3answers
2k views

Remove columns with a certain number of consecutive zeros

I'm using pandas and I am dealing with time series of sales. What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...
4
votes
1answer
167 views

Pandas dataframe with multiple hierarchical indices

I have a data frame which looks like this ...
0
votes
1answer
380 views

Segmenting data in a Dataframe and assigning order numbers (Python using Pandas)

I have a large Excel data set which I load as Pandas. In one column I have repeating strings/numbers (the information can be a text and or integers), in another column I have a unique identifier of ...
1
vote
1answer
951 views

Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?

I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same ...
7
votes
2answers
101k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
1
vote
2answers
660 views
0
votes
1answer
567 views

How to transform entire pandas data frame in one hot representation?

I want all the columns one hot encoded without the need of listing out the columns or apply one hot encode one by one. I know how to do it one column then another.
0
votes
1answer
2k views

How to set limits of Y-axes in countplot?

df in my program happens to be a dataframe with these columns : ...
8
votes
2answers
29k views

Pandas merge column duplicate and sum value [closed]

How to merge duplicate column and sum their value? What I have A 30 A 40 B 50 What I need A 70 B 50 DF for this ...
0
votes
2answers
5k views

How to fill an missing values in a column based on another column

I have a dataframe called shoes ...
0
votes
1answer
842 views

How to Convert a Pandas Column having duration details in string format (ex:1hr 50m) into a integer column with value in minutes

Lets say i have the following data like below: ...
4
votes
1answer
5k views

Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

I want to store JSON Data into MySQL Database using Python. I used dataframe of pandas. I found that ...
1
vote
2answers
4k views

Is there a way to drop duplicated rows based on an unhashable column?

i have a pandas dataframe df with one column z filled with set values i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values (...
8
votes
4answers
41k views

How to Write Multiple Data Frames in an Excel Sheet

I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. ...
3
votes
1answer
22k views

dataframe.columns.difference() use

I am trying to find the working of dataframe.columns.difference() but couldn't find a satisfactory explanation about it. Can anyone explain the working of this ...
2
votes
2answers
809 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
1
vote
2answers
13k views

'DataFrame' object has no attribute 'to_dataframe'

I'm sure I have a small error here that I'm overlooking, but am having a tough time figuring out what I need to change. Here is my code up until the error I'm getting. ...
3
votes
2answers
712 views

Inputting (a lot of )data into a dataframe one row at a time

I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this: ...
2
votes
1answer
311 views

Breaking a DataFrame based on Specific Column Values [closed]

Assume I have a following dataframe which has years of experience and salary. I would like to create a new dataframe from the existing one, only including observations where salary is above 45000. I ...
1
vote
1answer
7k views

Numpy array from pandas dataframe

I am new in using python for data science. What is the difference between selecting a a column with: df['name'].values and ...
1
vote
1answer
707 views

Divide a column by itself with mutate_at dplyr

Hi I'd like to turn each non zero value of my selected columns to a 1 using mutate_at() ...
-1
votes
1answer
73 views

LabelEncoding a Dataframe

I have a dataframe with integer and categorical variables. Should I label encode all variables (both integer and categorical) or should I encode only the categorical features?
1
vote
1answer
7k views

how to update column in data frame based on condition

How to update column IsLCap column in dataframe based on Lvalue column whether it is capitilized or not. df.loc[df.Lvalue.istitle(), 'IsLCap'] = 1 # need to be corrected getting an error -...
1
vote
1answer
267 views

Eliminating rows in a dataframe based on specific conditions

I have two columns in a dataframe namely 'origin' and 'destination' which contains the names of different places. I need to remove the rows that don't contain 'PHX', 'JFK', 'NTU' in either of these ...
0
votes
2answers
1k views

How to split data frame into groups, combine rows

I have a large data set with 405 columns, many rows, and data from 15 sites. Each site has 27 columns, each one one quadrats data. Rows are species data. I would like to split the data into the 15 ...
0
votes
2answers
4k views

How to create a new column based on two other columns in Pandas?

I am searching for a way to create a new column in my data. I have tried using iterows() but found it extremely time consuming in my dataset containing 40 lakh rows. So here is what I want. Consider I ...
2
votes
2answers
149 views

Pandas: Assign back to table from grouping by column and index

I am trying to implement Exponential Moving Average calculation on a DataFrame. The formula is An additional complication is that my table is grouped and there is ...
1
vote
0answers
97 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
0
votes
1answer
205 views

How can I extract bootstrap generated datasets into individual dataframes?

I am having a bit of trouble understanding Bootstrapping and what/how I can manipulate the bootstrap created dataset. This is all in R My original dataset is structured like this: ...
0
votes
1answer
20k views

Joining two dataframes on the basis of specific conditions [closed]

I've added the images of both the dataframes here. Now I need to combine the two dataframes on the basis of two conditions: Condition 1: The element in the 'arrivalTS' column in the first dataframe(...
1
vote
3answers
4k views

Mutate with custom function in R does not work

I have a data frame, containing a column called: "Frequency". Frequency has values like "Year", "Week", "Month" etc. Now I want to create a new column based on the Frequency column where Year's new ...
3
votes
1answer
2k views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
9
votes
2answers
30k views

How to rename columns that have the same name? [closed]

I would like to rename the column names, but the Data Frame contains similar column names. How do I rename them? df.columns Output: ...
0
votes
1answer
43 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
6
votes
1answer
13k views

How to find the count of consecutive same string values in a pandas dataframe? [closed]

Assume that we have the following pandas dataframe: ...
1
vote
0answers
32 views

Best way to find dissimilarity in a 6x2 DataFrame?

I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists. The goal is to find the most ...
3
votes
1answer
2k views

How can I combine two single-column datasets into a single Pandas data frame?

I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label ...
2
votes
2answers
104 views

How to group this dataframe in python?

I have this problem: ...
0
votes
1answer
4k views

How can I merge 2+ DataFrame objects without duplicating column names?

This is for work. TLDR: Bottom-line question at the bottom. I am gathering and parsing test results produced by an old test setup whose output formatting is not likely to change anytime soon. I've ...
10
votes
2answers
86k views

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. ...
0
votes
1answer
1k views

Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?

I am running the Kaggle Video Games sales dataset through an XGboost algo. I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5 when I use: data = pd.get_dummies(data=...
1
vote
2answers
1k views

Theoretical Question: Data.table vs Data.frame with Big Data

I know that I can read in a very large csv file much faster with fread using the data.table ...