Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

0
votes
0answers
24 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
0answers
9 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
0
votes
1answer
22 views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
2
votes
0answers
19 views
2
votes
1answer
42 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
0
votes
1answer
20 views

Remove columns with a certain number of consecutive zeros

I'm using pandas and I am dealing with time series of sales. What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...
0
votes
0answers
19 views

Add Header name without using pandas

I want to add header name without using pandas. In my program I have given two columns in my txt file where header name is missing: ...
1
vote
1answer
27 views

Pandas dataframe with multiple hierarchical indices

I have a data frame which looks like this ...
0
votes
1answer
27 views

Segmenting data in a Dataframe and assigning order numbers (Python using Pandas)

I have a large Excel data set which I load as Pandas. In one column I have repeating strings/numbers (the information can be a text and or integers), in another column I have a unique identifier of ...
0
votes
0answers
19 views

How to graphically present the statistical interpretations when using Kolmogorov–Smirnov test (K-S) test?

I am referring to my previous question (Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?) related to visualising the Pandas dataframes to show that they have ...
1
vote
1answer
57 views

Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?

I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same ...
4
votes
1answer
2k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
0
votes
1answer
34 views

How to transform entire pandas data frame in one hot representation?

I want all the columns one hot encoded without the need of listing out the columns or apply one hot encode one by one. I know how to do it one column then another.
0
votes
1answer
85 views

How to set limits of Y-axes in countplot?

df in my program happens to be a dataframe with these columns : ...
2
votes
1answer
345 views

Pandas merge column duplicate and sum value

How to merge duplicate column and sum their value? What I have A 30 A 40 B 50 What I need A 70 B 50 DF for this ...
0
votes
2answers
83 views
3
votes
1answer
631 views

Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

I want to store JSON Data into MySQL Database using Python. I used dataframe of pandas. I found that ...
1
vote
2answers
145 views

Is there a way to drop duplicated rows based on an unhashable column?

i have a pandas dataframe df with one column z filled with set values i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values (...
3
votes
3answers
776 views

How to Write Multiple Data Frames in an Excel Sheet

I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. ...
0
votes
1answer
404 views

dataframe.columns.difference() use

I am trying to find the working of dataframe.columns.difference() but couldn't find a satisfactory explanation about it. Can anyone explain the working of this ...
2
votes
2answers
53 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
0
votes
2answers
793 views

'DataFrame' object has no attribute 'to_dataframe'

I'm sure I have a small error here that I'm overlooking, but am having a tough time figuring out what I need to change. Here is my code up until the error I'm getting. ...
2
votes
1answer
109 views

Inputting (a lot of )data into a dataframe one row at a time

I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this: ...
2
votes
1answer
53 views

Breaking a DataFrame based on Specific Column Values [closed]

Assume I have a following dataframe which has years of experience and salary. I would like to create a new dataframe from the existing one, only including observations where salary is above 45000. I ...
1
vote
2answers
426 views

Numpy array from pandas dataframe

I am new in using python for data science. What is the difference between selecting a a column with: df['name'].values and ...
0
votes
1answer
134 views

Divide a column by itself with mutate_at dplyr

Hi I'd like to turn each non zero value of my selected columns to a 1 using mutate_at() ...
-1
votes
1answer
19 views

LabelEncoding a Dataframe

I have a dataframe with integer and categorical variables. Should I label encode all variables (both integer and categorical) or should I encode only the categorical features?
0
votes
0answers
18 views

String Matching columns based on weight

I have the following 2 DataFrames and want to match skills to the key_skills column based on weights. for eg: suppose student 1-{python, Java} job - {python, Ruby, Perl, etc } weight = 1 In the end,...
0
votes
1answer
208 views

how to update column in data frame based on condition

How to update column IsLCap column in dataframe based on Lvalue column whether it is capitilized or not. df.loc[df.Lvalue.istitle(), 'IsLCap'] = 1 # need to be corrected getting an error -...
1
vote
1answer
34 views

Eliminating rows in a dataframe based on specific conditions

I have two columns in a dataframe namely 'origin' and 'destination' which contains the names of different places. I need to remove the rows that don't contain 'PHX', 'JFK', 'NTU' in either of these ...
0
votes
1answer
97 views

How to split data frame into groups, combine rows

I have a large data set with 405 columns and many rows, with data from 15 sites. Each site has 27 columns, each one one quadrats data. Rows are species data. I want to split the data into the 15 ...
0
votes
2answers
1k views

How to create a new column based on two other columns in Pandas?

I am searching for a way to create a new column in my data. I have tried using iterows() but found it extremely time consuming in my dataset containing 40 lakh rows. So here is what I want. Consider I ...
2
votes
2answers
77 views

Pandas: Assign back to table from grouping by column and index

I am trying to implement Exponential Moving Average calculation on a DataFrame. The formula is An additional complication is that my table is grouped and there is ...
1
vote
0answers
16 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
0
votes
1answer
56 views

How can I extract bootstrap generated datasets into individual dataframes?

I am having a bit of trouble understanding Bootstrapping and what/how I can manipulate the bootstrap created dataset. This is all in R My original dataset is structured like this: ...
0
votes
1answer
416 views

Joining two dataframes on the basis of specific conditions [closed]

I've added the images of both the dataframes here. Now I need to combine the two dataframes on the basis of two conditions: Condition 1: The element in the 'arrivalTS' column in the first dataframe(...
0
votes
2answers
330 views

Mutate with custom function in R does not work

I have a data frame, containing a column called: "Frequency". Frequency has values like "Year", "Week", "Month" etc. Now I want to create a new column based on the Frequency column where Year's new ...
1
vote
1answer
96 views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
0
votes
1answer
30 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
0
votes
1answer
1k views

How to find the count of consecutive same string values in a pandas dataframe? [closed]

Assume that we have the following pandas dataframe: ...
1
vote
0answers
17 views

Best way to find dissimilarity in a 6x2 DataFrame?

I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists. The goal is to find the most ...
3
votes
1answer
336 views

How can I combine two single-column datasets into a single Pandas data frame?

I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label ...
1
vote
2answers
50 views

How to group this dataframe in python?

I have this problem: ...
0
votes
1answer
463 views

How can I merge 2+ DataFrame objects without duplicating column names?

This is for work. TLDR: Bottom-line question at the bottom. I am gathering and parsing test results produced by an old test setup whose output formatting is not likely to change anytime soon. I've ...
3
votes
2answers
8k views

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. ...
0
votes
1answer
320 views

Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?

I am running the Kaggle Video Games sales dataset through an XGboost algo. I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5 when I use: data = pd.get_dummies(data=...
1
vote
2answers
184 views

Theoretical Question: Data.table vs Data.frame with Big Data

I know that I can read in a very large csv file much faster with fread using the data.table ...
1
vote
1answer
2k views

shifting the last column in the dataframe to the first place [closed]

While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other ...