Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dataframe]

The tag has no usage guidance.

1
vote
0answers
13 views

Aggregate values of same name pandas dataframe columns to single column [migrated]

I have multiple csv files that were produced by tokenizing code. These files contain keywords in uppercase and lowercase. I would like to merge all those files in one single dataframe which contains ...
0
votes
1answer
15 views

How to set limits of Y-axes in countplot?

df in my program happens to be a dataframe with these columns : ...
2
votes
1answer
35 views

Pandas merge column duplicate and sum value

How to merge duplicate column and sum their value? What I have A 30 A 40 B 50 What I need A 70 B 50 DF for this ...
0
votes
2answers
29 views
0
votes
0answers
30 views

Applying function on non-null values of two columns in a data frame

I am trying to apply a function on a data frame, the function calculates fuzzy matching and Jellyfish features on two columns. I want this function to do the calculation only on non-null values, even ...
3
votes
1answer
118 views

Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

I want to store JSON Data into MySQL Database using Python. I used dataframe of pandas. I found that ...
1
vote
2answers
29 views

Is there a way to drop duplicated rows based on an unhashable column?

i have a pandas dataframe df with one column z filled with set values i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values (...
3
votes
3answers
161 views

How to Write Multiple Data Frames in an Excel Sheet

I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. ...
0
votes
1answer
23 views

dataframe.columns.difference() use

I am trying to find the working of dataframe.columns.difference() but couldn't find a satisfactory explanation about it. Can anyone explain the working of this ...
2
votes
2answers
36 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
0
votes
2answers
121 views

'DataFrame' object has no attribute 'to_dataframe'

I'm sure I have a small error here that I'm overlooking, but am having a tough time figuring out what I need to change. Here is my code up until the error I'm getting. ...
2
votes
1answer
83 views

Inputting (a lot of )data into a dataframe one row at a time

I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this: ...
2
votes
1answer
38 views

Breaking a DataFrame based on Specific Column Values [closed]

Assume I have a following dataframe which has years of experience and salary. I would like to create a new dataframe from the existing one, only including observations where salary is above 45000. I ...
1
vote
2answers
55 views

Numpy array from pandas dataframe

I am new in using python for data science. What is the difference between selecting a a column with: df['name'].values and ...
0
votes
1answer
49 views

Divide a column by itself with mutate_at dplyr

Hi I'd like to turn each non zero value of my selected columns to a 1 using mutate_at() ...
-1
votes
1answer
18 views

LabelEncoding a Dataframe

I have a dataframe with integer and categorical variables. Should I label encode all variables (both integer and categorical) or should I encode only the categorical features?
0
votes
0answers
17 views

String Matching columns based on weight

I have the following 2 DataFrames and want to match skills to the key_skills column based on weights. for eg: suppose student 1-{python, Java} job - {python, Ruby, Perl, etc } weight = 1 In the end,...
0
votes
1answer
37 views

how to update column in data frame based on condition

How to update column IsLCap column in dataframe based on Lvalue column whether it is capitilized or not. df.loc[df.Lvalue.istitle(), 'IsLCap'] = 1 # need to be corrected getting an error -...
1
vote
1answer
30 views

Eliminating rows in a dataframe based on specific conditions

I have two columns in a dataframe namely 'origin' and 'destination' which contains the names of different places. I need to remove the rows that don't contain 'PHX', 'JFK', 'NTU' in either of these ...
0
votes
1answer
40 views

How to split data frame into groups, combine rows

I have a large data set with 405 columns and many rows, with data from 15 sites. Each site has 27 columns, each one one quadrats data. Rows are species data. I want to split the data into the 15 ...
0
votes
2answers
139 views

How to create a new column based on two other columns in Pandas?

I am searching for a way to create a new column in my data. I have tried using iterows() but found it extremely time consuming in my dataset containing 40 lakh rows. So here is what I want. Consider I ...
2
votes
2answers
53 views

Pandas: Assign back to table from grouping by column and index

I am trying to implement Exponential Moving Average calculation on a DataFrame. The formula is An additional complication is that my table is grouped and there is ...
1
vote
0answers
11 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
0
votes
1answer
43 views

How can I extract bootstrap generated datasets into individual dataframes?

I am having a bit of trouble understanding Bootstrapping and what/how I can manipulate the bootstrap created dataset. This is all in R My original dataset is structured like this: ...
0
votes
1answer
85 views

Joining two dataframes on the basis of specific conditions [closed]

I've added the images of both the dataframes here. Now I need to combine the two dataframes on the basis of two conditions: Condition 1: The element in the 'arrivalTS' column in the first dataframe(...
0
votes
2answers
167 views

Mutate with custom function in R does not work

I have a data frame, containing a column called: "Frequency". Frequency has values like "Year", "Week", "Month" etc. Now I want to create a new column based on the Frequency column where Year's new ...
1
vote
1answer
32 views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
0
votes
1answer
20 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
0
votes
1answer
583 views

How to find the count of consecutive same string values in a pandas dataframe? [closed]

Assume that we have the following pandas dataframe: ...
1
vote
0answers
14 views

Best way to find dissimilarity in a 6x2 DataFrame?

I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists. The goal is to find the most ...
3
votes
1answer
99 views

How can I combine two single-column datasets into a single Pandas data frame?

I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label ...
0
votes
2answers
40 views

How to group this dataframe in python?

I have this problem: ...
0
votes
1answer
169 views

How can I merge 2+ DataFrame objects without duplicating column names?

This is for work. TLDR: Bottom-line question at the bottom. I am gathering and parsing test results produced by an old test setup whose output formatting is not likely to change anytime soon. I've ...
1
vote
2answers
4k views

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. ...
0
votes
1answer
221 views

Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?

I am running the Kaggle Video Games sales dataset through an XGboost algo. I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5 when I use: data = pd.get_dummies(data=...
1
vote
2answers
97 views

Theoretical Question: Data.table vs Data.frame with Big Data

I know that I can read in a very large csv file much faster with fread using the data.table ...
1
vote
1answer
860 views

shifting the last column in the dataframe to the first place [closed]

While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other ...
2
votes
1answer
897 views

Adding a new custom column in a python data frame [closed]

I need to add a new column in a python data frame that has the dates of January in a column. Each date is repeated 24 times in the column without any intervention. The total number of entries will ...
0
votes
1answer
654 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
0
votes
2answers
107 views

how to I replace numeric values with a string in an R dataframe?

I want to replace all numeric values in a column in my data frame with a string value. The following doesn't seem to work. ...
1
vote
0answers
74 views

Converting time series data to MNIST like data? [closed]

There are some time series data like this: ...
1
vote
1answer
236 views

Value extraction from a python dataframe [ problem statement specific ]

This is an update on one of the older posts of a friend of mine (That post stopped getting responses as I got a partial solution to my problem) and I'm reworking that problem statement. Link to that ...
1
vote
2answers
455 views

LOOKUP using 2 dataframes in Python

I have 2 dataframes: df1 Id CategoryId 1 A 1 B 2 A 2 E 2 F df2: ...
0
votes
1answer
1k views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
1
vote
1answer
104 views

Coding Problem - Extracting values from a column and forming a new dataframe [edited]

The problem statement requires extracting certain weather parameters for every hour in a particular date as denoted in the dataframe. The column 'hourly' consists of 24 lists in each entry, denoting ...
0
votes
1answer
746 views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
2
votes
1answer
183 views

How to remove rows from a data frame that are identical to other df?

I have two data frames. So how do I remove rows that have identical email addresses in df2 from df1? ...
2
votes
1answer
114 views

how to improve searching index in dataframe

Given a pandas dataframe with a timestamp index, sorted. I have a label and I need to find the closest index to that label. Also, I need to find a smaller timestamp, so the search should be computed ...
1
vote
0answers
58 views

Create a new row when a character exists in Python Dataframe

I have an excel sheet which look like this: ColumnA ColumnB ColumnC a 1|2|3 b r 4|5 d b 6|7|8|9|10 z When columnB ...