Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dataframe]

The tag has no usage guidance.

0
votes
1answer
26 views

How to split data frame into groups, combine rows

I have a large data set with 405 columns and many rows, with data from 15 sites. Each site has 27 columns, each one one quadrats data. Rows are species data. I want to split the data into the 15 ...
0
votes
2answers
39 views

How to create a new column based on two other columns in Pandas?

I am searching for a way to create a new column in my data. I have tried using iterows() but found it extremely time consuming in my dataset containing 40 lakh rows. So here is what I want. Consider I ...
2
votes
2answers
44 views

Pandas: Assign back to table from grouping by column and index

I am trying to implement Exponential Moving Average calculation on a DataFrame. The formula is An additional complication is that my table is grouped and there is ...
1
vote
0answers
6 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
0
votes
1answer
33 views

How can I extract bootstrap generated datasets into individual dataframes?

I am having a bit of trouble understanding Bootstrapping and what/how I can manipulate the bootstrap created dataset. This is all in R My original dataset is structured like this: ...
0
votes
1answer
28 views

Joining two dataframes on the basis of specific conditions [closed]

I've added the images of both the dataframes here. Now I need to combine the two dataframes on the basis of two conditions: Condition 1: The element in the 'arrivalTS' column in the first dataframe(...
0
votes
1answer
40 views

Mutate with custom function in R does not work

I have a data frame, containing a column called: "Frequency". Frequency has values like "Year", "Week", "Month" etc. Now I want to create a new column based on the Frequency column where Year's new ...
1
vote
1answer
18 views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
0
votes
1answer
13 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
0
votes
1answer
133 views

How to find the count of consecutive same string values in a pandas dataframe? [closed]

Assume that we have the following pandas dataframe: ...
1
vote
0answers
12 views

Best way to find dissimilarity in a 6x2 DataFrame?

I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists. The goal is to find the most ...
3
votes
1answer
57 views

How can I combine two single-column datasets into a single Pandas data frame?

I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label ...
0
votes
2answers
34 views

How to group this dataframe in python?

I have this problem: ...
0
votes
1answer
42 views

How can I merge 2+ DataFrame objects without duplicating column names?

This is for work. TLDR: Bottom-line question at the bottom. I am gathering and parsing test results produced by an old test setup whose output formatting is not likely to change anytime soon. I've ...
0
votes
0answers
12 views

what does frame mean? what is its difference in comparison with training epoch in DQN?

what does frame mean? what is its difference in comparison with training epoch in DQN? http://arxiv.org/abs/1810.07286 Also, I have seen in DQN nature paper it has written they trained on 1 ...
1
vote
2answers
1k views

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. ...
0
votes
1answer
103 views

Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?

I am running the Kaggle Video Games sales dataset through an XGboost algo. I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5 when I use: data = pd.get_dummies(data=...
1
vote
2answers
58 views

Theoretical Question: Data.table vs Data.frame with Big Data

I know that I can read in a very large csv file much faster with fread using the data.table ...
1
vote
1answer
223 views

shifting the last column in the dataframe to the first place [closed]

While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other ...
2
votes
1answer
370 views

Adding a new custom column in a python data frame [closed]

I need to add a new column in a python data frame that has the dates of January in a column. Each date is repeated 24 times in the column without any intervention. The total number of entries will ...
0
votes
1answer
324 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
0
votes
2answers
52 views

how to I replace numeric values with a string in an R dataframe?

I want to replace all numeric values in a column in my data frame with a string value. The following doesn't seem to work. ...
1
vote
0answers
52 views

Converting time series data to MNIST like data? [closed]

There are some time series data like this: ...
1
vote
1answer
115 views

Value extraction from a python dataframe [ problem statement specific ]

This is an update on one of the older posts of a friend of mine (That post stopped getting responses as I got a partial solution to my problem) and I'm reworking that problem statement. Link to that ...
1
vote
2answers
306 views

LOOKUP using 2 dataframes in Python

I have 2 dataframes: df1 Id CategoryId 1 A 1 B 2 A 2 E 2 F df2: ...
0
votes
1answer
867 views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
1
vote
1answer
88 views

Coding Problem - Extracting values from a column and forming a new dataframe [edited]

The problem statement requires extracting certain weather parameters for every hour in a particular date as denoted in the dataframe. The column 'hourly' consists of 24 lists in each entry, denoting ...
0
votes
1answer
304 views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
2
votes
1answer
41 views

How to remove rows from a data frame that are identical to other df?

I have two data frames. So how do I remove rows that have identical email addresses in df2 from df1? ...
0
votes
0answers
157 views

Timeseries sensor data for one process regression modeling

The dataset: I have a dataset containing the data of a manufactoring process. The dataset contains the process ID ("Sarzs_no"), the ID of the manufactoring machine ("Unit"), the data logged by two ...
2
votes
1answer
76 views

how to improve searching index in dataframe

Given a pandas dataframe with a timestamp index, sorted. I have a label and I need to find the closest index to that label. Also, I need to find a smaller timestamp, so the search should be computed ...
1
vote
0answers
55 views

Create a new row when a character exists in Python Dataframe

I have an excel sheet which look like this: ColumnA ColumnB ColumnC a 1|2|3 b r 4|5 d b 6|7|8|9|10 z When columnB ...
0
votes
0answers
44 views

Streaming Recommender System using Spark

I have trained a recommender model using ALS from spark.mlib. Now I want to consider the incoming ratings to my model. My data is in form of spark's DataFrame. The idea is to use spark streaming, so I ...
1
vote
1answer
31 views

Is there any way to collect categorical features quickly in Julia DataFrames?

I'm using Julia 0.6.3 with Dataframes.jl I was wondering if there was any way to get categorial features easily in Julia? For large datasets it can be impossible to enter everything by hand. My ...
1
vote
0answers
3k views

Adding multiple columns to spark dataframe

I would like to add several columns to a spark (actually pyspark) dataframe , these columns all being functions of several input columns in the df. There seems to be no 'add_columns' in spark, and ...
1
vote
1answer
2k views

Remove Local Outliers from Dataframe using pandas

Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe....
1
vote
1answer
71 views

Predicting Customer Activity Absence

Could you please assist me with to following question? I have a customer activity dataframe that looks like this: It contains at least 500.000 customers and a "timeseries" of 42 months. The ones and ...
1
vote
1answer
2k views

What’s the best way to save many pandas dataframes together?

I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted,...
9
votes
4answers
39k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
0
votes
1answer
1k views

Implement Sliding Window Algorithm to Detect Spikes

How do I implement sliding window algorithm with a window size of 10 and visualize the data iteratively to see spikes/possible outliers in the dataframe, using python? Please help a beginner.
3
votes
1answer
60 views

Look for previous date in dataframe that has certain column category in R

I have the following data frame: ...
3
votes
1answer
541 views

Handling Concat and Shift Feature in Pandas for Data Science [closed]

I am trying to USE Lag features AND Concat() and Shift() function, ...
0
votes
1answer
385 views

Pandas column count mismatch after insert into dataframe [closed]

I am trying to insert a column of values from one dataframe to another. The total count of values of both the indiviual dataframes is 91744 but when I use ...
1
vote
1answer
1k views

Segmenting pandas dataframe with lists as elements

I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. ...
1
vote
1answer
444 views

Formatting categories of data with pandas in Python

I have around 9500 cities that I want to put in a pandas dataframe, then store in a file for later use. For example, below, these are 3 cities that I have data for. Some range in size based on year. ...
1
vote
2answers
24 views

Help with replacing values in a data frame

I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine ...
0
votes
1answer
247 views

Counting how many rows contain chinese in mixed value dataset using python [closed]

I'm wanting to count how often Chinese is used, I've tried langdetect but the issue is the values are a mix of numbers, text and even URLs which langdetect doesn't seem to support ...
1
vote
1answer
21k views

Create new data frames from existing data frame based on unique column values

I have a large data set (4.5 million rows, 35 columns). The columns of interest are company_id (string) and company_score (float)...
1
vote
1answer
3k views

Replacing words by numbers in multiple columns of a data frame in R

I want to replace the values in a data set (sample in the picture) using numbers instead of words, e.g., 1 instead of D, ...
0
votes
0answers
15 views

What is the best datastructure for more dimensional timeseries prediction?

I'm working on a prediction model for quarter hourly traded electricity contracts. Therefore the usual timeseries based approach won't work. What I want my structure to look like: \begin{array} {|c|...