Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
-1
votes
2answers
31 views

Creating Pandas DataFrames from a dictionary containing various CSV files

I want to create 3 dataframes (i.e., gs, test and train) from the following dictionary: ...
0
votes
1answer
16 views

Color cells based off data in another dataframe [closed]

I have 2 pandas dataframes with the same row and column headers but the data they contain is different. ...
0
votes
1answer
45 views

Renaming Column Elements of DataFrames

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
0
votes
0answers
17 views

Matplotlib: Lines on Top of Bars, Legend on the Left, No Splines [migrated]

I'm having trouble tweaking the graph below. Here's what the dataframe looks like: ...
-1
votes
0answers
22 views

Renaming Column Elements [closed]

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
2
votes
1answer
37 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
1
vote
1answer
30 views

Comparing excel data sets in Pandas

Pretty new to Python, but as an SEO I'm looking at the benefits of using notebooks in my workflow. I've got two excel files which I've cleaned and imported into a new notebook using pandas. I'm ...
2
votes
0answers
11 views

Using an unmerged branch from StatsModels for Anaconda

I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive. See this blog, which as some point mentions, almost ...
0
votes
0answers
25 views

How to fill missing value based on other column in Pandas dataframe? [closed]

For the below sample dataframe, I would like to map the respective state with zone and fill na's with the respective zone names. How can we code for the below instead manually filling missing values? ...
2
votes
1answer
24 views

Trying to return more than just the top result from sklearn NearestNeighbors

I'm trying to compare a list of names (duplicated into a clean file and a messy file). I then compare the files against each other. My problem is that it returns only the top 1 result for each, ...
0
votes
0answers
6 views

How to find out if a given number/text is missing from index of a multi-index pandas dataframe? [migrated]

I have a multi-index pandas dataframe as in below. What I want is to figure out if a particular level1 index is present given a particular level 0 index. To be clear, notice that for level 0 index "...
0
votes
1answer
20 views

Conditionally replace dataframe cells with value from another cell

I have a couple pandas dataframe questions. I would like to replace the values in only certain cells (based on a boolean condition) with a value identified from another cell. I have defined the ...
3
votes
1answer
30 views

How to properly store a list of pandas dataframes, so that I can import them in a new python script?

I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
1
vote
1answer
19 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
2
votes
1answer
34 views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
3
votes
1answer
75 views

sklearn SimpleImputer too slow for categorical data represented as string values

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
1
vote
0answers
34 views

passing a dataframe as a argument while applying a lambda function on a dataframe [closed]

I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
1
vote
0answers
22 views

Training on small, sparse vectors included in one big, denormalized table/dataframe?

So I have a table like this: ...
0
votes
0answers
27 views

Different extraction pipeline for train and test

I'm trying to create a production-ready ML model. The problem is as follows: Training data: Database A + Plus python aggregations. Testing data: Database B + Plus python aggregations. Both ...
3
votes
1answer
43 views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
1
vote
0answers
20 views

Lemmatization of pandas column using Wordnet after POS

I have a pandas column df_travail[line_text] with text. I want to lemmatize each word of this column. First I Lowercase the text : ...
0
votes
0answers
28 views

Pandas DataFrame - compare the first element of each row with other elements of the row

How to correctly compare the first element of each row with other elements of the row in a Pandas DataDrame? Below the code which gives the correct result but I think it is not an optimal solution ...
1
vote
1answer
28 views

Time series with additional information

Given a time series with job-submission counts, how can I predict which certain features about the jobs? I need to predict how many jobs and which jobs arrived in some system. Using ...
0
votes
0answers
9 views

ValueError while Comparing with Multilevel Index

My dataframe - ncg_vol contains Multilevel index - 'Year' and 'Month' and single column 'std'. I used .apply() on std column to perform operation based on the number of days in that year. I used the ...
1
vote
1answer
22 views

Is it possible to change pandas column data type within a sklearn pipeline?

Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables. Basically I need squeeze in a: <...
0
votes
1answer
20 views

CASE WHEN SQL Type Statement in pandas

I'm trying to create a new column called Class which is a calculated column. Basically when the Math class is NA then the class will be the CS class, otherwise it will be vice versa. How can I do ...
2
votes
2answers
49 views

Python Pandas agg error

I am trying to generate descriptive statistics using agg function in Pandas. I am having trouble with one line with a lambda function. They work when I run them as separate lines of code, but when I ...
2
votes
1answer
63 views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
1
vote
1answer
23 views

Pandas dataframe with multiple observations per model

I currently have a pandas dataframe with the following format ...
0
votes
1answer
48 views

why does making the target variable normally distributed helps?

while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that? Should we also ...
0
votes
1answer
39 views

Getting Rid of duplicate data

I extract data from a website, on which I have no control. The data is multiplied. The original report should look like this. ...
0
votes
3answers
121 views

pandas apply function with multiple condition?

if i want to apply lambda with multiple condition how should I do it? df.train.age.apply(lambda x:0 (if x>=0 and x<500)) or is there much better methods?
-1
votes
1answer
355 views

Exception: Data must be 1-dimensional appears when trying to make a submission [closed]

I am doing image classification, and until now I have built my network and evaluated it. The only thing that remains to do is to do the submission, so I have a blind test set which contains images ...
0
votes
0answers
12 views

Group a dataframe or numpy array with cumulative value effectively

Suppose we have an array x = [1, 2, 1, 3, 2, 4, 0, 1, 1, 3] I want to divide it into sub-groups sequentially and in each sub-group the summation is just larger ...
1
vote
1answer
23 views

Help getting corresponding dataframe values

I have two dataframes: ...
0
votes
0answers
22 views

Scatter matrix takes a long time to process, why?

I'm performing two scatter matrixes on two machines launching on one sns.pairplot and on the other scatter_matrix of pandas. I ...
1
vote
1answer
26 views

Is there a python code to perform the following task in python

Is there a way by which I can find how many people rated below 60% or how much percentage of people gave a rating of less than ...
0
votes
1answer
44 views

Problem with sort by and group by in pandas

I have a DataFrame with duplicate id's but have different dates. For Example: ...
0
votes
1answer
34 views

How would I load a database with multiple inconsistent headers?

I have a database with multiple headers like so: ...
0
votes
1answer
28 views

How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]

I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are: A <...
-1
votes
1answer
25 views

Recommend another product only on purchase history of users available

I have data in the following format. ...
0
votes
0answers
29 views

How the Pandas Dataframe and Numpy masking technique works

I'm trying to understand how the masking technique works. I've a dataframe with some values (say one_row_incidents) some of the are not unique. By grouping them .by the duplicate index, I can get the ...
0
votes
2answers
71 views

Create a new column in Pandas Dataframe based on the 'NaN' values in another column [closed]

What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large) ...
0
votes
1answer
99 views

I got the following error : 'DataFrame' object has no attribute 'data' can you help please

I do have the following error: AttributeError: 'DataFrame' object has no attribute 'feature_names' appreciate your input ...
0
votes
1answer
25 views

empty fields to be filled with previous values

In the above sample data, I have empty fields and now the task is to fill the fields with previous values. my columns are dates and the values are a number of items present for that particular article ...
2
votes
2answers
29 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
0
votes
0answers
17 views

What types of variables are these? And what types of tests could I run?

So, I recently made a sheet of data collected from the 2010 Census on Cincinnati neighborhoods. The data is from demographics about the neighborhoods of the city such as median household income and ...
0
votes
1answer
11 views

From a mix of real-world and back-calculated data how to remove the part that was back-calculated?

I have a geostatistical dataset and I've been building linear regression model, but when I plotted the data I've noticed that part of the data shows an absolute straight line trend, i.e. it is most ...
0
votes
2answers
34 views

Is it a best practice to exclude retweets from the data set?

I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is ...
1
vote
0answers
36 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...