Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
742
questions
-1
votes
2answers
31 views
Creating Pandas DataFrames from a dictionary containing various CSV files
I want to create 3 dataframes (i.e., gs, test and train) from the following dictionary:
...
0
votes
1answer
16 views
Color cells based off data in another dataframe [closed]
I have 2 pandas dataframes with the same row and column headers but the data they contain is different.
...
0
votes
1answer
45 views
Renaming Column Elements of DataFrames
I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following:
Data Retrieval: Take partial strings from column elements from DataFrame1
...
0
votes
0answers
17 views
Matplotlib: Lines on Top of Bars, Legend on the Left, No Splines [migrated]
I'm having trouble tweaking the graph below.
Here's what the dataframe looks like:
...
-1
votes
0answers
22 views
Renaming Column Elements [closed]
I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following:
Data Retrieval: Take partial strings from column elements from DataFrame1
...
2
votes
1answer
37 views
How to separate words that are together in a large data set
in twitter data i came across words that are glued together like 'boycottbears'
i want them as 'boycott' 'bears' 'man'
i tried this but this is slow
...
1
vote
1answer
30 views
Comparing excel data sets in Pandas
Pretty new to Python, but as an SEO I'm looking at the benefits of using notebooks in my workflow.
I've got two excel files which I've cleaned and imported into a new notebook using pandas.
I'm ...
2
votes
0answers
11 views
Using an unmerged branch from StatsModels for Anaconda
I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive.
See this blog, which as some point mentions, almost ...
0
votes
0answers
25 views
How to fill missing value based on other column in Pandas dataframe? [closed]
For the below sample dataframe, I would like to map the respective state with zone and fill na's with the respective zone names. How can we code for the below instead manually filling missing values?
...
2
votes
1answer
24 views
Trying to return more than just the top result from sklearn NearestNeighbors
I'm trying to compare a list of names (duplicated into a clean file and a messy file). I then compare the files against each other. My problem is that it returns only the top 1 result for each, ...
0
votes
0answers
6 views
How to find out if a given number/text is missing from index of a multi-index pandas dataframe? [migrated]
I have a multi-index pandas dataframe as in below. What I want is to figure out if a particular level1 index is present given a particular level 0 index. To be clear, notice that for level 0 index "...
0
votes
1answer
20 views
Conditionally replace dataframe cells with value from another cell
I have a couple pandas dataframe questions. I would like to replace the values in only certain cells (based on a boolean condition) with a value identified from another cell. I have defined the ...
3
votes
1answer
30 views
How to properly store a list of pandas dataframes, so that I can import them in a new python script?
I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
1
vote
1answer
19 views
find bigrams in pandas
I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
2
votes
1answer
34 views
how to sort a multi-level pandas data-frame by a particular column?
I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean:
...
3
votes
1answer
75 views
sklearn SimpleImputer too slow for categorical data represented as string values
I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
1
vote
0answers
34 views
passing a dataframe as a argument while applying a lambda function on a dataframe [closed]
I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
1
vote
0answers
22 views
Training on small, sparse vectors included in one big, denormalized table/dataframe?
So I have a table like this:
...
0
votes
0answers
27 views
Different extraction pipeline for train and test
I'm trying to create a production-ready ML model.
The problem is as follows:
Training data: Database A + Plus python aggregations.
Testing data: Database B + Plus python aggregations.
Both ...
3
votes
1answer
43 views
Is there a real C4.5 implementation in Python ? (handling missing value)
To my understanding, C4.5 comes with 4 improvements compared to ID3:
Handling missing values in both training data and "test" data,
Handling continuous data
Handling costs on attributes.
The pruning
...
1
vote
0answers
20 views
Lemmatization of pandas column using Wordnet after POS
I have a pandas column df_travail[line_text] with text.
I want to lemmatize each word of this column.
First I Lowercase the text :
...
0
votes
0answers
28 views
Pandas DataFrame - compare the first element of each row with other elements of the row
How to correctly compare the first element of each row with other elements of the row in a Pandas DataDrame?
Below the code which gives the correct result but I think it is not an optimal solution ...
1
vote
1answer
28 views
Time series with additional information
Given a time series with job-submission counts, how can I predict which certain features about the jobs?
I need to predict how many jobs and which jobs arrived in some system. Using ...
0
votes
0answers
9 views
ValueError while Comparing with Multilevel Index
My dataframe - ncg_vol contains Multilevel index - 'Year' and 'Month' and single column 'std'. I used .apply() on std column to perform operation based on the number of days in that year. I used the ...
1
vote
1answer
22 views
Is it possible to change pandas column data type within a sklearn pipeline?
Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables.
Basically I need squeeze in a:
<...
0
votes
1answer
20 views
CASE WHEN SQL Type Statement in pandas
I'm trying to create a new column called Class which is a calculated column.
Basically when the Math class is NA then the class will be the CS class, otherwise it will be vice versa.
How can I do ...
2
votes
2answers
49 views
Python Pandas agg error
I am trying to generate descriptive statistics using agg function in Pandas. I am having trouble with one line with a lambda function. They work when I run them as separate lines of code, but when I ...
2
votes
1answer
63 views
How to create a new dataframe using the another dataframe
I have created and worked on a DataFrame for a project. It looks like the following:
Critics Items Ratings
a...........1..........5
b...........2..........3
b...........3..........2
c...........
1
vote
1answer
23 views
Pandas dataframe with multiple observations per model
I currently have a pandas dataframe with the following format
...
0
votes
1answer
48 views
why does making the target variable normally distributed helps?
while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that?
Should we also ...
0
votes
1answer
39 views
Getting Rid of duplicate data
I extract data from a website, on which I have no control. The data is multiplied.
The original report should look like this.
...
0
votes
3answers
121 views
pandas apply function with multiple condition?
if i want to apply lambda with multiple condition how should I do it?
df.train.age.apply(lambda x:0 (if x>=0 and x<500))
or is there much better methods?
-1
votes
1answer
355 views
Exception: Data must be 1-dimensional appears when trying to make a submission [closed]
I am doing image classification, and until now I have built my network and evaluated it. The only thing that remains to do is to do the submission, so I have a blind test set which contains images ...
0
votes
0answers
12 views
Group a dataframe or numpy array with cumulative value effectively
Suppose we have an array
x = [1, 2, 1, 3, 2, 4, 0, 1, 1, 3]
I want to divide it into sub-groups sequentially and in each sub-group the summation is just larger ...
1
vote
1answer
23 views
0
votes
0answers
22 views
Scatter matrix takes a long time to process, why?
I'm performing two scatter matrixes on two machines launching on one sns.pairplot and on the other scatter_matrix of pandas. I ...
1
vote
1answer
26 views
Is there a python code to perform the following task in python
Is there a way by which I can find how many people rated below 60% or how much percentage of people gave a rating of less than ...
0
votes
1answer
44 views
Problem with sort by and group by in pandas
I have a DataFrame with duplicate id's but have different dates.
For Example:
...
0
votes
1answer
34 views
How would I load a database with multiple inconsistent headers?
I have a database with multiple headers like so:
...
0
votes
1answer
28 views
How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]
I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are:
A <...
-1
votes
1answer
25 views
Recommend another product only on purchase history of users available
I have data in the following format.
...
0
votes
0answers
29 views
How the Pandas Dataframe and Numpy masking technique works
I'm trying to understand how the masking technique works.
I've a dataframe with some values (say one_row_incidents) some of the are not unique. By grouping them .by the duplicate index, I can get the ...
0
votes
2answers
71 views
Create a new column in Pandas Dataframe based on the 'NaN' values in another column [closed]
What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large)
...
0
votes
1answer
99 views
I got the following error : 'DataFrame' object has no attribute 'data' can you help please
I do have the following error: AttributeError: 'DataFrame' object has no attribute 'feature_names'
appreciate your input
...
0
votes
1answer
25 views
empty fields to be filled with previous values
In the above sample data, I have empty fields and now the task is to fill the fields with previous values.
my columns are dates and the values are a number of items present for that particular article ...
2
votes
2answers
29 views
Optimization of pandas row iteration and summation
i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result.
What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
0
votes
0answers
17 views
What types of variables are these? And what types of tests could I run?
So, I recently made a sheet of data collected from the 2010 Census on Cincinnati neighborhoods. The data is from demographics about the neighborhoods of the city such as median household income and ...
0
votes
1answer
11 views
From a mix of real-world and back-calculated data how to remove the part that was back-calculated?
I have a geostatistical dataset and I've been building linear regression model, but when I plotted the data I've noticed that part of the data shows an absolute straight line trend, i.e. it is most ...
0
votes
2answers
34 views
Is it a best practice to exclude retweets from the data set?
I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is ...
1
vote
0answers
36 views
Reshape Time series data for Conv2d Block
I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial.
...
