Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
1answer
38 views

What is the best way to store images in python for machine learning

I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values. So far ...
0
votes
1answer
20 views

How to sum all price values having in count the quantity of each one

Using pandas, I would like to sum all of the price to get the count value for each one. The Dataframe looks like this: ...
1
vote
2answers
222 views

is it ok to get 100% accuracy in random forest classifier algorithm?

while i was building the model to predict the performance of machine using the features like OEF,working time,performance/head etc... I splitted the training data using ...
1
vote
1answer
145 views

Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
0
votes
0answers
15 views

Same Output for every row when doing prediction. Thus I am having wrong predictions

I have implemented an emotion analysis using lstm method, I have trained my model in the first part of the codes and i am now am doing the prediction part. When I run my system, I am having same ...
0
votes
1answer
317 views

Plot dataframe with two columns on the x axis

how would I plot the data below with the x axis as Year & Month? Each Year-month combination has a unique monthly count (Y). I am unsure how to proceed, given than they are in different columns, ...
0
votes
0answers
12 views

count movements of values from 1 quarter to the other using pandas

i want to count all 10's that moved to 20 from 1 Quarter to the other and total of all 10's in each Quarter Below is the result i want ...
0
votes
1answer
24 views

Assigning new word by creating a new column when finding match

I have a survey df and I want to assign a new value of either existing customer or new customer based on their answers. if someone has 3 answers but one of them matches "coca cola" I want to ...
1
vote
1answer
19 views

How to summarize multiple time series like dataset

I am running an experiment where I am sampling CPU utilization every second. A snippet of the data from each experiment run is something like this: ...
0
votes
2answers
36 views

How to extract a subset from a data frame? [closed]

Let's say I have a dataframe that contains several columns including a column called A (containing integers). I would only like to consider the rows of table that correspond to $A \ge 3$. What would ...
1
vote
2answers
2k views

Export pandas dataframe to a nested dictionary from multiple columns

Its a similar question to Export pandas to dictionary by combining multiple row values But in this case I want something different. ...
0
votes
1answer
27 views

How to train a classification algorithm with normalized data set using scikit-learn python

I have a dataset in CSV format, 6 columns and 1877 rows. The full dataset can be viewed at ShareCSV. The first five columns are characteristics and the final column is a binary result, I want to ...
0
votes
1answer
277 views

How do I read a dat file, for which I don't know its structure?

Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it. How do I figure out the contents in my dat file. ...
2
votes
2answers
184 views

What do I do when my column has 50% data missing?

I have a data set i am pre-processing. However in my categorical columns (3 of them) i have "??" in it's place. They constitute 50% of the data. In fact 3 columns have this. My question is how should ...
0
votes
0answers
17 views

Are there decisive leaders in programming with tabular data?

What are the most effective bread-and-butter in-memory open source tabular data frameworks today? I have been working with tabular data for years with an in-house solution that integrates with Excel ...
2
votes
1answer
548 views

matplotlib graph to plot values and variance

I am really new to the world of matplotlib graphing as well as using those graphs to understand data. I have written a simple python code where I read a .csv file ...
3
votes
0answers
469 views

How to apply a groupby rolling function to create multiple columns in the dataframe

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo and the link to the data is here and the example here. Some Sample of data ...
3
votes
1answer
47 views

Pandas dataframe, create columns depending on the row value

I get a csv that if you read it, looks like: ...
1
vote
0answers
58 views

Pandas - dataframe not storing images properly

I'm trying to organize image files I have into a Pandas dataframe. I'm using Keras's preprocessing library for working with them. Here's how I load them in: ...
0
votes
0answers
40 views

ValueError while Plotting a specific column using K-Means

Following a tutorial, I was able to plot an array of data using KMeans. This is a csv file ...
-1
votes
2answers
57 views

Creating Pandas DataFrames from a dictionary containing various CSV files

I want to create 3 dataframes (i.e., gs, test and train) from the following dictionary: ...
-1
votes
1answer
74 views

Renaming Column Elements of DataFrames

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
2
votes
1answer
45 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
1
vote
1answer
43 views

Comparing excel data sets in Pandas

Pretty new to Python, but as an SEO I'm looking at the benefits of using notebooks in my workflow. I've got two excel files which I've cleaned and imported into a new notebook using pandas. I'm ...
2
votes
0answers
23 views

Using an unmerged branch from StatsModels for Anaconda

I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive. See this blog, which as some point mentions, almost ...
2
votes
1answer
37 views

Trying to return more than just the top result from sklearn NearestNeighbors

I'm trying to compare a list of names (duplicated into a clean file and a messy file). I then compare the files against each other. My problem is that it returns only the top 1 result for each, ...
0
votes
1answer
695 views

Conditionally replace dataframe cells with value from another cell

I have a couple pandas dataframe questions. I would like to replace the values in only certain cells (based on a boolean condition) with a value identified from another cell. I have defined the ...
3
votes
1answer
517 views

How to properly store a list of pandas dataframes, so that I can import them in a new python script?

I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
2
votes
1answer
241 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
1
vote
1answer
671 views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
3
votes
1answer
619 views

sklearn SimpleImputer too slow for categorical data represented as string values

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
1
vote
0answers
107 views

passing a dataframe as a argument while applying a lambda function on a dataframe [closed]

I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
1
vote
0answers
22 views

Training on small, sparse vectors included in one big, denormalized table/dataframe?

So I have a table like this: ...
0
votes
0answers
29 views

Different extraction pipeline for train and test

I'm trying to create a production-ready ML model. The problem is as follows: Training data: Database A + Plus python aggregations. Testing data: Database B + Plus python aggregations. Both ...
3
votes
1answer
98 views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
1
vote
0answers
362 views

Lemmatization of pandas column using Wordnet after POS

I have a pandas column df_travail[line_text] with text. I want to lemmatize each word of this column. First I Lowercase the text : ...
0
votes
0answers
65 views

Pandas DataFrame - compare the first element of each row with other elements of the row

How to correctly compare the first element of each row with other elements of the row in a Pandas DataDrame? Below the code which gives the correct result but I think it is not an optimal solution ...
1
vote
1answer
32 views

Time series with additional information

Given a time series with job-submission counts, how can I predict which certain features about the jobs? I need to predict how many jobs and which jobs arrived in some system. Using ...
0
votes
0answers
168 views

ValueError while Comparing with Multilevel Index

My dataframe - ncg_vol contains Multilevel index - 'Year' and 'Month' and single column 'std'. I used .apply() on std column to perform operation based on the number of days in that year. I used the ...
1
vote
1answer
55 views

Is it possible to change pandas column data type within a sklearn pipeline?

Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables. Basically I need squeeze in a: <...
0
votes
1answer
52 views

CASE WHEN SQL Type Statement in pandas

I'm trying to create a new column called Class which is a calculated column. Basically when the Math class is NA then the class will be the CS class, otherwise it will be vice versa. How can I do ...
1
vote
1answer
231 views

Python Pandas agg error

I am trying to generate descriptive statistics using agg function in Pandas. I am having trouble with one line with a lambda function. They work when I run them as separate lines of code, but when I ...
1
vote
1answer
3k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
1
vote
1answer
36 views

Pandas dataframe with multiple observations per model

I currently have a pandas dataframe with the following format ...
1
vote
2answers
182 views

why does making the target variable normally distributed helps?

while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that? Should we also ...
0
votes
2answers
53 views

Getting Rid of duplicate data

I extract data from a website, on which I have no control. The data is multiplied. The original report should look like this. ...
0
votes
3answers
1k views

pandas apply function with multiple condition?

if i want to apply lambda with multiple condition how should I do it? df.train.age.apply(lambda x:0 (if x>=0 and x<500)) or is there much better methods?
0
votes
1answer
9k views

Exception: Data must be 1-dimensional appears when trying to make a submission [closed]

I am doing image classification, and until now I have built my network and evaluated it. The only thing that remains to do is to do the submission, so I have a blind test set which contains images ...
0
votes
0answers
14 views

Group a dataframe or numpy array with cumulative value effectively

Suppose we have an array x = [1, 2, 1, 3, 2, 4, 0, 1, 1, 3] I want to divide it into sub-groups sequentially and in each sub-group the summation is just larger ...
1
vote
1answer
23 views

Help getting corresponding dataframe values

I have two dataframes: ...

1 2 3
4
5
18