Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
3answers
19 views

pandas apply function with multiple condition?

if i want to apply lambda with multiple condition how should I do it? df.train.age.apply(lambda x:0 (if x>=0 and x<500)) or is there much better methods?
2
votes
3answers
426 views

Handling a combined dataset of numerical and categorical features for Regression

I have a dataset that has a large number of categorical features and a few numerical features, and I want to predict the probability that any given input is one of two types of a certain binary output ...
0
votes
1answer
695 views

how to extract substrings from a dataframe column

I have a dataset as below: col1 {:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,jjjhjhd :70 I want to add 2 more cols such as col_50 &...
-1
votes
1answer
13 views

Exception: Data must be 1-dimensional appears when trying to make a submission [closed]

I am doing image classification, and until now I have built my network and evaluated it. The only thing that remains to do is to do the submission, so I have a blind test set which contains images ...
0
votes
1answer
2k views

how to export the tables into a csv file pandas

The following is a piece of code I wrote to create a pivot table for categorical vs continuous variable. ...
0
votes
2answers
31 views

Is it a best practice to exclude retweets from the data set?

I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is ...
0
votes
0answers
9 views

Group a dataframe or numpy array with cumulative value effectively

Suppose we have an array x = [1, 2, 1, 3, 2, 4, 0, 1, 1, 3] I want to divide it into sub-groups sequentially and in each sub-group the summation is just larger ...
0
votes
1answer
121 views

Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data

I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with ...
3
votes
2answers
11k views

Merging information of rows with the same date [closed]

My data looks like this: ...
1
vote
1answer
22 views
0
votes
1answer
179 views

Calling function on a list object

I have a list (sub_indices) containing dataframe index values. Using these index values, I need to make changes in another dataframe (main_df). Changes must be done on these index values that are in ...
0
votes
1answer
18 views

My small script on value alteration in columns of a data not working

I have a data set which has "Speed" as one of the columns (features). The column contains both zero and non-zero values. I want to randomly set 10% of the non-zero values to zeros. This will change ...
0
votes
0answers
17 views

Scatter matrix takes a long time to process, why?

I'm performing two scatter matrixes on two machines launching on one sns.pairplot and on the other scatter_matrix of pandas. I ...
0
votes
1answer
29 views

How would I load a database with multiple inconsistent headers?

I have a database with multiple headers like so: ...
2
votes
1answer
137 views

memory problem due to large file

I'm new in Python and definitely, I'm sure I do some mistake. Here is my problem and thanks all to your help in advance. I have 2 files (one is Hive) and the other is csv and merging them. I have ...
1
vote
1answer
16 views

Is there a python code to perform the following task in python

Is there a way by which I can find how many people rated below 60% or how much percentage of people gave a rating of less than ...
0
votes
1answer
133 views

Upload 'tsv' file to Google Colab / Jupyter Notebook

TSV(Tab separated Value) extension file can't be uploaded to google colab using pandas Used this to upload my file ...
1
vote
1answer
54 views

ValueError from statsmodels ExponentialSmoothing

I've been having a frustrating issue with the ExponentialSmoothing module from statsmodels. My data is a pandas series with 74 weekly data points that looks like this: ...
1
vote
1answer
42 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
1
vote
2answers
106 views

plot multiple strings from a dataframe

I am trying to read and plot several files which looks like as below when I open with python using pandas read.table. The columns to plot are "dev" vs timestamp. The timestamp should be compose from ...
3
votes
6answers
65k views

I got the following error : 'DataFrame' object has no attribute 'data'

I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result: Program: ...
0
votes
1answer
39 views

Problem with sort by and group by in pandas

I have a DataFrame with duplicate id's but have different dates. For Example: ...
0
votes
1answer
42 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
0
votes
1answer
1k views

Comparing dataframe object with string value in django

I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL ...
1
vote
4answers
43k views

Using pandas, check a column for matching text and update new column if TRUE

My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE. From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, ...
0
votes
1answer
26 views

How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]

I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are: A <...
0
votes
2answers
27 views

How do i find the most frequently occuring category for each unique customer?

I have a dataset with 30k unique users with each customer having multiple transaction. I aggregated the dataset with one record per user and containing aggregated information such as average ...
1
vote
1answer
89 views

pd.qcut bins error!

Hi Data Science community! I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error. ...
0
votes
1answer
18 views

Getting extra characters while loading data

While loading data using read_sas from sas dataset in Python getting b' along with every string. How can I fix this?
1
vote
2answers
3k views

How to save Tensorflow predictions to data frame?

I am new to Tensorflow. I have trained Tensorflow model, but I need to take model predictions and add them to my original test set as a column. How can I do that? ...
1
vote
1answer
321 views

How to calculate customer purchase interval and predict next purchase in python?

Suppose we have a data set consists of columns TransactionId, CardNo, TransactionDate then how can we calculate the customer purchase interval (means if customer A purchased on Jan 1st and after ...
-1
votes
1answer
24 views

Recommend another product only on purchase history of users available

I have data in the following format. ...
43
votes
9answers
150k views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
0
votes
0answers
28 views

How the Pandas Dataframe and Numpy masking technique works

I'm trying to understand how the masking technique works. I've a dataframe with some values (say one_row_incidents) some of the are not unique. By grouping them .by the duplicate index, I can get the ...
2
votes
2answers
243 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
0
votes
2answers
46 views

Create a new column in Pandas Dataframe based on the 'NaN' values in another column [closed]

What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large) ...
0
votes
1answer
444 views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...
1
vote
1answer
48 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
2
votes
1answer
67 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
2
votes
2answers
103 views

Error plotting with datetime and value using matplotlib in python

Here I have a dataset with three values. I plot these three values in one graph using python. While I plot the graph first plot line is not coming properly. Because start point and end point combined ...
0
votes
1answer
63 views

I got the following error : 'DataFrame' object has no attribute 'data' can you help please

I do have the following error: AttributeError: 'DataFrame' object has no attribute 'feature_names' appreciate your input ...
0
votes
0answers
48 views

ANOVA Cohen's d matrix using python

I'm running a one-way ANOVA within 7 groups and would like to create a matrix containing Cohen's d for each couple of groups. To generalize the problem, it's creating a matrix based on one vector and ...
0
votes
1answer
25 views

empty fields to be filled with previous values

In the above sample data, I have empty fields and now the task is to fill the fields with previous values. my columns are dates and the values are a number of items present for that particular article ...
1
vote
2answers
24 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
3
votes
2answers
36k views

Could not convert string to float error on KDDCup99 dataset

I am trying to perform a comparison between 5 algorithms against the KDD Cup 99 dataset and the NSL-KDD datasets using Python and I am having an issue when trying to build and evaluate the models ...
0
votes
0answers
13 views

Excel date-time auto fill error [migrated]

I have data of 10 metrics for a 24 hour period which is in 5 minute intervals. I merged all the 10 data metrics into one file and converted it into a .xlsx file. After opening the excel file I ...
0
votes
1answer
477 views

How to measure correlation between several categorical features and a numerical label in Python?

I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product ...