Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
2answers
600 views

Create a new column in Pandas Dataframe based on the 'NaN' values in another column [closed]

What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large) ...
0
votes
1answer
321 views

I got the following error : 'DataFrame' object has no attribute 'data' can you help please

I do have the following error: AttributeError: 'DataFrame' object has no attribute 'feature_names' appreciate your input ...
0
votes
1answer
30 views

empty fields to be filled with previous values

In the above sample data, I have empty fields and now the task is to fill the fields with previous values. my columns are dates and the values are a number of items present for that particular article ...
2
votes
2answers
37 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
0
votes
0answers
17 views

What types of variables are these? And what types of tests could I run?

So, I recently made a sheet of data collected from the 2010 Census on Cincinnati neighborhoods. The data is from demographics about the neighborhoods of the city such as median household income and ...
0
votes
1answer
12 views

From a mix of real-world and back-calculated data how to remove the part that was back-calculated?

I have a geostatistical dataset and I've been building linear regression model, but when I plotted the data I've noticed that part of the data shows an absolute straight line trend, i.e. it is most ...
0
votes
2answers
49 views

Is it a best practice to exclude retweets from the data set?

I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is ...
1
vote
0answers
62 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...
1
vote
2answers
2k views

Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode

Hopefully, you don't mind me posting a question here instead of the regular stack exchange forum. I'm learning ML basics and practising pandas. assuming the data frame is called df, column name = ...
2
votes
2answers
393 views

Pandas datetime error when reading from excel file

I am trying to read an excel file that has two columns using pandas. This is how the data looks in excel file: ...
0
votes
1answer
16 views

fast processing: changing the column value in a geospatial environment

I have a table whose header looks like this: complaint_type, borough, street_name, incident_zip, latitude, longitude 1) I want to check if the "incident_zip" column of each row is in a specific list ...
1
vote
3answers
301 views

creating a csv file from two csv files

I have two csv files: sub_compiler.to_csv('sub_compiler.csv') sub_compiler.head() and ...
0
votes
1answer
30 views

Merging two pandas dataframes [duplicate]

I have two csv files,and if I do: df_compiler = pd.read_csv('sub_compiler.csv') I obtain: and for the other csv file I do: ...
1
vote
1answer
56 views

Is there any way to create column based on some previous column in PANDAS dataframe?

Given: I have Pandas Dataframe as shown below ...
0
votes
1answer
25 views

error when submitting machine learning project

I am trying to make a submission of a machine learning classification problem. I have a test dataset where to try my model. To submit I have to build a csv file. The prblem is that when I go building ...
0
votes
1answer
7k views

'list' object has no attribute 'values' when we are using append in python

Here I have a dataset with three inputs. Here I generated y value using append. After the append I got the output like this: y.append(rec.iloc[0]['y']) Then I ...
2
votes
1answer
34 views

Dataset from sequence of messages

I have a sorted dataset by datestamp which looks like this: ...
1
vote
1answer
31 views

Changing the value of a column based on the condition on another column

I have a table whose header looks like this: complaint_type borough street_name incident_zip latitude longitude I want to check if the "incident_zip" column of each row is in a specific ...
1
vote
1answer
46 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
0
votes
0answers
214 views

Viewing TensorFlow datasets vs. pandas DataFrames: Differences?

A simple preview of a pandas.DataFrame is typically done by calling head(k) and will simply return the first ...
0
votes
1answer
635 views

Replace entire columns in pandas dataframe

I would like to replace entire columns of a pandas dataframe with other columns, for example: and I would like to replace the columns A and B. What I did is the following: ...
2
votes
1answer
182 views

Get elements from lists in pandas dataframe [closed]

I have the following column of a data frame: I get it by doing dataFrame['opcodes]. and I would like to consider only the first 20 and the last 20 elements of ...
0
votes
1answer
21 views

Getting extra characters while loading data

While loading data using read_sas from sas dataset in Python getting b' along with every string. How can I fix this?
0
votes
2answers
414 views

use .apply() function to change values to a column of the dataframe

I have a dataframe which is the following: and I would like to consider only the column of instructions and keep just the values push, test, mov, test ,....., so just the first word of each string ...
2
votes
1answer
996 views

pd.qcut bins error!

Hi Data Science community! I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error. ...
2
votes
1answer
402 views

Why is pandas corr() deleting columns?

I'm doing a basic correlation analysis but for some reason pandas corr() is deleting columns, not sure why. ...
1
vote
1answer
40 views

Scikit model is not able to predict sequence correctly

I am trying to create a regression model using scikit-learn for predicting car price. The input data are, car model(trim), kilometers used, past resale price of similar car and age of used car. I am ...
0
votes
1answer
27 views

Problem building a feature vector

I am trying build a classifier for malware analysis for which basing in the instructions of an assembly code, such as push, mov,... I want to predict the compiler, and in a second time the ...
-1
votes
2answers
2k views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
1
vote
1answer
926 views

MemoryError when concatenating a large data-frame

I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns. My issue is that these CSV files are quite large and when ...
0
votes
0answers
521 views

Connecting to IEX with Pandas Datareader

My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
1
vote
0answers
35 views

Method for 'Simple' Clustering?

In a recent adult education class I took the prof shared a spreadsheet that was written in VBA code and takes ages to execute, I'm sure ...
0
votes
0answers
29 views

How to use groupby, resample, fillna fast with a large dataset

I have a time series of data in the following form: ...
1
vote
0answers
36 views

Pipeline that cached the results

I use pandas to do feature extraction for machine learning. I hope to achieve the following: Consider I have five data processing steps done sequentially, and I execute it once, the results will be ...
-1
votes
1answer
1k views
2
votes
0answers
37 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
0
votes
1answer
114 views

How to print nullity correlation matrix

I've a trainingset which has 400 features and most of them have null value. I tried to draw the heatmap of nullity correlation matrix by means of Python and ...
2
votes
1answer
202 views

Numpy arithmetic operation between two columns

Below is the numpy array. I need to perform two operations on this array. Add one column with value [column 1] - [column 3]. Add another column with value [column 1] - [previous value of column1]. I ...
1
vote
0answers
151 views

Pandas nested/recursive groupby count [closed]

I have a DataFrame with more than 2 million rows, that looks like this: ...
0
votes
1answer
62 views

Find the most relevant columns for each single class in pandas

The following question (this one) did not help me. I have a big dataset, and I want to know which Columns are the most relevant for the Target Variable. I know that, in my case, for each class in the ...
0
votes
3answers
135 views

Gradient decent in Python

I just finished working on my first machine learning algorithm i.e Linear regression. I want to reduce the rmse by optimising the model. I found out that gradient decent does the same job. But i dont ...
0
votes
1answer
259 views

Sublime Text 2 with Pandas for Excel (Combining Data) & Data Visualization

I'm new to programming with Python, and so far it's been a headache to create a build environment- and need your support and expertise in this area. Background I'm running a MacBook, and using ...
0
votes
1answer
102 views

General approach on time series for customer retention/churn in retail

I have a time series of data in the following form: ...
0
votes
1answer
21 views

Interpreting DataFrame.where() documentation

From examples outside of the documentation, I thought I understood the examples of the .where() method. Basically, it seems to be a another way to filter a dataframe. However, when I checked the ...
-1
votes
1answer
137 views

How to plot datasets 1 factors for K mean clustering python

I'm unable to plot the data for K mean clusering algo usingsklearn as it throws this error : TypeError: scatter() missing 1 required positional argument: 'y' Here is the function I have written to ...
0
votes
1answer
39 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
1
vote
1answer
190 views

pandas.isna() vs pandas.DataFrame.isna()

I've seen the two documentation pages for pandas.isna() and pandas.DataFrame.isna() but the difference is still unclear to me. Could someone explain the difference to me using examples?
0
votes
0answers
26 views

How to use countVectorizer and TfidfVectorizer?

I am new to data science and trying to figure things out. I have been asked to create a countVectorizer with binary counts of 1-grams and TfidfVecortizer with 1-gram, both avoiding English stop words. ...
1
vote
2answers
122 views

Pandas: How can I update dataframe values? [closed]

I have two spreadsheets where one is updating the other. How can I update this data using the pandas library? Example, where 'b' updates 'a': ...

1
3 4
5
6 7
18