Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
1answer
36 views

find the difference between two columns in specific rows

I have the following columns from one data frame ...
0
votes
1answer
14 views

What is the best way to match street addresses in a dataset in Python?

I have a dataset of land parcels owned by the government and I am attempting to match street addresses to an existing list of government agencies. I've used fuzzy matching and used a regex that ...
-2
votes
0answers
8 views

Create Pandas data frame using dictionary and list [closed]

index = {'California': 2000, 'California': 2010, 'New York': 2000, 'New York': 2010, 'Texas': 2000, 'Texas': 2010} ...
2
votes
2answers
25 views

How to perform one hot encoding on multiple categorical columns

I am trying to perform one-hot encoding on some categorical columns. From the tutorial I am following, I am supposed to do LabelEncoding before One hot encoding. I have successfully performed the ...
0
votes
2answers
14 views

How to add jpg images information as a column in a data frame [closed]

I have jpg images stored in a folder. For ex: 11_lion_king.jpg,22_avengers.jpg etc. I have a data frame as below: ...
0
votes
0answers
14 views

Pandas Key Error: Fine Above/Below Error [closed]

I have a pandas series such that: ...
-2
votes
2answers
15 views

Replacing column values starting with [closed]

Hi the following is my data frame. I would like to replace all the values in in item_id colum to just FOODS.
0
votes
1answer
12 views
0
votes
1answer
12 views

Cannot create new percent columns from CSV file

I am fairly new to coding and cannot figure out how to create the AB_perc and CD_perc columns in the Desired Output table below. The original csv file I am using has all sorts ot DNA sequencing data, ...
1
vote
0answers
19 views

How to find the position of a company given other companies in a Pandas DataFrame with Python

So I have a Pandas DataFrame. I am doing some data analysis with Python with some sets of ...
0
votes
0answers
17 views

Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below: ...
0
votes
0answers
9 views

fastai - using 'untar_data' function in kaggle kernel

I have recently started with fastai lesson 1 and I am using kaggle to run the course notebooks. While going through the ‘lesson1-pets’ notebook we use ...
10
votes
3answers
2k views

What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not?

I spent the past year learning Python. As a person who thought coding was impossible to learn for those outside of the CS/IT sphere, I was obviously gobsmacked by the power of a few lines of Python ...
-2
votes
1answer
104 views

'numpy.ndarray' object has no attribute 'plot' [closed]

I am trying to balance my data set and using imblearn library for this but After performing fit operation when i try to see the data count in dependent variable its showing me below error ...
-1
votes
0answers
9 views

How to select best features SVM- numerical inputs and categorical output

I have a number of features and I want to reduce the dimensionality to ensure good model accuracy. How do I select the best features where all the inputs are numerical and the outputs are categorical. ...
3
votes
1answer
36 views

Iterate over multiple dataframe rows at the same time

I have 16 different dataframes with the same number of rows/columns and another 2 separate dataframes with that same shape that i'm using to compare with the 16 dataframe values. I need to loop over ...
4
votes
2answers
101 views
+50

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
0
votes
0answers
8 views

Dynamically create dependent column based other column values in dash

Guys i have this table in dash the following columns are drop down columns Reason Code, Staging and Overwrite. Based on their values i want to create values under Final Staging column dynamically in ...
0
votes
1answer
15 views

How to transform and cummulate data in python

I would like to transform dataframe and cummulate them using pandas. year country value 1999 JAPAN 10 2000 KOREA 15 2000 USA 20 2001 USA 13 2002 JAPAN 30 * I want to transform ...
0
votes
0answers
15 views

conversion of any date to datetime in pandas dataframe

I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I want to convert those fields ...
0
votes
2answers
20 views

Updating 1st dataframe columns from 2nd data frame coulmns

I have two data frames df1 and df2 which look something like this. ...
0
votes
1answer
18 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
0
votes
1answer
20 views

redundancy or functional dependency of two columns

Being a beginner in Data Analysis I ask you not to be much judgemental about this question. I was tring to find if there is a standard way (function?) in Pandas module to identify redundancy of a ...
1
vote
1answer
49 views

Python dataframe iterate over rows

Dataset ...
0
votes
1answer
102 views

Problem to display decision tree: InvocationException: GraphViz's executables not found in Python scikit-learn?

I try to display Decision Tree in Jupyter Notebook in Python. This is my code: ...
3
votes
0answers
25 views

sum of occurrences in bins [migrated]

I'm looking at popularity of food stalls in a pop-up market: ...
0
votes
0answers
15 views

Calculate the correlation coefficient without one-hot encoding the category variable

I just read about Cramers V statistic to calculate the correlation coefficient without one-hot encoding the category variable. I'm trying to use it for finding the correlation coefficient of my target ...
1
vote
0answers
14 views

Resul Logit, meaning of values Python scikit-learn?

What does the marked paces mean ?: Current function value, Df Model, Df Residuals, Pseudo R-squared, Log-Likehood, LL-Null, LLR p-value, Coef, Std Err., z, [0,025 0.975] ? Could you give mean a short ...
1
vote
0answers
16 views

How do I pivot a table while keeping the other columns intact

I have looked around and I have not been able to exactly get the result I am looking for. I have the following Dataframe: My goal, is to have a dataframe that for every year-state combination, has ...
0
votes
0answers
13 views

DataFrame to Dictionary with different numbers of values Pandas [migrated]

I am trying to create a dictionary from a DataFrame where the key sometimes has multiple values. For example: ...
-2
votes
0answers
20 views

Clustering for Mixed Categorical and Numeric Data

My dataset contains three categorical variables and three numerical variables. brand model trim mileage price age honda civic vti oriel 91000 1750000 8 I need to find ...
1
vote
0answers
22 views

How to plot a bar and line on the same chart with separate axes? [migrated]

I'm trying to plot bars and a line on different y axes on the same chart using matplotlib. They don't appear on the same chart for some reason, the following code works in certain situations but it ...
-1
votes
1answer
39 views

Importance of variable to implement in Logistic Regression model in Python sklearn?

Logistic regression using Python sklearn In Logistic Regression model we can implement variables which meet 3 conditions: independent variables have to be correlated with dependent variable ...
0
votes
0answers
17 views

How to label sequential data for LSTM usage

I want to process data to feed it to a LSTM later, each 100 rows correspond to single category how should I label the data? Should I concatenate the 100 rows into a single row? Data Sample:- ...
0
votes
1answer
23 views

How to obtain unique count of categorical variable based on another categorical variable?

I have a dataframe with date and cat_1 variables among other variables. I want to get number of unique values of ...
-1
votes
1answer
36 views

Sampling randomly from pd.DataFrame, but ignoring NaN values

I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column. My sampling function at the moment looks like this: ...
1
vote
1answer
17 views

In feature selection, I came across a situation where NaN were filled by median of the column values

Why the median value is used for NaN? Why not something else like mean? What is the logic behind using the median value?
0
votes
1answer
42 views

I have a pandas dataframe and i need to clear all the special characters other than space

Input: import pandas as pd df=pd.read_excel("OCRFinal.xlsx") df['OCR_Text']=df['OCR_Text'].str.replace(r'\W+'," ") print(df['OCR_Text']) Output: The excel ...
-1
votes
1answer
32 views

Custom Pandas function for formatting

My data consists of a lot of Dataframes having the format as below: ...
0
votes
1answer
29 views

Reshaping Pandas DataFrame

I am trying to do the following as shown below. ...
0
votes
0answers
18 views

How to deal with errors of defining data types in pandas' read_csv ()?

I have a table with 118,000 rows and 80 columns. I would like to select 8 columns from the table. I am reading the file using the pandas function pd.read_csv ...
0
votes
1answer
38 views

pivoting two column in pandas

How can I transfer two column features into pivot table on the following dataset I have tried the aggfunc function but this fill the value either 0 or 1. I want to transfer the row as cell value. ...
1
vote
2answers
28 views

Replace Values with corresponding numbers in the same dataframe

Please help me to replace NR values with corresponding values as in the dataframe. Thanks, Phani.
2
votes
1answer
26 views
0
votes
0answers
6 views

Handling numpy data in highly frequently updated environment

Having the following environment: Where the input rows (150 million per day) are getting directed into 500 independent DataFrames (300k rows per DataFrame). In ...
0
votes
1answer
33 views

What is the best way to store images in python for machine learning

I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values. So far ...
0
votes
1answer
20 views

How to sum all price values having in count the quantity of each one

Using pandas, I would like to sum all of the price to get the count value for each one. The Dataframe looks like this: ...
1
vote
2answers
64 views

is it ok to get 100% accuracy in random forest classifier algorithm?

while i was building the model to predict the performance of machine using the features like OEF,working time,performance/head etc... I splitted the training data using ...
1
vote
1answer
50 views

Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...

1
2 3 4 5
16