Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
796
questions
0
votes
1answer
36 views
find the difference between two columns in specific rows
I have the following columns from one data frame
...
0
votes
1answer
14 views
What is the best way to match street addresses in a dataset in Python?
I have a dataset of land parcels owned by the government and I am attempting to match street addresses to an existing list of government agencies. I've used fuzzy matching and used a regex that ...
-2
votes
0answers
8 views
Create Pandas data frame using dictionary and list [closed]
index = {'California': 2000, 'California': 2010, 'New York': 2000,
'New York': 2010, 'Texas': 2000, 'Texas': 2010}
...
2
votes
2answers
25 views
How to perform one hot encoding on multiple categorical columns
I am trying to perform one-hot encoding on some categorical columns. From the tutorial I am following, I am supposed to do LabelEncoding before One hot encoding. I have successfully performed the ...
0
votes
2answers
14 views
How to add jpg images information as a column in a data frame [closed]
I have jpg images stored in a folder. For ex: 11_lion_king.jpg,22_avengers.jpg etc.
I have a data frame as below:
...
0
votes
0answers
14 views
-2
votes
2answers
15 views
Replacing column values starting with [closed]
Hi the following is my data frame.
I would like to replace all the values in in item_id colum to just FOODS.
0
votes
1answer
12 views
0
votes
1answer
12 views
Cannot create new percent columns from CSV file
I am fairly new to coding and cannot figure out how to create the AB_perc and CD_perc columns in the Desired Output table below.
The original csv file I am using has all sorts ot DNA sequencing data, ...
1
vote
0answers
19 views
How to find the position of a company given other companies in a Pandas DataFrame with Python
So I have a Pandas DataFrame. I am doing some data analysis with Python with some sets of ...
0
votes
0answers
17 views
Best practice - memory management when data wrangling
I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below:
...
0
votes
0answers
9 views
fastai - using 'untar_data' function in kaggle kernel
I have recently started with fastai lesson 1 and I am using kaggle to run the course notebooks. While going through the ‘lesson1-pets’ notebook we use ...
10
votes
3answers
2k views
What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not?
I spent the past year learning Python. As a person who thought coding was impossible to learn for those outside of the CS/IT sphere, I was obviously gobsmacked by the power of a few lines of Python ...
-2
votes
1answer
104 views
'numpy.ndarray' object has no attribute 'plot' [closed]
I am trying to balance my data set and using imblearn library for this but After performing fit operation when i try to see the data count in dependent variable its showing me below error
...
-1
votes
0answers
9 views
How to select best features SVM- numerical inputs and categorical output
I have a number of features and I want to reduce the dimensionality to ensure good model accuracy. How do I select the best features where all the inputs are numerical and the outputs are categorical. ...
3
votes
1answer
36 views
Iterate over multiple dataframe rows at the same time
I have 16 different dataframes with the same number of rows/columns and another 2 separate dataframes with that same shape that i'm using to compare with the 16 dataframe values.
I need to loop over ...
4
votes
2answers
101 views
+50
Unformatted data entries
I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
0
votes
0answers
8 views
Dynamically create dependent column based other column values in dash
Guys i have this table in dash the following columns are drop down columns Reason Code, Staging and Overwrite. Based on their values i want to create values under Final Staging column dynamically in ...
0
votes
1answer
15 views
How to transform and cummulate data in python
I would like to transform dataframe and cummulate them using pandas.
year country value
1999 JAPAN 10
2000 KOREA 15
2000 USA 20
2001 USA 13
2002 JAPAN 30
* I want to transform ...
0
votes
0answers
15 views
conversion of any date to datetime in pandas dataframe
I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object.
I want to convert those fields ...
0
votes
2answers
20 views
Updating 1st dataframe columns from 2nd data frame coulmns
I have two data frames df1 and df2 which look something like this.
...
0
votes
1answer
18 views
Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?
If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
0
votes
1answer
20 views
redundancy or functional dependency of two columns
Being a beginner in Data Analysis I ask you not to be much judgemental about this question.
I was tring to find if there is a standard way (function?) in Pandas module to identify redundancy of a ...
1
vote
1answer
49 views
0
votes
1answer
102 views
Problem to display decision tree: InvocationException: GraphViz's executables not found in Python scikit-learn?
I try to display Decision Tree in Jupyter Notebook in Python.
This is my code:
...
3
votes
0answers
25 views
sum of occurrences in bins [migrated]
I'm looking at popularity of food stalls in a pop-up market:
...
0
votes
0answers
15 views
Calculate the correlation coefficient without one-hot encoding the category variable
I just read about Cramers V statistic to calculate the correlation coefficient without one-hot encoding the category variable. I'm trying to use it for finding the correlation coefficient of my target ...
1
vote
0answers
14 views
Resul Logit, meaning of values Python scikit-learn?
What does the marked paces mean ?: Current function value, Df Model, Df Residuals, Pseudo R-squared, Log-Likehood, LL-Null, LLR p-value, Coef, Std Err., z, [0,025 0.975] ?
Could you give mean a short ...
1
vote
0answers
16 views
How do I pivot a table while keeping the other columns intact
I have looked around and I have not been able to exactly get the result I am looking for.
I have the following Dataframe:
My goal, is to have a dataframe that for every year-state combination, has ...
0
votes
0answers
13 views
DataFrame to Dictionary with different numbers of values Pandas [migrated]
I am trying to create a dictionary from a DataFrame where the key sometimes has multiple values.
For example:
...
-2
votes
0answers
20 views
Clustering for Mixed Categorical and Numeric Data
My dataset contains three categorical variables and three numerical variables.
brand model trim mileage price age
honda civic vti oriel 91000 1750000 8
I need to find ...
1
vote
0answers
22 views
How to plot a bar and line on the same chart with separate axes? [migrated]
I'm trying to plot bars and a line on different y axes on the same chart using matplotlib. They don't appear on the same chart for some reason, the following code works in certain situations but it ...
-1
votes
1answer
39 views
Importance of variable to implement in Logistic Regression model in Python sklearn?
Logistic regression using Python sklearn
In Logistic Regression model we can implement variables which meet 3 conditions:
independent variables have to be correlated with dependent variable
...
0
votes
0answers
17 views
How to label sequential data for LSTM usage
I want to process data to feed it to a LSTM later, each 100 rows correspond to single category how should I label the data? Should I concatenate the 100 rows into a single row?
Data Sample:-
...
0
votes
1answer
23 views
How to obtain unique count of categorical variable based on another categorical variable?
I have a dataframe with date and cat_1 variables among other variables. I want to get number of unique values of ...
-1
votes
1answer
36 views
Sampling randomly from pd.DataFrame, but ignoring NaN values
I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column.
My sampling function at the moment looks like this:
...
1
vote
1answer
17 views
In feature selection, I came across a situation where NaN were filled by median of the column values
Why the median value is used for NaN? Why not something else like mean? What is the logic behind using the median value?
0
votes
1answer
42 views
I have a pandas dataframe and i need to clear all the special characters other than space
Input:
import pandas as pd
df=pd.read_excel("OCRFinal.xlsx")
df['OCR_Text']=df['OCR_Text'].str.replace(r'\W+'," ")
print(df['OCR_Text'])
Output:
The excel ...
-1
votes
1answer
32 views
Custom Pandas function for formatting
My data consists of a lot of Dataframes having the format as below:
...
0
votes
1answer
29 views
0
votes
0answers
18 views
How to deal with errors of defining data types in pandas' read_csv ()?
I have a table with 118,000 rows and 80 columns. I would like to select 8 columns from the table. I am reading the file using the pandas function pd.read_csv ...
-1
votes
1answer
2k views
0
votes
1answer
38 views
pivoting two column in pandas
How can I transfer two column features into pivot table on the following dataset
I have tried the aggfunc function but this fill the value either 0 or 1. I want to transfer the row as cell value.
...
1
vote
2answers
28 views
Replace Values with corresponding numbers in the same dataframe
Please help me to replace NR values with corresponding values as in the dataframe.
Thanks,
Phani.
2
votes
1answer
26 views
Assign a unique cluster based on a dataframe column with KMeans Algorithm [closed]
I have the following df
...
0
votes
0answers
6 views
Handling numpy data in highly frequently updated environment
Having the following environment:
Where the input rows (150 million per day) are getting directed into 500 independent DataFrames (300k rows per DataFrame).
In ...
0
votes
1answer
33 views
What is the best way to store images in python for machine learning
I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values.
So far ...
0
votes
1answer
20 views
How to sum all price values having in count the quantity of each one
Using pandas, I would like to sum all of the price to get the count value for each one. The Dataframe looks like this:
...
1
vote
2answers
64 views
is it ok to get 100% accuracy in random forest classifier algorithm?
while i was building the model to predict the performance of machine using the features like OEF,working time,performance/head etc...
I splitted the training data using
...
1
vote
1answer
50 views
Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?
I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
