Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1
vote
1answer
11 views
How to find the values that never matched in a full join
Hello I have done this full join query:
...
1
vote
0answers
23 views
Inputting (a lot of )data into a dataframe one row at a time
I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this:
...
0
votes
0answers
9 views
How to combine scipy sparse csr matrix to pandas dataframe. | Combining text feature with numerical features
I have both textual and numerical features on my data set. (1 textual feature, 4 numeric features).
For the textual feature I use tfidf vectoriser to convert my textual data into numbers.
X_train is ...
0
votes
1answer
26 views
MAE,MSE and MAPE aren't comparable?
I'm a newbie in data science. I'm working on a regression problem. I'm getting 2.5 MAPE. 400 MAE 437000 MSE. As my MAPE is quite low but why I'm getting high MSE and MAE? This is the link to my data
...
0
votes
2answers
28 views
Trying to change pandas column dtype from str to float
I am trying to convert a pandas column from a str to a float. Before I convert the strings to floats using astype(float), I need to remove characters from the string which cannot be converted into ...
1
vote
0answers
9 views
Visulazing a specific value of a column with the corresponding values
I have recently started using data studio to make visualizations before diving into analysis and data prep!
I have a column named NAME OF STORE that good different names and other columns like Date, ...
1
vote
1answer
22 views
How to place multiple graphs in same coordinate system (pandas,mathplotlib) -> look pic [on hold]
in short how to set graphs in same coordinate system, not separated as in pic each in its own coordinate system in one frame....
2
votes
1answer
37 views
Plotting three lines on the same plot (with four hr frequency)
I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used:
...
0
votes
0answers
14 views
is there any package in python similar to tidyrin R? Except Pandas [on hold]
is there any package in python similar to tidyr in R? Except Pandas
0
votes
1answer
22 views
How to measure correlation between several categorical features and a numerical label in Python?
I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product ...
3
votes
1answer
36 views
Is shuffling training data beneficial for machine learning?
I was curious to know if shuffling ML training data is beneficial to better results?
Sorry not a lot of wisdom here, but I have been reading a post from pythonprogramming.net for this topic.
I ...
1
vote
1answer
24 views
Breaking a DataFrame based on Specific Column Values [closed]
Assume I have a following dataframe which has years of experience and salary.
I would like to create a new dataframe from the existing one, only including observations where salary is above 45000. I ...
1
vote
0answers
12 views
Training anomaly detection on text string [on hold]
I have text string stored in a column, resulting after pre-processing of web traffic data. Now I want to apply anomaly detection on it and for that purpose I applied ...
3
votes
0answers
35 views
Why I am getting prediction score 1 i.e. 100%
I am reading few parameters and trying to predict target value using Linear regression and GB. Surpicingly I am getting score = 1 on test data. How come? Can anyone tell me whats wrong with this code?
...
1
vote
1answer
12 views
How do I handle string feature while performing model generation
I have data which looks like this
...
1
vote
0answers
7 views
Pandas DataFrame Rollup Error
I am trying to compute Percentage Contributions on a pandas dataframe. I have a data frame which has three columns (Buyer, Product, CountofProductBought).
I create a new dataframe using groupby at ...
1
vote
0answers
22 views
Selecting on multiple criteria [migrated]
I made this dataframe, which contains dates as datetime64 values.
What I want to do is a bit of a stupid example, but it illustrates my point of selecting on multiple criteria.
I want to:
For the ...
0
votes
0answers
5 views
merge 2 dataframe with Memory Error
trying to merge 2 data frame size with (13647309, 48)
and I'm using 32 memory.
df_train = train.merge(train_lag,on=['ncodpers','int_date'], how='left')
after I ...
0
votes
0answers
19 views
How to convert a time series data which can be fed to a Machine Leanring model for unsupervised learning?
I am very new to time series data. I am working with server data and want to classify if the bunch of server times are healthy or not.
My data looks like this in pandas
...
3
votes
1answer
18 views
Group datetime64 values per week in dataframe
I want to plot a bar of my data grouped per week. I made the following dataframe, with nonsense data, but it should do the trick.
What I want to do is plot per week a bar chart (%)
Does anyone know ...
2
votes
1answer
30 views
How to fill in missing value of the mean of the other columns?
I had a movie dataset including 'budget' and 'genres' attributes.
I'd like to fill in the missing value of budget with the mean budget of each genre.
I first create two dataframes with or without ...
1
vote
2answers
44 views
Dataframe has no column names. How to add a header?
I am using a dataset to practice for building a decision tree classifier.
Here is my code:
...
1
vote
2answers
32 views
Numpy array from pandas dataframe
I am new in using python for data science.
What is the difference between selecting a a column with: df['name'].values and ...
1
vote
0answers
18 views
Split datetime64 column in dataframe on year
I have a column in my dataframe containing data in the datetime64 format.
so example: 2005-02-01 00:00:00 and 2007:02-02 00:00:00.
I want to split this into multiple columns which per year. Does ...
1
vote
0answers
32 views
PySpark v Pandas Dataframe Memory Issue
Suppose I have a csv file with 20k rows, when I import in Pandas dataframe format and run the ML algos like Random Forest or Logistic Regression from sklearn package it just runs fine.
But same thing ...
2
votes
1answer
38 views
Better code completion in python notebooks
How come python code suggestion is awful in notebooks and spyder? Smetimes it shows classes methods and variables, sometimes not
1
vote
4answers
53 views
operating on a dataset with 125,497,040 records [closed]
I'm trying to run add_datepart() which converts a df column from a datetime64 to many columns in place
Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear','Ismonthend', 'Ismonthstart', '...
1
vote
1answer
57 views
How to compare two dataframes and put the counted unique values in a first dataframe's column? [closed]
I have two different (geo)dataframes, one has 690 and the other has 1826 rows. The first one is a grouped based on the nearness (spatial near) of the second dataframe. Thus, they bound with FID_1 and ...
2
votes
2answers
104 views
Can I save prediction value in same csv file as a another column using panda python
I have csv data file and I design LSTM model to predict values. Then I want to save that prediction value in same csv file. Can I do that? I tried using one code then in my csv file only had ...
0
votes
1answer
33 views
How do I add together multiple columns based on header names?
Here's what my dataframe looks like
...
0
votes
0answers
13 views
Organizing Results of a simulation (MC) in Pandas
I am running a simulation and finding a rank for several different products with each simulation. So for each simulation I will get a pandas dataframe looking something like this (my real example is ...
1
vote
1answer
15 views
Using Pandas, how do I strip part of a cell from an Excel file?
Thanks in advance, I have been struggling with this for a couple of hours now. I have a column named 'CaseNumber'. the data in that column is structured as 'Case ID Number - 00123456'.
I am trying ...
1
vote
0answers
31 views
How to calculate customer purchase interval and predict next purchase in python?
Suppose we have a data set consists of columns
TransactionId, CardNo, TransactionDate
then how can we calculate the customer purchase interval (means if customer A purchased on Jan 1st and after ...
0
votes
0answers
11 views
How to create new columns from existing columns with get_dummies [migrated]
I have this Dataframe:
...
0
votes
0answers
15 views
String Matching columns based on weight
I have the following 2 DataFrames and want to match skills to the key_skills column based on weights.
for eg:
suppose student 1-{python, Java}
job - {python, Ruby, Perl, etc }
weight = 1
In the end,...
0
votes
1answer
43 views
data science student [closed]
I need to replace a specific value in a particular column with NaN. Here is what I tried but it didn't work.
...
0
votes
0answers
6 views
Pandas keyError : ('%s not in index' % objarr[mask]) [migrated]
I am a beginner in the field of Machine Learning. I have started with the below code:
...
-1
votes
0answers
17 views
Apply to original Pandas Dataset result of gpoupby agregation
I have some original dataset with card information
also i have second dataset with transactions by this cards
and i made groupby and some operation and have such result
I need to merge this result ...
1
vote
1answer
62 views
Merging dataframes in Pandas is taking a surprisingly long time
I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could ...
0
votes
2answers
48 views
Change values of consecutive repeating letters in pandas column
I've got one Dataframe like:
id value block
1 a 1
2 a 1
3 a 2
4 a 2
5 b 3
6 c 4
And I want to change the value column to the next value based where
the series ...
1
vote
0answers
24 views
IQR Score outliers detection in Python [closed]
I have a large dataset "train" incl. a column ClientFreeSource from which I want to exclude an outliers. For that purpose I decide to use IQR score.
I've tried:
...
1
vote
2answers
38 views
Reg. Pandas factorize()
-Hi Experts-
I just read about factorise() function in Pandas. Using this I'm able to encode (enumerate) my string values into numbers. But, now I'm not able to understand what numbers corresponds ...
0
votes
0answers
15 views
Repeated groups of columns in data analysis
I have a got a medical records for analysis, which have repeated groups of columns holding information about the medicines taken.
The records are initially stored as an Excel table, having multiple ...
0
votes
0answers
212 views
ValueError: Found input variables with inconsistent numbers of samples
I am trying to apply randomized search from scikit learn to my Neural Network but i am having trouble with shape of X and Y
My code follows
...
1
vote
1answer
43 views
Oversampling before Cross-Validation, is it a problem?
I have a multi-class classification problem to solve which is highly imbalanced. Obviously I'm doing oversampling, but I'm doing cross-validation with the over-sampled dataset, as a result of which I ...
2
votes
2answers
29 views
Filtering Pandas Dataframe Results without saving to a variable first
Is there a way to more efficiently filter a result on a data frame without having to explicitly save it in a variable and then filter? For instance, in the code below I would like to add something to ...
0
votes
0answers
16 views
ANOVA Cohen's d matrix using python
I'm running a one-way ANOVA within 7 groups and would like to create a matrix containing Cohen's d for each couple of groups. To generalize the problem, it's creating a matrix based on one vector and ...
0
votes
1answer
34 views
Machine Learning library in Python, list or numpy or pandas
Background: We are trying to build a customized ML library in Python 3 to tackle analysis we often repeat, in a general fashion. But it would not be nearly as general as ...
3
votes
2answers
51 views
Mean across every several rows in pandas
I have a table of features and labels where each row has a time stamp. Labels are categorical. They go in a batch where one label repeats several times. Batches with the same label do not have a ...
0
votes
0answers
29 views
Clustering geodata into same size group with K-means in Python
I have the following sample geo-data which have 100 rows, and I want to cluster those POIs to 10 groups and 10 points in each group, also, if possible data in each group from same areas.
...
