Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
0
votes
1answer
17 views
LSTM : Bad Data or Low preparation
I have three features {feature1, feature2, feature3} the middle have negative sign for some values and i am trying to predict the middle one using LSTM with window size of 10 but i get very high RMS ...
0
votes
1answer
9 views
Determining the correlations between aggregated data and non aggregated data
I have a dataset which is essentially a list of lists produced by a sql query output. Here is what it looks like
...
0
votes
2answers
41 views
How to caculate time difference in between rows using loop in panda python
I have csv file with time and date. Here I want to calculate time difference row by row in time column. I wrote a code and when I run it it's not giving time difference row by row. Can anyone help me ...
0
votes
1answer
20 views
Joining two dataframes on the basis of specific conditions [on hold]
I've added the images of both the dataframes here. Now I need to combine the two dataframes on the basis of two conditions:
Condition 1: The element in the 'arrivalTS' column in the first dataframe(...
1
vote
1answer
41 views
'int' object is not iterable using panda python
Here I want to calculate time interval in between row by row in time column import from csv file. my start time 6.00 a.m and my end time is next day 6.00 a.m. In between this time periods how to find ...
0
votes
1answer
14 views
The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() using panda python
enter preformatted text hereHere I have data to import from a CSV file.I wrote an equation inside the class and to solve the equation data will import from the CSV ...
1
vote
0answers
19 views
Allocation from a central stock in Pandas
I am struggling with creating an allocation algorithm against multiple demand.
eg: consider 2 banks operating in multiple cities. For each bank in each city there are available funds that can be ...
1
vote
0answers
17 views
Issues with pandas chunk merge
I'm trying to solve a kaggle competition - https://www.kaggle.com/c/ga-customer-revenue-prediction
Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back ...
0
votes
1answer
28 views
Obtaining a confidence interval for the prediction of a linear regression
The data I am working with is being used to predict the duration of a trip between two points. There are about 100 different trips in the data and ~90k observations.
I am using the standard pattern:
...
2
votes
2answers
35 views
How to transform a folder of images in a csv file
I have a folder with a lot of images that I want to use to bild a classificator using a SVM model in python with sklearn.
I've always used csv file as train/test set with sklearn, how can I make it? (...
-1
votes
1answer
32 views
Handling Categorical Data and Numerical Data Together [on hold]
I have a Dataframe containing 3 features: Product_detail, S.I_Units and Value.
...
4
votes
1answer
57 views
How to calculate Cumulative Sum with Groupby in Python?
I am trying to calculate cumulative sum with groupby using Pandas's DataFrame. However, I don't get expected output.
My Source Code:
...
1
vote
1answer
16 views
Pandas DataFrame: Aggregating multi-level groups by matching keys
I have some data that looks like this;
...
0
votes
0answers
16 views
How do I find frequency of Authors and plot this using Python? [migrated]
Here ABC news is observed 5 times but the column Times reflects it as 1 for each row. ...
0
votes
0answers
28 views
Export values from Pandas DataFrame to Excel [migrated]
I have a Pandas DataFrame that can be reproduced using: ...
0
votes
0answers
8 views
Pandas groupby with numpy array_split take to much time [migrated]
I'm working on to get a lots of rows out around 2.000.000 rows where i'm group it on a single columen.
Its working fine so long, but when i'm in Python3 try to make a Numpy array_split its take for ...
2
votes
2answers
42 views
How to rename those columns which having same name? [closed]
I want to rename the column names , but the data frame contains similar column names. so how to rename..?
df.columns
...
0
votes
1answer
42 views
How to find the count of consecutive same string values in a pandas dataframe? [closed]
Assume that we have the following pandas dataframe:
...
1
vote
0answers
15 views
visualization idea on a panel data (roughly speaking)
I wanted to construct a heat map on the following data:
...
3
votes
1answer
102 views
Pandas: How can I merge two dataframes? [closed]
I found (How do I merge two data frames in Python Pandas?), but do not get the expected result.
I have these two CSV files:
...
0
votes
1answer
11 views
Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data
I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with ...
2
votes
1answer
22 views
memory problem due to large file
I'm new in Python and definitely, I'm sure I do some mistake.
Here is my problem and thanks all to your help in advance.
I have 2 files (one is Hive) and the other is csv and merging them. I have ...
2
votes
0answers
12 views
How to predict multiple iterations in the future based on current close prices? Keras Python
I have data from CoinMarketCap for the close prices of Bitcoin. How can I use this data to predict multiple days into the future using past days with a function? I want the function to look like:
<...
0
votes
0answers
25 views
What is the seaborn equivalent for matplotlilb histrogram of a column values? [migrated]
I have code snippet in matplotlilb like below:
df['coln1'].plot(kind='hist') # This works!
I tried to use the seaborn API to plot the same and my Jupyter ...
-1
votes
0answers
26 views
Choose value in one column based on value subtraction in another column
I have a dataframe containing 3 columns. I want to subtract the last entry of col2 from second to last entry of col2 and see if ...
1
vote
1answer
28 views
Pandas, R, and interoperable data frame formats
I want to have a single format which is compatible both with R and Python's Pandas (and preferably, other things like Stata and SAS), for which I will not ever need to impute whether things are ...
1
vote
1answer
28 views
What are the best practices for data formatting?
I used to do some analysis on Excel but my company want to use Python to increase the analysis.
I am novice on python and overwhelmed by all of this knowledge I have to learn :'(
So for my first ...
0
votes
1answer
23 views
PANDAS Within Category Normalization
I'm want to normalize sales data of multiple point of sales (POS), Products and weeks. The dataframe looks like this:
...
0
votes
1answer
24 views
Replacing column values in pandas with specific column with multiple database operation?
Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can ...
2
votes
3answers
677 views
Can we remove features that have zero-correlation with the target/label?
So I draw a pairplot/heatmap from the feature correlations of a dataset and see a set of features that bears Zero-correlations both with:
every other feature and
also with the target/label
....
3
votes
1answer
49 views
How can I combine two single-column datasets into a single Pandas data frame?
I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label ...
0
votes
1answer
17 views
how to calculate number of datapoints within a given time interval?
I have a dataframe with one column which is a timestamp. I have converted that column to a datetime object using pandas to_datetime() method. However what I want is to ...
0
votes
2answers
47 views
how to make a file name as a column in panda? [closed]
i have a CSV file which name is data.CSV and i want to make column from file name for example data is column and value must be data as well how can i do this?
0
votes
2answers
68 views
how to convert multiple columns into single columns in pandas? [closed]
I have a dataframe like this
my desire format is like this
how can i do this?
0
votes
0answers
17 views
What is the best database to use for storing timeseries created in python but viewed in excel?
I currently use Python/Pandas to extract time series and then store them in CSV files. I then link the CSV to an excel file using the get data/from file/csv and it can easily be refreshed each time I ...
1
vote
2answers
48 views
pandas: how to change the specific column as index and change index into various columns
Hi I'm new to data science. Learning data science from course-era.
I'm having pandas data frame as follows,
...
0
votes
2answers
30 views
0
votes
1answer
59 views
how to make new columns using another column in pandas?
i have a dataframe like this
i want to add new column and my desire format is like this
i use indexing eg
dll.loc[:6,'category'] = "sectowise"
dll.loc[7:11,'category'] = "productwise"
dll.loc[12:16,...
1
vote
1answer
39 views
how to create new columns in pandas using some rows of existing columns?
i have a dataset like this
my desire format is like this
I tried using index slicing eg
dll.loc[:4,'category'] = "CAPITAL FUND"
dll.loc[5:10,'category'] = "BORROWING"
but this idea is risky so is ...
1
vote
1answer
42 views
Prediction based on more dataframes [closed]
I'm trying to predict a score that user gives to a restaurant.
The data I have can be grouped into two dataframes
data about user (taste, personal traits, family, ...)
data about restaurant(open ...
1
vote
1answer
32 views
Combine results from multiple models
I am using chunks of 100000 rows at a time from the CSV file to train the a simple LASSO model.
How do i combine all of these models trained from these different chunks? I would like to use all ...
0
votes
1answer
23 views
How can I merge 2+ DataFrame objects without duplicating column names?
This is for work. TLDR: Bottom-line question at the bottom.
I am gathering and parsing test results produced by an old test setup whose output formatting is not likely to change anytime soon. I've ...
1
vote
1answer
309 views
Mapping column values of one DataFrame to another DataFrame using a key with different header names
I have two data frames df1 and df2 which look something like this.
...
1
vote
2answers
111 views
Pandas Conditional Fill NaN Forward/Backward
Updated 22 Oct. 2018:
I have the following dataset:
...
0
votes
1answer
66 views
pandas: How to impute the categorical column by the nearest neighbors?
I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...
-1
votes
2answers
64 views
how to transformation of row to column and column to row in python pandas? [closed]
I have a large dataset
I want to transform this dataset into this format
I have try it through transpose but i couldn't figure out
0
votes
3answers
91 views
How to access substrings in pandas column and store it into new columns?
I'm working on a dataset for building permits. In the dataset there is a column that gives the location (lattitude and longitude) for the building permit. The data in the location column look like ...
0
votes
3answers
147 views
How to replace a part string value of a column using another column
How to replace a part string value of a column using another column.
My DataSet here is :
...
0
votes
1answer
33 views
Missing Values in Classification
I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't.
I also have ...
0
votes
1answer
51 views
Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?
I am running the Kaggle Video Games sales dataset through an XGboost algo.
I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5
when I use:
data = pd.get_dummies(data=...
