Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
609
questions
0
votes
0answers
12 views
How to put the variable names of pandas data frame on Diagonal of Seaborn PairGrid?
The given below code is not working now(earlier it was working) to put the column names on the diagonal of Seaborn PairGrid.
...
2
votes
0answers
20 views
Filtering a panda dataframe in one line
I had the following data-cleaning question in an interview test that I struggled on (I've changed the details to anonymise it and protect the company's interview process)
Given the following ...
-1
votes
1answer
31 views
MATCHING AN INTEGER
I have this dataset. I want to create a dataframe from the dataset which starts with a particular integers.
This is a sample:
...
1
vote
1answer
22 views
0
votes
0answers
12 views
Pandas - how to make further sub-groups after groupby( ) and get_group( )
Sample data in .csv format
...
0
votes
0answers
8 views
Scale a column with respect to the deviation in another
I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
1
vote
2answers
49 views
pandas - under a column, count the total number of a specific value, instead of using value_counts()
value_counts()
function outputs the number of all unique values in a column, for example
apple 3
orange 2
banana 1
I ...
0
votes
0answers
19 views
Error when checking input: expected dense_67_input to have shape (9,) but got array with shape (8,)
I have a dataset like
ab, 1:8 9:9 10:10
cd, 2:0 3:1
mn, 4:5
om, 6:9 9:5
First read and encode data like
...
0
votes
0answers
16 views
how to prevent pandas from rewriting formatted header to csv for every chunk [closed]
I am reading csv file into pandas in chunks and replacing csv header with new header but it's writing header after reading and writing chunk to the output file. I have tried several ways to exclude it ...
1
vote
1answer
19 views
Removing Rows based on criteria using python
This is just a dummy data, the actual data has over 9,000 rows.
The goal is to remove all the rows with wit (Rev) in Desc columns and also remove the row that <...
0
votes
3answers
32 views
Select samples from a dataframe in python [closed]
I have a data set (pandas dataframe) with a variable that corresponds to the country for each sample. I have to take the samples that corresponds with the countries that appears the most.
thanks
-2
votes
0answers
14 views
Error in changing datetime format
I am getting errors trying to convert datetime format. My code is:
...
0
votes
0answers
17 views
Upload 'tsv' file to Google Colab / Jupyter Notebook
TSV(Tab separated Value) extension file can't be uploaded to google colab using pandas
Used this to upload my file
...
0
votes
1answer
18 views
Appending the values of multiple columns into one whilst retaining UIDs
I have a table structured as below:
UID|ColA|ColB|ColC|ColD
I want to restructure so that all columns are combined (appended) into one, retaining the UID of each ...
1
vote
3answers
28 views
convert dataframe without header to dictionary with a row of number
I am trying to put a dataframe into dictionary ,with the first column as the key ,the numbers in a row would be the value . how should I do it ? I tried to_dict but did not work , my df has no header ...
0
votes
1answer
15 views
Add ID information from one dataframe to every row in another dataframe without a common key
I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two.
The first dataframe (df1) contains the identifing information, ...
0
votes
0answers
14 views
1
vote
1answer
22 views
String handling by OneHotEncoder
I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features.
Moreover, the documentation is what looks more ambiguous. Here are the ...
0
votes
1answer
46 views
0
votes
2answers
20 views
I keep getting KeyError
Can anyone help with this error?
import pandas as pd
data = pd.read_csv('test', dtype=str)
data.head()
data = data[pd.notnull(data['Sequence'])]
Data looks like ...
2
votes
3answers
96 views
Pandas change value of a column based another column condition
I have values in column1, I have columns in column2.
What I want to achieve: Condition: where column2 == 2 leave to be 2 if
column1 < 30 elsif change to 3 if column1 > 90.
Here is what i did so ...
0
votes
1answer
19 views
How to barplot output of pandas.describe() from multiple datasets
I'm trying to compare the differences and similarities between 10 dataframes. I have decided to df.describe() each dataframe in turn and accumulate the results into a new dataframe.
...
3
votes
2answers
43 views
How to count the number of times a sequence in a panda (time) series is less than a specific duration
Given a panda series where a value is either {0, +1, -1} and the sequence always starts with a +1 and end with a -1.
For example,
...
1
vote
2answers
37 views
Pandas, make the bars of a bar graph have equal widths
What do I need to change so that ax1 uses the same width for bars as ax2?
...
-1
votes
0answers
28 views
Pandas is not showing missing valued rows
My Train Dataset shape is (5315,4). I could see this shape in excel but when I read this dataset in colab editior using pandas it's showing shape as (5279, 4). In fact , the dataset contains 36 ...
0
votes
0answers
29 views
NLP - How can I encode single words from one feature (No word frequency)
I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
1
vote
1answer
50 views
Any one read the over 100gb csv file and successfully concatenation?
I have been searching for the deal with large CSV file read method
Its over 100gb and need to know how deal with the chunk file processing
and make concatenation faster
...
0
votes
1answer
31 views
How to read specific column with specific row in x_test using python
I have a dataset with four input. here I am trying to predict X1 input in next time period using lstm model.
Here I put x_train 80% and x_test 20% .
In x_test first input some of rows are with 0 ...
0
votes
1answer
38 views
Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies
I'm still in the process of learning, so I'm sorry if this doesn't make much sense.
I'm doing Kaggle learns micro courses, and to work with missing tabular data we learned about using pipelines with ...
0
votes
0answers
11 views
pd.read_csv() not reading /tmp folder AWS Lambda
I am using AWS Lambda to download a CSV from S3 and then upload each recod to a mongo.
I download the csv to /tmp folder using the boto3 client and with pandas read it with pd.read_csv().
However the ...
0
votes
1answer
52 views
1
vote
1answer
29 views
Incorrect correlation results
I am working on a predictive system of a finite outcome with output -1 or 1, depending on the values detected by 10 sensors.
...
0
votes
1answer
18 views
Create new column based on two columns (different length) from different data frames
I have two data frames df3 ['categories'] and df8 with different length.
...
0
votes
0answers
57 views
cannot reshape array of size 4 into shape (4,3)
I have a dataset with three inputs X1,X2,X3. I tried to predict next value of X1 using lstm .
I wrote the code for lstm and while trying model.predict I got an error ...
0
votes
1answer
29 views
Want to apply multiple conditions but not working with Python. Help
Where is my mistake? Please help me
0
votes
1answer
36 views
How can I arrange data in different columns (for different constituents) from one?
I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want ...
0
votes
1answer
25 views
How to match a word from column and compare with other column in pandas dataframe
I have the below dataframe
...
0
votes
0answers
12 views
How do i annotate my markers in such a way to visualize buy and sell? [duplicate]
my dummy data for close price
...
0
votes
1answer
60 views
how to extract substrings from a dataframe column
I have a dataset as below:
col1
{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,jjjhjhd :70
I want to add 2 more cols such as col_50 &...
0
votes
1answer
47 views
Getting stock data in a discipline manner from Yahoo finance
I used the below code for downloading stock data from yahoo finance:-
...
0
votes
2answers
21 views
Find columns with numeric values, but stored as string
I need to find the columns in data frame, which has numeric values and are stored as string.
...
0
votes
1answer
20 views
How to get mean of column using groupby() and another condition [closed]
For the following
df = pd.DataFrame()
df['1'] = 1,2,1,2,1,2
df['2'] = 3,6,5,4,7,8
df['3'] = 1,1,1,2,2,2
I want to do
...
0
votes
1answer
34 views
Pandas datastructure
I'm trying to analyze database performance over a period of time and detect anomalies. The database server consists of many threads that perform different actions. I run a query to determine the ...
0
votes
0answers
25 views
Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment
I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark).
Question
Need a ...
0
votes
2answers
22 views
How to restrict the values of predicted variable to be positive?
I am using Python Linear Regression to predict the weekly orders for a food deliver company. But some of my orders are coming out as negative. Is there any way to restrict the predicted values to be ...
0
votes
1answer
27 views
Why we add a constant value column in our DataFrame sometimes?
Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1.
I need to ...
0
votes
3answers
31 views
String to Data frame column
I have 2 column in data frame, X and Y. And I have some string values stored in text, which I want to put in X, Y as shown in the example.
Example :
...
0
votes
1answer
64 views
Adding and subtract inbetween row inputs and value equal to the first column next row using pandas
Assume I have a dataset with three inputs:
...
2
votes
2answers
243 views
'tuple' object is not callable while reshaping training data using python
I have data csv file with three inputs names temperature, humidity, wind. Here I want to predict temperature value in every 60 minute using LSTM model.
Here I write the code to reshape the train . ...
0
votes
0answers
10 views
How to draw proper chart of distributional tree?
I am using python with matplotlib and need to visualize distribution percentage of sub-groups of an data set.
imagine this tree:
...
