Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

How to put the variable names of pandas data frame on Diagonal of Seaborn PairGrid?

The given below code is not working now(earlier it was working) to put the column names on the diagonal of Seaborn PairGrid. ...
2
votes
0answers
20 views

Filtering a panda dataframe in one line

I had the following data-cleaning question in an interview test that I struggled on (I've changed the details to anonymise it and protect the company's interview process) Given the following ...
-1
votes
1answer
31 views

MATCHING AN INTEGER

I have this dataset. I want to create a dataframe from the dataset which starts with a particular integers. This is a sample: ...
1
vote
1answer
22 views
0
votes
0answers
12 views

Pandas - how to make further sub-groups after groupby( ) and get_group( )

Sample data in .csv format ...
0
votes
0answers
8 views

Scale a column with respect to the deviation in another

I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
1
vote
2answers
49 views

pandas - under a column, count the total number of a specific value, instead of using value_counts()

value_counts() function outputs the number of all unique values in a column, for example apple 3 orange 2 banana 1 I ...
0
votes
0answers
19 views

Error when checking input: expected dense_67_input to have shape (9,) but got array with shape (8,)

I have a dataset like ab, 1:8 9:9 10:10 cd, 2:0 3:1 mn, 4:5 om, 6:9 9:5 First read and encode data like ...
0
votes
0answers
16 views

how to prevent pandas from rewriting formatted header to csv for every chunk [closed]

I am reading csv file into pandas in chunks and replacing csv header with new header but it's writing header after reading and writing chunk to the output file. I have tried several ways to exclude it ...
1
vote
1answer
19 views

Removing Rows based on criteria using python

This is just a dummy data, the actual data has over 9,000 rows. The goal is to remove all the rows with wit (Rev) in Desc columns and also remove the row that <...
0
votes
3answers
32 views

Select samples from a dataframe in python [closed]

I have a data set (pandas dataframe) with a variable that corresponds to the country for each sample. I have to take the samples that corresponds with the countries that appears the most. thanks
-2
votes
0answers
14 views

Error in changing datetime format

I am getting errors trying to convert datetime format. My code is: ...
0
votes
0answers
17 views

Upload 'tsv' file to Google Colab / Jupyter Notebook

TSV(Tab separated Value) extension file can't be uploaded to google colab using pandas Used this to upload my file ...
0
votes
1answer
18 views

Appending the values of multiple columns into one whilst retaining UIDs

I have a table structured as below: UID|ColA|ColB|ColC|ColD I want to restructure so that all columns are combined (appended) into one, retaining the UID of each ...
1
vote
3answers
28 views

convert dataframe without header to dictionary with a row of number

I am trying to put a dataframe into dictionary ,with the first column as the key ,the numbers in a row would be the value . how should I do it ? I tried to_dict but did not work , my df has no header ...
0
votes
1answer
15 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
1
vote
1answer
22 views

String handling by OneHotEncoder

I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features. Moreover, the documentation is what looks more ambiguous. Here are the ...
0
votes
1answer
46 views

Drop all nan value in Series

I have try to drop a nan value like so ...
0
votes
2answers
20 views

I keep getting KeyError

Can anyone help with this error? import pandas as pd data = pd.read_csv('test', dtype=str) data.head() data = data[pd.notnull(data['Sequence'])] Data looks like ...
2
votes
3answers
96 views

Pandas change value of a column based another column condition

I have values in column1, I have columns in column2. What I want to achieve: Condition: where column2 == 2 leave to be 2 if column1 < 30 elsif change to 3 if column1 > 90. Here is what i did so ...
0
votes
1answer
19 views

How to barplot output of pandas.describe() from multiple datasets

I'm trying to compare the differences and similarities between 10 dataframes. I have decided to df.describe() each dataframe in turn and accumulate the results into a new dataframe. ...
3
votes
2answers
43 views

How to count the number of times a sequence in a panda (time) series is less than a specific duration

Given a panda series where a value is either {0, +1, -1} and the sequence always starts with a +1 and end with a -1. For example, ...
1
vote
2answers
37 views

Pandas, make the bars of a bar graph have equal widths

What do I need to change so that ax1 uses the same width for bars as ax2? ...
-1
votes
0answers
28 views

Pandas is not showing missing valued rows

My Train Dataset shape is (5315,4). I could see this shape in excel but when I read this dataset in colab editior using pandas it's showing shape as (5279, 4). In fact , the dataset contains 36 ...
0
votes
0answers
29 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
1
vote
1answer
50 views

Any one read the over 100gb csv file and successfully concatenation?

I have been searching for the deal with large CSV file read method Its over 100gb and need to know how deal with the chunk file processing and make concatenation faster ...
0
votes
1answer
31 views

How to read specific column with specific row in x_test using python

I have a dataset with four input. here I am trying to predict X1 input in next time period using lstm model. Here I put x_train 80% and x_test 20% . In x_test first input some of rows are with 0 ...
0
votes
1answer
38 views

Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies

I'm still in the process of learning, so I'm sorry if this doesn't make much sense. I'm doing Kaggle learns micro courses, and to work with missing tabular data we learned about using pipelines with ...
0
votes
0answers
11 views

pd.read_csv() not reading /tmp folder AWS Lambda

I am using AWS Lambda to download a CSV from S3 and then upload each recod to a mongo. I download the csv to /tmp folder using the boto3 client and with pandas read it with pd.read_csv(). However the ...
0
votes
1answer
52 views

Extracting values in Pandas [closed]

I have a case like below: ...
1
vote
1answer
29 views

Incorrect correlation results

I am working on a predictive system of a finite outcome with output -1 or 1, depending on the values detected by 10 sensors. ...
0
votes
1answer
18 views

Create new column based on two columns (different length) from different data frames

I have two data frames df3 ['categories'] and df8 with different length. ...
0
votes
0answers
57 views

cannot reshape array of size 4 into shape (4,3)

I have a dataset with three inputs X1,X2,X3. I tried to predict next value of X1 using lstm . I wrote the code for lstm and while trying model.predict I got an error ...
0
votes
1answer
29 views

Want to apply multiple conditions but not working with Python. Help

Where is my mistake? Please help me
0
votes
1answer
36 views

How can I arrange data in different columns (for different constituents) from one?

I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want ...
0
votes
1answer
25 views
0
votes
0answers
12 views
0
votes
1answer
60 views

how to extract substrings from a dataframe column

I have a dataset as below: col1 {:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,jjjhjhd :70 I want to add 2 more cols such as col_50 &...
0
votes
1answer
47 views

Getting stock data in a discipline manner from Yahoo finance

I used the below code for downloading stock data from yahoo finance:- ...
0
votes
2answers
21 views

Find columns with numeric values, but stored as string

I need to find the columns in data frame, which has numeric values and are stored as string. ...
0
votes
1answer
20 views

How to get mean of column using groupby() and another condition [closed]

For the following df = pd.DataFrame() df['1'] = 1,2,1,2,1,2 df['2'] = 3,6,5,4,7,8 df['3'] = 1,1,1,2,2,2 I want to do ...
0
votes
1answer
34 views

Pandas datastructure

I'm trying to analyze database performance over a period of time and detect anomalies. The database server consists of many threads that perform different actions. I run a query to determine the ...
0
votes
0answers
25 views

Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment

I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark). Question Need a ...
0
votes
2answers
22 views

How to restrict the values of predicted variable to be positive?

I am using Python Linear Regression to predict the weekly orders for a food deliver company. But some of my orders are coming out as negative. Is there any way to restrict the predicted values to be ...
0
votes
1answer
27 views

Why we add a constant value column in our DataFrame sometimes?

Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1. I need to ...
0
votes
3answers
31 views

String to Data frame column

I have 2 column in data frame, X and Y. And I have some string values stored in text, which I want to put in X, Y as shown in the example. Example : ...
0
votes
1answer
64 views

Adding and subtract inbetween row inputs and value equal to the first column next row using pandas

Assume I have a dataset with three inputs: ...
2
votes
2answers
243 views

'tuple' object is not callable while reshaping training data using python

I have data csv file with three inputs names temperature, humidity, wind. Here I want to predict temperature value in every 60 minute using LSTM model. Here I write the code to reshape the train . ...
0
votes
0answers
10 views

How to draw proper chart of distributional tree?

I am using python with matplotlib and need to visualize distribution percentage of sub-groups of an data set. imagine this tree: ...