Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
4
votes
0answers
33 views

Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
0
votes
1answer
77 views

How to retrieve images from a url in a pandas dataframe and store them as PIL object in a new column

I'm trying to store as a PIL object in a new column of a dataframe pictures that are located in a column of the same dataframe in the form of URL's. I've tried the following code: ...
1
vote
1answer
28 views

normal conda install vs forge install

In order to install some libraries, we just use: conda install pandas While in other cases we use: ...
2
votes
1answer
212 views

How to shuffle only a fraction of a column in a Pandas dataframe?

I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe. How would you do it? Is there a simple idiomatic way to do that, maybe using ...
0
votes
0answers
24 views

How to merge all the rows with same date

I have the data as in below screenshot and I want the output as the second screenshot where all the rows of a date are merged. Thanks for the help.
0
votes
2answers
97 views

Conditional Statement to update columns based on range

I have a dataframe called clients that has 5000+ rows with the following: ...
1
vote
1answer
36 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
0
votes
1answer
31 views

Populate free space between two dates

I'm new to data science. I'm trying to increase the time-series length for a special calculation. In the original time-series I have 20 weekly reports and I want to increase the amount of occurrences ...
0
votes
1answer
70 views

Choice of f1 score for highly imbalanced dataset?

I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation. Given my dataset is highly imbalanced(600:100000)
2
votes
1answer
114 views

Difference between sklearn make_pipeline and imblearn make_pipeline

Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.
0
votes
0answers
16 views

how to run pyspark on Hadoop cluster, cache data, then read file using pandas on local machine via its URI?

I connect to a Hadoop cluster via vpn. I want to execute a script that makes sql call and saves the data to a file on the remote machine, then using its URI I want to read on my local running Jupiter ...
0
votes
1answer
55 views

Why is my Python Altair chart printing a blank line?

Wondering if anyone can help me understand why python altair chart is not printing... I’m only seeing this when I run the code: It literally isn’t outputting anything. This is what my chart looks ...
0
votes
1answer
40 views

Make column as dictionary key and row as value in pandas dataframe

I have Dataframe as below. name value id A 123 1 B 345 5 C 567 4 D 789 2 I need to create a ...
1
vote
1answer
38 views

How to put the variable names of pandas data frame on Diagonal of Seaborn PairGrid?

The given below code is not working now(earlier it was working) to put the column names on the diagonal of Seaborn PairGrid. ...
2
votes
1answer
53 views

Filtering a panda dataframe in one line

I had the following data-cleaning question in an interview test that I struggled on (I've changed the details to anonymise it and protect the company's interview process) Given the following ...
-1
votes
2answers
50 views

MATCHING AN INTEGER

I have this dataset. I want to create a dataframe from the dataset which starts with a particular integers. This is a sample: ...
1
vote
1answer
27 views
0
votes
1answer
148 views

Pandas - how to make further sub-groups after groupby( ) and get_group( )

Sample data in .csv format ...
0
votes
0answers
8 views

Scale a column with respect to the deviation in another

I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
1
vote
2answers
210 views

pandas - under a column, count the total number of a specific value, instead of using value_counts()

value_counts() function outputs the number of all unique values in a column, for example apple 3 orange 2 banana 1 I ...
0
votes
3answers
109 views

Select samples from a dataframe in python [closed]

I have a data set (pandas dataframe) with a variable that corresponds to the country for each sample. I have to take the samples that corresponds with the countries that appears the most. thanks
0
votes
1answer
131 views

Upload 'tsv' file to Google Colab / Jupyter Notebook

TSV(Tab separated Value) extension file can't be uploaded to google colab using pandas Used this to upload my file ...
0
votes
1answer
23 views

Appending the values of multiple columns into one whilst retaining UIDs

I have a table structured as below: UID|ColA|ColB|ColC|ColD I want to restructure so that all columns are combined (appended) into one, retaining the UID of each ...
1
vote
3answers
366 views

convert dataframe without header to dictionary with a row of number

I am trying to put a dataframe into dictionary ,with the first column as the key ,the numbers in a row would be the value . how should I do it ? I tried to_dict but did not work , my df has no header ...
0
votes
1answer
24 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
0
votes
0answers
156 views

Better way to fix Json error: Invalid control character at: line 1 column 116 (char 115)

While running my script, to parse the Json file which I've done every month, but this time I got an error that says with a new set of data that say: ...
1
vote
1answer
170 views

String handling by OneHotEncoder

I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features. Moreover, the documentation is what looks more ambiguous. Here are the ...
0
votes
2answers
68 views

Drop all nan value in Series

I have try to drop a nan value like so ...
3
votes
3answers
9k views

Pandas change value of a column based another column condition

I have values in column1, I have columns in column2. What I want to achieve: Condition: where column2 == 2 leave to be 2 if column1 < 30 elsif change to 3 if column1 > 90. Here is what i did so ...
0
votes
1answer
22 views

How to barplot output of pandas.describe() from multiple datasets

I'm trying to compare the differences and similarities between 10 dataframes. I have decided to df.describe() each dataframe in turn and accumulate the results into a new dataframe. ...
3
votes
2answers
83 views

How to count the number of times a sequence in a panda (time) series is less than a specific duration

Given a panda series where a value is either {0, +1, -1} and the sequence always starts with a +1 and end with a -1. For example, ...
1
vote
2answers
60 views

Pandas, make the bars of a bar graph have equal widths

What do I need to change so that ax1 uses the same width for bars as ax2? ...
0
votes
0answers
34 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
2
votes
1answer
271 views

Any one read the over 100gb csv file and successfully concatenation?

I have been searching for the deal with large CSV file read method Its over 100gb and need to know how deal with the chunk file processing and make concatenation faster ...
0
votes
1answer
47 views

How to read specific column with specific row in x_test using python

I have a dataset with four input. here I am trying to predict X1 input in next time period using lstm model. Here I put x_train 80% and x_test 20% . In x_test first input some of rows are with 0 ...
0
votes
1answer
96 views

Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies

I'm still in the process of learning, so I'm sorry if this doesn't make much sense. I'm doing Kaggle learns micro courses, and to work with missing tabular data we learned about using pipelines with ...
0
votes
0answers
23 views

pd.read_csv() not reading /tmp folder AWS Lambda

I am using AWS Lambda to download a CSV from S3 and then upload each recod to a mongo. I download the csv to /tmp folder using the boto3 client and with pandas read it with pd.read_csv(). However the ...
0
votes
1answer
146 views

Extracting values in Pandas [closed]

I have a case like below: ...
2
votes
1answer
48 views

Incorrect correlation results

I am working on a predictive system of a finite outcome with output -1 or 1, depending on the values detected by 10 sensors. ...
0
votes
0answers
1k views

cannot reshape array of size 4 into shape (4,3)

I have a dataset with three inputs X1,X2,X3. I tried to predict next value of X1 using lstm . I wrote the code for lstm and while trying model.predict I got an error ...
0
votes
1answer
40 views

How can I arrange data in different columns (for different constituents) from one?

I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want ...
1
vote
1answer
67 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
0
votes
0answers
13 views
0
votes
1answer
687 views

how to extract substrings from a dataframe column

I have a dataset as below: col1 {:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,jjjhjhd :70 I want to add 2 more cols such as col_50 &...
0
votes
1answer
160 views

Getting stock data in a discipline manner from Yahoo finance

I used the below code for downloading stock data from yahoo finance:- ...
0
votes
2answers
28 views

Find columns with numeric values, but stored as string

I need to find the columns in data frame, which has numeric values and are stored as string. ...
0
votes
1answer
185 views

How to get mean of column using groupby() and another condition [closed]

For the following df = pd.DataFrame() df['1'] = 1,2,1,2,1,2 df['2'] = 3,6,5,4,7,8 df['3'] = 1,1,1,2,2,2 I want to do ...
0
votes
1answer
36 views

Pandas datastructure

I'm trying to analyze database performance over a period of time and detect anomalies. The database server consists of many threads that perform different actions. I run a query to determine the ...
0
votes
0answers
135 views

Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment

I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark). Question Need a ...
0
votes
2answers
32 views

How to restrict the values of predicted variable to be positive?

I am using Python Linear Regression to predict the weekly orders for a food deliver company. But some of my orders are coming out as negative. Is there any way to restrict the predicted values to be ...