Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
1
vote
0answers
52 views
2
votes
2answers
84 views

How to deal with millions or rows of data for analysis/visualization purpose [closed]

I have data in 2 tables in Sql server. First table has around 10 million rows and 8 columns. Second table has 6 million rows and 60 columns. I want to import those tables into a Python notebook ...
2
votes
1answer
197 views

Change values of a particular column to value_count()

Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a ...
2
votes
1answer
38 views

Finding Outliers in Resource Forecast Data

Unsure if this is the correct place to place this, please close if so. I'm a workforce analyst at a large retail company, I own and maintain all the forecasting for our retail stores. This is based ...
0
votes
2answers
691 views

Removing duplicates and keeping the last entry in pandas

I have a table and I'm trying to remove all the duplicate and keep the the rows that has the latest datestamp. Here is the table: ...
0
votes
1answer
94 views

Extracting email id gives error

I am extracting email ids and storing them into a new column variable, but I am getting the issue: ...
0
votes
2answers
951 views

What does pandas describe() percentiles values tell about our data?

Let say this is my dataframe ...
1
vote
0answers
234 views

How to create a heatmap inside a polar chart [closed]

I would like to build something like this. Creating a polar chart isn´t an issue, but i have no idea how to implement the round areas and the color gradients into the plot.
1
vote
0answers
33 views

Email clustering using numpy and pandas

Im trying to cluster my raw email data using numpy and pandas only. Hence Im Trying to create an email dataframe from raw emails that I have. However the issue that I have is that I keep getting an ...
0
votes
0answers
234 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
1answer
502 views

What is Difference Between np.zeros() and np.empty() [duplicate]

Can Anyone Please Explain me the Difference Between them In Terms Of Any Operation or Computation and Values Stored in them
0
votes
1answer
77 views

Script to convert non-numeric elements that are numbers to numeric

I'm self-teaching myself some scikit-learn and tensorflow right now, and trying to get better at cleaning data since the rest seems reasonably straightforward. I downloaded some Google Play Store ...
1
vote
0answers
136 views

linear interpolation on python

Given 1000 objects, the data set tracks each object's location ordered by time, such as 3/3/18 9:01, 3/3/18 9:04.......how to define the first observation of each object together with xy-...
2
votes
2answers
5k views
1
vote
1answer
46 views

What is the difference between these 2 Python Pandas queries

I am trying to make sense out of some data, here is the Pandas data frame: ...
1
vote
1answer
135 views

Scatter Plot with Conditions

Suppose I have a data frame name = ['A', 'B', 'C'] score = [2,4,6] I want to create a scatter plot with the following conditions, color the bubble as green if ...
0
votes
0answers
5 views

I wanted to get the priority based on some column

I wanted to give priority based on values like in Col_2 there are 2 values A,B i Need A to be given more priority as compared to B incase if it has 2 values as per col_1 Col_1 Col_2 new_col 1 ...
2
votes
1answer
69 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
1
vote
1answer
1k views

Pandas throwing “Error tokenizing data. C error” while loading data sets from URL

I am trying to work on the Titanic competition to get hands on experience with data science & machine learning. I tried to load up the datasets from GitHub but pandas threw the following error: <...
0
votes
2answers
24 views

How to consume single piece of text for classification in a model?

Here's a high-level blueprint of my model - Input data:: ...
1
vote
1answer
23 views

Problem building dictionary from series

I have a pandas dataframe with a column CAS_BRM_IDA of type category (even if its values seem of foat type ... but they are not meaningful per se) I built the following serie from that dataframe : <...
2
votes
1answer
382 views

Pandas Groupby makes kernel die in Jupyter notebook/Python

I have a groupby in jupyter-notebook that takes ages to run and after 10 minutes of running it says 'kernel died...', The groupby looks like this: ...
-1
votes
2answers
42 views

How to calculate occurance of a name per hour basis of monthly csv data using python

I have csv file which have DateTime and application OS name. DateTime have for 1 month ,per hour basis. The csv looks as follows, ...
7
votes
4answers
816 views

How to replace NaN values for image data?

My data set has a total of 200 columns, where each column corresponds to the same pixel in all of my images. In total, I have 48,500 rows. The labels for the data range from 0-9. The data looks ...
0
votes
1answer
85 views

convert keywords in one column into several dummy columns

My dataframe has some free text fields named: {'title', 'description', 'location'} I prepared this text column by: concatenating all into a new column, dropping ...
0
votes
0answers
12 views

Complex merges between three data sources

I'm new to the field and I need some help in figuring out how to approach this problem. I have three data sets: Truck logs: Includes data about truck and trailer loading, arrival, and departure. ...
0
votes
0answers
24 views

Frequency of attributes (analysis)

If I have dataset with many features (attributes)and two class labels, how can I compute the frequency of one or all attributes with respect to class in python?
6
votes
3answers
124 views

What is the best algorithm/solution for predicting the following?

I have a dataset that comprises 76 countries, and 6 columns of distinct quantitative variables, which are the mean values of that variable relative to each country: If I were to take a random sample ...
5
votes
1answer
62 views

Text extraction from large pool of documents of different formats

I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, ...
1
vote
0answers
286 views

“Memory Error” - Kmeans in python using pandas DataFrame

I am trying to predict on my "dataset_to_predict" having size of (297000 x 5120). While Memory usage is under 50%. No Specific Error message. I'm trying to find # of k using elbow method - Got ...
0
votes
1answer
22 views

How to load pre-processed .txt data and use it for training convolutional networks?

I have a lot of '.txt' (pre-processed) files related to C3D and I need to use this to train and test while using CNN model. How does one go about this? Most of data in each .txt files are formatted ...
0
votes
1answer
40 views

Normalization(minmax) gives me worse results than before in KNN, follow up actions?

Hello I'm studying a classification problem with KNN right now. I have many numeric features that I normalized with MinMaxScaler, I also got some OHE categorical features that not seem to cause the ...
0
votes
1answer
74 views

Convert a repetitive list into Pandas Dataframe [closed]

How do you convert something like this: A: 1 B: 2 C: 3 ### A: 5 B: 5 C: 6 ### A: 2 B: 5 C: 7 into a dataset where the first row would be the first section with ...
1
vote
1answer
107 views

retrieve food groups from food item list

I have a dataframe of food items as follows: I have to create a food_group list that gives the food group it belongs to, for-example all type of yogurts should be in one group called yogurt. I used ...
0
votes
1answer
310 views

Remove columns with a certain number of consecutive zeros

I'm using pandas and I am dealing with time series of sales. What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series ...
0
votes
0answers
45 views

Add Header name without using pandas

I want to add header name without using pandas. In my program I have given two columns in my txt file where header name is missing: ...
2
votes
2answers
983 views

Combining Latitude/Longitude position into single feature

I have been playing with 2 dimensional machine learning using pandas (Trying to do something like this: https://github.com/freeman-lab/spark-ml-streaming), and I'd like to combine Lat/Long into a ...
0
votes
1answer
3k views

How to create dictionary with multiple keys from dataframe in python?

I have a pandas dataframe as follows, I want to convert it to a dictionary format with 2 keys as shown: ...
2
votes
1answer
86 views

Pandas dataframe with multiple hierarchical indices

I have a data frame which looks like this ...
2
votes
1answer
167 views

Delete all rows between two values in a dataframe that repeat multiple times in a column

I have dataframe and let's say inside of it is a column_A. This column_A has 3 strings as values, call them 'new_records', 'deletions', 'changes' that repeat across the dataframe multiple times in ...
3
votes
2answers
174 views

How to compare paired count data?

I am working with a machine learning approach that counts cars in images. I have a predicted dataset, which is the predicted output from the machine learning approach and a paired "true" dataset, ...
0
votes
2answers
48 views

Training and test split for time series analysis

I need help with below code: ...
1
vote
3answers
83 views

Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?

It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model ...
0
votes
1answer
323 views

Groupby date and find number of occurrences of a value a in another column using pandas

My Dataset is looking like this I would like to get the output something like this. I need to group by date and find the occurrences if each feedback. In this case, i have 5 feedback types (namely 5,...
0
votes
1answer
49 views

Segmenting data in a Dataframe and assigning order numbers (Python using Pandas)

I have a large Excel data set which I load as Pandas. In one column I have repeating strings/numbers (the information can be a text and or integers), in another column I have a unique identifier of ...
0
votes
2answers
114 views

Replace method in pandas not giving expected result

I have data as below: 123.12.23.2 110.22.21.23 I want to mask this data as below one 1xx.xx.xx.x So I tried below code : ...
0
votes
1answer
2k views

Pandas to_sql insert only some columns

I am new using pandas. I am asking about how to insert specific columns using to_sql. This is my explanation. I have a database with a table datasiswa with columns: id: as int, with autoincrement,...
0
votes
0answers
39 views

How to pass inputs (interactively) to a model?

Let me give you a high-level design (blueprint) of my model. Input data:: ...
0
votes
1answer
20 views

Why my frame index is full of 0 when I make it to copy another frame?

I've got such a code: ...
1
vote
0answers
25 views

How to concatenate many .psv files in google collaboratory?

I have a folder named 'training' in my local drive which has 20000 .psv files. I zipped it and uploaded to google collaboratory, with the upload option in the Files section. I unzipped it with the ...