Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
1
vote
0answers
13 views

k-means in pandas dataframe, optimal k (number of clusters)? [closed]

I have a dataframe like this. At first, i would like to calculate the optimal number of clusters and then I would like to use k-means without the last column (id). After that i would like to know ...
-1
votes
2answers
31 views

Creating Pandas DataFrames from a dictionary containing various CSV files

I want to create 3 dataframes (i.e., gs, test and train) from the following dictionary: ...
0
votes
1answer
45 views

Renaming Column Elements of DataFrames

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
-1
votes
0answers
22 views

Renaming Column Elements [closed]

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
3
votes
0answers
30 views

Sources to learn Anomaly Detection

Project Goal- Create an Anomaly Detection tool. As the first step I worked on some python code that would parse my XML files into a Dataframe in csv file. The next step is to go into machine ...
3
votes
1answer
30 views

How to properly store a list of pandas dataframes, so that I can import them in a new python script?

I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
3
votes
4answers
260 views

Clustering algorithm which does not require to tell the number of clusters

I have a dataframe with 2 columns of numerical values. I want to apply a clustering algorithm to put all the entries into the same group, which have a relatively small distance to the other entries. ...
1
vote
0answers
34 views

passing a dataframe as a argument while applying a lambda function on a dataframe [closed]

I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
1
vote
0answers
22 views

Training on small, sparse vectors included in one big, denormalized table/dataframe?

So I have a table like this: ...
0
votes
1answer
19 views

Python - Appending a new Dataframe column that is a function of two separate numerical columns

I have a dataset that gives me the geographic coordinates of residential properties. My goal is to append a new column to this dataframe that shows the properties' distances to various prominent ...
1
vote
1answer
23 views

Help getting corresponding dataframe values

I have two dataframes: ...
0
votes
1answer
45 views

Problem with sort by and group by in pandas

I have a DataFrame with duplicate id's but have different dates. For Example: ...
0
votes
1answer
28 views

How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]

I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are: A <...
0
votes
2answers
72 views

Create a new column in Pandas Dataframe based on the 'NaN' values in another column [closed]

What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large) ...
2
votes
2answers
29 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
1
vote
2answers
237 views

Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode

Hopefully, you don't mind me posting a question here instead of the regular stack exchange forum. I'm learning ML basics and practising pandas. assuming the data frame is called df, column name = ...
1
vote
3answers
131 views

creating a csv file from two csv files

I have two csv files: sub_compiler.to_csv('sub_compiler.csv') sub_compiler.head() and ...
0
votes
1answer
28 views

Merging two pandas dataframes [duplicate]

I have two csv files,and if I do: df_compiler = pd.read_csv('sub_compiler.csv') I obtain: and for the other csv file I do: ...
1
vote
1answer
25 views

Changing the value of a column based on the condition on another column

I have a table whose header looks like this: complaint_type borough street_name incident_zip latitude longitude I want to check if the "incident_zip" column of each row is in a specific ...
2
votes
1answer
37 views

Get elements from lists in pandas dataframe [closed]

I have the following column of a data frame: I get it by doing dataFrame['opcodes]. and I would like to consider only the first 20 and the last 20 elements of ...
0
votes
2answers
52 views

use .apply() function to change values to a column of the dataframe

I have a dataframe which is the following: and I would like to consider only the column of instructions and keep just the values push, test, mov, test ,....., so just the first word of each string ...
-1
votes
2answers
499 views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
0
votes
0answers
28 views

Convert column of float valus to time format using panda

I have a data frame with a column of float data (e.g. 1.15 or 12.50).The first number represent hour and the digit represent minutes. I want to format it in order to have 4 numbers by adding 0 to ...
-1
votes
1answer
293 views
2
votes
0answers
30 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
0
votes
0answers
79 views

Empty Dataframe, Columns [], Index [], not sure how to resolve

The goal for this application is to read stock tickers from excel to python, The information reads and prints great, however when I run the program that reads this 'Tickers' data frame, I obtain this: ...
0
votes
1answer
50 views

Join the List and the Data frame for matching list elements in Python

I have a list =[a,b] and Dataframe as below If the text in the 'Text column' matches with List, then need to add additional column matchedItem So output should be Please suggest on this..How ...
0
votes
1answer
21 views

generate normally distributed dataframe with 3 columns

I am trying to generate a normally distributed data frame in python with 3 columns. Each columns requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate ...
-1
votes
1answer
343 views

Finding number of common elements between different columns of a DataFrame

I have a dataframe which has almost 70-80 columns. Each column consists of 100-150 rows in which values are stored as strings. I would like to find, for each column, what is the number of common ...
0
votes
3answers
86 views

Compare dataframes columns

...
1
vote
3answers
170 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
1answer
28 views

how to create multiple plot from a panda Dataframe

I want to plot multiple plots. The data is stored in a pandas dataframe and each row should be a seperate plot. Each row has an ID (ZRD_ID) which doenst matter and a date (TAG) and 24 values to be ...
0
votes
2answers
918 views

Delete/Drop only the rows which has all values as NaN in pandas [closed]

I have a Dataframe, i need to drop the rows which has all the values as NaN. ID Age Gender 601 21 M 501 NaN F NaN NaN NaN The ...
1
vote
0answers
36 views

visualize images inside the Dataframe [closed]

I use python 3.7.3 and I run this code on Jupyter Notebook . I have a column named 'image' in my Dataframe named df that contains Url of images I wanna visualize inside the df. I saw some answers ...
0
votes
1answer
4k views

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

Firstly I have a pandas series of recommended product (recmd_prdt_list). In this series there is a possibility of presence of deleted products. So as to remove deleted products from the recommended ...
0
votes
1answer
145 views

How to change a Pandas dataframe into feature vector?

I have a Pandas dataframe with 10 columns, 9 of which are features to be used to predict the 10th column. How is it ossible to convert this Pandas dataframe into X and y vectors to use in a linear ...
3
votes
1answer
360 views

How to shuffle only a fraction of a column in a Pandas dataframe?

I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe. How would you do it? Is there a simple idiomatic way to do that, maybe using ...
2
votes
1answer
66 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
0
votes
1answer
62 views

Make column as dictionary key and row as value in pandas dataframe

I have Dataframe as below. name value id A 123 1 B 345 5 C 567 4 D 789 2 I need to create a ...
1
vote
1answer
32 views

very important features, but rarely input

The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would ...
0
votes
1answer
28 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
0
votes
0answers
50 views

sorting pyspark dataframe

I'm sorting a pyspark DataFrame in databricks by df.sort(desc('x')). However, when I look at the DataFrame by display(df), I don't see the DataFrame being sorted in 'x'. What could be the reason?
0
votes
0answers
34 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
0
votes
1answer
43 views

How can I arrange data in different columns (for different constituents) from one?

I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want ...
1
vote
1answer
77 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
0
votes
2answers
29 views

Find columns with numeric values, but stored as string

I need to find the columns in data frame, which has numeric values and are stored as string. ...
0
votes
1answer
100 views

Why we add a constant value column in our DataFrame sometimes?

Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1. I need to ...
1
vote
1answer
524 views

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
1
vote
1answer
44 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...