Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
0answers
24 views

Renaming Column Elements of DataFrames

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
-1
votes
0answers
22 views

Renaming Column Elements [closed]

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
3
votes
0answers
28 views

Sources to learn Anomaly Detection

Project Goal- Create an Anomaly Detection tool. As the first step I worked on some python code that would parse my XML files into a Dataframe in csv file. The next step is to go into machine ...
3
votes
1answer
28 views

How to properly store a list of pandas dataframes, so that I can import them in a new python script?

I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
3
votes
4answers
260 views

Clustering algorithm which does not require to tell the number of clusters

I have a dataframe with 2 columns of numerical values. I want to apply a clustering algorithm to put all the entries into the same group, which have a relatively small distance to the other entries. ...
1
vote
0answers
34 views

passing a dataframe as a argument while applying a lambda function on a dataframe [closed]

I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
1
vote
0answers
22 views

Training on small, sparse vectors included in one big, denormalized table/dataframe?

So I have a table like this: ...
0
votes
1answer
18 views

Python - Appending a new Dataframe column that is a function of two separate numerical columns

I have a dataset that gives me the geographic coordinates of residential properties. My goal is to append a new column to this dataframe that shows the properties' distances to various prominent ...
1
vote
1answer
23 views

Help getting corresponding dataframe values

I have two dataframes: ...
0
votes
1answer
43 views

Problem with sort by and group by in pandas

I have a DataFrame with duplicate id's but have different dates. For Example: ...
0
votes
1answer
28 views

How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]

I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are: A <...
0
votes
2answers
67 views

Create a new column in Pandas Dataframe based on the 'NaN' values in another column [closed]

What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large) ...
2
votes
2answers
29 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
1
vote
2answers
212 views

Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode

Hopefully, you don't mind me posting a question here instead of the regular stack exchange forum. I'm learning ML basics and practising pandas. assuming the data frame is called df, column name = ...
1
vote
3answers
128 views

creating a csv file from two csv files

I have two csv files: sub_compiler.to_csv('sub_compiler.csv') sub_compiler.head() and ...
0
votes
1answer
28 views

Merging two pandas dataframes [duplicate]

I have two csv files,and if I do: df_compiler = pd.read_csv('sub_compiler.csv') I obtain: and for the other csv file I do: ...
1
vote
1answer
24 views

Changing the value of a column based on the condition on another column

I have a table whose header looks like this: complaint_type borough street_name incident_zip latitude longitude I want to check if the "incident_zip" column of each row is in a specific ...
2
votes
1answer
35 views

Get elements from lists in pandas dataframe [closed]

I have the following column of a data frame: I get it by doing dataFrame['opcodes]. and I would like to consider only the first 20 and the last 20 elements of ...
0
votes
2answers
50 views

use .apply() function to change values to a column of the dataframe

I have a dataframe which is the following: and I would like to consider only the column of instructions and keep just the values push, test, mov, test ,....., so just the first word of each string ...
-1
votes
2answers
454 views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
0
votes
0answers
28 views

Convert column of float valus to time format using panda

I have a data frame with a column of float data (e.g. 1.15 or 12.50).The first number represent hour and the digit represent minutes. I want to format it in order to have 4 numbers by adding 0 to ...
-1
votes
1answer
249 views
2
votes
0answers
30 views

Need help on pandas data frames filtering on a csv file

Raw data gets generated daily We need to get success rate for current month on a daily basis & success rate of each cell by end of that month from Jan till current month. Raw data has following ...
0
votes
0answers
69 views

Empty Dataframe, Columns [], Index [], not sure how to resolve

The goal for this application is to read stock tickers from excel to python, The information reads and prints great, however when I run the program that reads this 'Tickers' data frame, I obtain this: ...
0
votes
1answer
46 views

Join the List and the Data frame for matching list elements in Python

I have a list =[a,b] and Dataframe as below If the text in the 'Text column' matches with List, then need to add additional column matchedItem So output should be Please suggest on this..How ...
0
votes
1answer
21 views

generate normally distributed dataframe with 3 columns

I am trying to generate a normally distributed data frame in python with 3 columns. Each columns requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate ...
-1
votes
1answer
319 views

Finding number of common elements between different columns of a DataFrame

I have a dataframe which has almost 70-80 columns. Each column consists of 100-150 rows in which values are stored as strings. I would like to find, for each column, what is the number of common ...
0
votes
3answers
83 views

Compare dataframes columns

...
1
vote
3answers
152 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
1answer
28 views

how to create multiple plot from a panda Dataframe

I want to plot multiple plots. The data is stored in a pandas dataframe and each row should be a seperate plot. Each row has an ID (ZRD_ID) which doenst matter and a date (TAG) and 24 values to be ...
0
votes
2answers
895 views

Delete/Drop only the rows which has all values as NaN in pandas [closed]

I have a Dataframe, i need to drop the rows which has all the values as NaN. ID Age Gender 601 21 M 501 NaN F NaN NaN NaN The ...
1
vote
0answers
36 views

visualize images inside the Dataframe [closed]

I use python 3.7.3 and I run this code on Jupyter Notebook . I have a column named 'image' in my Dataframe named df that contains Url of images I wanna visualize inside the df. I saw some answers ...
0
votes
1answer
4k views

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

Firstly I have a pandas series of recommended product (recmd_prdt_list). In this series there is a possibility of presence of deleted products. So as to remove deleted products from the recommended ...
0
votes
1answer
141 views

How to change a Pandas dataframe into feature vector?

I have a Pandas dataframe with 10 columns, 9 of which are features to be used to predict the 10th column. How is it ossible to convert this Pandas dataframe into X and y vectors to use in a linear ...
3
votes
1answer
341 views

How to shuffle only a fraction of a column in a Pandas dataframe?

I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe. How would you do it? Is there a simple idiomatic way to do that, maybe using ...
1
vote
1answer
58 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
0
votes
1answer
58 views

Make column as dictionary key and row as value in pandas dataframe

I have Dataframe as below. name value id A 123 1 B 345 5 C 567 4 D 789 2 I need to create a ...
1
vote
1answer
32 views

very important features, but rarely input

The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would ...
0
votes
1answer
28 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
0
votes
0answers
49 views

sorting pyspark dataframe

I'm sorting a pyspark DataFrame in databricks by df.sort(desc('x')). However, when I look at the DataFrame by display(df), I don't see the DataFrame being sorted in 'x'. What could be the reason?
0
votes
0answers
34 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
0
votes
1answer
43 views

How can I arrange data in different columns (for different constituents) from one?

I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want ...
1
vote
1answer
74 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
0
votes
2answers
29 views

Find columns with numeric values, but stored as string

I need to find the columns in data frame, which has numeric values and are stored as string. ...
0
votes
1answer
98 views

Why we add a constant value column in our DataFrame sometimes?

Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1. I need to ...
1
vote
1answer
499 views

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
1
vote
1answer
43 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
0
votes
0answers
19 views

Dataframe header colouring doesn't work

I would like to construct dataframe whos header has specified colour. I tried the following code but for some unknown reason it doesn't work. Any recommendations to solve this task? ...
0
votes
1answer
764 views

Creating new column in dataframe based on conditions in 2 other columns [closed]

I would like to create a new column in my dataframe based on values from both the gender and experimental_grouping columns. As I have it written below, the column ...
0
votes
1answer
658 views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...