Questions tagged [csv]

Comma-Separated Values are a list of plain text values delimited by commas, or a file containing one or more lists in that format.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

NaNs not being ignored despite program being told to do so; Linear Regression

I have gone through the process of filtering a set of data that I have by removing excess numbers and values that are invalid. However, there is a lot of data that occasionally has ' ' in it, and ...
1
vote
1answer
17 views

Defining a function in Pandas [closed]

Hello Data Scientists, Disclaimer: I am a Beginner learning Python and at the same time just trying out a lot of things. I am doing some data extraction on a file with with weekly deaths in ...
1
vote
2answers
1k views

Getting some data from a csv file into a list of tuples using Python

I have a csv file of almost 2 million rows and about 20 columns. I am interested in about 40000 of these rows, and in 5 of the columns 'squat', 'bench', 'deadlift', 'bodyweight', 'total'. I'd like to ...
1
vote
1answer
747 views

How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names. what should ...
1
vote
1answer
174 views

Extract or subset hundreds of columns from a data frame

I need to extract many columns from a dataset. I have a very large csv file with thousands of columns and rows, and I read it into R using: ...
2
votes
2answers
917 views

Pandas loads a csv file incorrectly, but without throwing an error [closed]

So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas load the data improperly. And it does so consistently. Let me show you ...
0
votes
0answers
13 views

Issue while reading a csv file through Spark

I am trying to read a csv file through Spark. However one of the columns has the data in the below format and because of comma it is being split into multiple columns. The input csv file is a comma ...
2
votes
1answer
717 views

Whole time column in csv file convert into UTC (epoch) using python

I have a dataset with time and columns. I want to plot a graph with time and value. I tried many methods but didn't come proper graph. Because I have a time series. Then I thought I will convert time ...
1
vote
3answers
103 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
0
votes
0answers
16 views

Save CSVs as hyper file

I have three separate CSV files (Points.csv, PointsSynonym.csv, and LocalDataPoints.csv). I made these files after extracting data from shape files and .gpkg files. I would like to save them all a ...
0
votes
3answers
971 views

Convert .data file to .csv

I'm using a data called 'adults.data', I need to work with that data as a '.csv' file. Can anyone guide me on how can I change the format? I tried opening the file in excel and then save it as csv, ...
1
vote
0answers
16 views

Prediction of rain using DBSCAN and Spectral Clustering [closed]

I wanted to do a prediction model of rainfall after clustering of data. I don't know how to proceed.Is it possible? May be using csv data.
2
votes
1answer
30 views

How to work with hundreds of CSVs with millions of rows in each?

So I'm doing a project on the COVID-19 Tweets dataset from the IEEE port and I plan to analyse the tweets over the time period from March 2020 till date. The thing is there's more than 300 CSVs for ...
2
votes
1answer
6k views

File converter: from CSV to HDF5

Can anyone recommend any command line tool for converting large CSV file into HDF5 format?
2
votes
0answers
20 views

Is there a corpus of toy datasets specifically designed for finding bugs in data science software?

I'm looking for a corpus of toy tabular datasets that can be used to test data profiling, machine learning, data manipulation, etc. software. Some example attributes: Strange column names (empty ...
0
votes
1answer
250 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
2
votes
0answers
12 views

Is high school GPA included in IPEDs NCES databases?

I’m currently developing a computer tool which will sort through the US college and university NCES data provided in the IPEDs website to match users with potential universities. NCES shows university ...
1
vote
2answers
3k views

How to create a parquet file from a query to a mysql table

Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic: ...
0
votes
1answer
84 views

How do you see the element of a csv table with many columns (>30) which the names of its columns is more than 10 character in pandas?

How do you see in pandas the element of a csv table with many columns (>25) which the names of its columns is more than 10 character? I have 5000 rows and 32 columns and the label of some columns ...
2
votes
1answer
210 views

alternatives to pyspark to query a csv file locally (no distributed computation needed)

I am reading a csv file with pyspark to extract some information from it. I am running pyspark locally and I do not need distributed computation. On the contrary I need some plotting library such as ...
6
votes
2answers
20k views

Merging large CSV files in pandas

I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
2
votes
1answer
16k views

how to transform a folder of images into csv file?

I have a data set for segmentation project. The dataset have 2 folders containing train folder and annotated_train_data folder, both folders have images. input and label both are images. I want to ...
11
votes
2answers
58k views

How to store strings in CSV with new line characters?

My question is: what are ways I can store strings in a CSV that contain newline characters (i.e. \n), where each data point is in one line? Sample data This is a ...
1
vote
1answer
528 views

R: reading big data files in R

In R, there are libraries which help speed up the process of reading huge data files. Examples of these libraries include sqldf and ff. What would be the disadvantage of using these packages to read ...
0
votes
1answer
32 views

How to filter in pandas based on string value? [closed]

I want to filter a pandas dataframe based on the id of the row. The id is a sequence of patient names. For example, I have columns in this format: ID | A | B And ...
0
votes
1answer
4k views
3
votes
1answer
6k views

Pandas throwing "Error tokenizing data. C error" while loading data sets from URL

I am trying to work on the Titanic competition to get hands on experience with data science & machine learning. I tried to load up the datasets from GitHub but pandas threw the following error: <...
6
votes
3answers
235 views

How to get K most different rows in csv?

We have boring CSV with 10000 rows of ages (float), titles (enum/int), scores (float). How to select 1000 most different rows? I look for a general solution that ...
2
votes
1answer
42 views

Recommendations for Data Visualization solution for large CSV file on a non-profit website

I would like your advice and recommendations for a non-profit that is trying to turn a large CSV file into meaningful information. I am part of a non-profit that would like to turn a 23,000 row x 23 ...
0
votes
1answer
41 views

Scrapping data from a CSV file [closed]

So, I'm a rookie at web scrapping and Python in general, and I'm having quite a hard time learning it. I'm trying to get some data from a csv file, but so far I could only split lines and columns. I ...
0
votes
4answers
90 views

How to view entire 5788 rows and 7 columns.?

I want to see entire 5788 rows and 7 columns of my CSV data. I used below code and it is giving little data by stating [5788 rows x 7 columns] but showing only 10 rows and 5 columns in python prompt. ...
0
votes
1answer
38 views

CSVs: how to output missing data to make processing easier?

If supplying data (that can be either string or numeric) via CSVs, what's a good strategy for marking that a value is missing? Some non-empty sentinel, like NA ...
1
vote
1answer
274 views

How do I change the location of Google BigQuery

So, I'm trying to use Google BigQuery for the first time for a project of mine, and I'm a bit confused. The documentation isn't helping much, and it looks like all the Google employees are gone thanks ...
1
vote
2answers
45 views

I am working on a pose dataset and want to convert this data into CSV format. How to convert this data into a single row of data

This is the data for one pose. I have a lot of data like this which i want to convert to CSV to apply machine learning algorithm. I am assuming I have to convert this data into a single row of comma ...
1
vote
0answers
344 views

Dask error when reading data from a large zip file

I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this: ...
1
vote
1answer
54 views

Representing 2 dimensional data for facial recognition

My goal is to train a neural network to recognize faces based on a list of landmark points generated by Google Firebase ML-Kit. Since I just started familiarizing myself with ML, I only want my model ...
0
votes
1answer
201 views

How to split a CSV into a dataframe without newline

Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same ...
8
votes
2answers
120k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
4
votes
1answer
1k views

Any one read the over 100gb csv file and successfully concatenation?

I have been searching for the deal with large CSV file read method Its over 100gb and need to know how deal with the chunk file processing and make concatenation faster ...
1
vote
0answers
15 views

Built strong base for Unsupervised Learning [closed]

I’m am new into machine learning, recently I have put a task upon my shoulders to Detect Outliers in Dataset. The anomaly detection should be done using Unsupervised learning and preferably use ...
1
vote
2answers
594 views

Should I use MongoDB instead of storing data in CSV in python?

I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
1
vote
1answer
21k views

How to save API data into CSV format?

Is there an easy way to convert API data with python commands and output the results in CSV format? Here's what I'm doing now: ...
0
votes
1answer
40 views

Deep learning model gives random results

First I am new to machine learning if it is an obvious question, I am sorry. ...
8
votes
5answers
247k views

I got the following error : 'DataFrame' object has no attribute 'data'

I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result: Program: ...
0
votes
1answer
362 views

How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]

I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are: A <...
0
votes
1answer
10k views

np.loadtxt function showing error, could not convert string to float: '"Date"'

This is my stock market csv data: ...
0
votes
1answer
4k views
2
votes
2answers
93 views

File format where column names are repeated on each row

I have received a dataset in text file with the following format ...
2
votes
1answer
5k views

Getting stock data in a discipline manner from Yahoo finance

I used the below code for downloading stock data from yahoo finance:- ...