Questions tagged [csv]

Comma-Separated Values are a list of plain text values delimited by commas, or a file containing one or more lists in that format.

Filter by
Sorted by
Tagged with
13
votes
2answers
61k views

How to store strings in CSV with new line characters?

My question is: what are ways I can store strings in a CSV that contain newline characters (i.e. \n), where each data point is in one line? Sample data This is a ...
8
votes
2answers
121k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
7
votes
5answers
253k views

I got the following error : 'DataFrame' object has no attribute 'data'

I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result: Program: ...
6
votes
3answers
17k views

Reading CSVs with new lines in fields with Spark

I was trying to load the below weblogic domain log (application error log) into Spark dataframe. I created a RDD and converted the RDD into dataframe. I was able to load the data successfully for the ...
6
votes
2answers
20k views

Merging large CSV files in pandas

I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
6
votes
3answers
235 views

How to get K most different rows in csv?

We have boring CSV with 10000 rows of ages (float), titles (enum/int), scores (float). How to select 1000 most different rows? I look for a general solution that ...
6
votes
3answers
1k views

Summarize and visualize a CSV in Java/Scala?

I would like to summarize (as in R) the contents of a CSV (possibly after loading it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and ...
5
votes
2answers
355 views

What kind of statistical analyses can I do with my data?

I'm trying to analyze human intentions in clicking google ad word keywords. In this dataset I have the usual adword details, for example ...
4
votes
2answers
1k views

Recommendations for storing time series data

As part of my thesis I've done some experiments that have resulted in a reasonable amount of time-series data (motion-capture + eye movements). I have a way of storing and organizing all of this data, ...
4
votes
1answer
1k views

Any one read the over 100gb csv file and successfully concatenation?

I have been searching for the deal with large CSV file read method Its over 100gb and need to know how deal with the chunk file processing and make concatenation faster ...
3
votes
5answers
2k views

Quick way to visually explore data?

I have a very large csv file which Apple Numbers won't open. I can open it with TextEdit, but each row is so long it forms multiple lines in the document and makes the document difficult to understand....
3
votes
2answers
2k views

How to deal with large data sets

So i'm very new to this, forgive my silly questions. I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job. I have .csv files containing data ...
3
votes
3answers
17k views

How to read file from user in Shiny and assign it to a variable in global.r?

I want to read a csv file as input from user in Shiny and assign it to a variable in global.r file.The code I have in ui.R is ...
3
votes
2answers
8k views

How to transform a folder of images in a csv file

I have a folder with a lot of images that I want to use to bild a classificator using a SVM model in python with sklearn. I've always used csv file as train/test set with sklearn, how can I make it? (...
3
votes
3answers
5k views

Merging repeating data cells in csv

I have a CSV file with around 1 Million rows. Let say its have details like ...
3
votes
1answer
6k views

Pandas throwing "Error tokenizing data. C error" while loading data sets from URL

I am trying to work on the Titanic competition to get hands on experience with data science & machine learning. I tried to load up the datasets from GitHub but pandas threw the following error: <...
3
votes
1answer
1k views

ML algorithm for determining CSV file header names based on content

I have a large amount of CSV files, an example of which (for job titles) is listed below. The data is noisy (there are misspellings, difference in capitalisation, missing values, and they are not well-...
3
votes
1answer
246 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
2
votes
2answers
96 views

File format where column names are repeated on each row

I have received a dataset in text file with the following format ...
2
votes
2answers
630 views

Breaking down a column in Pandas into a separate CSV for display in Tableau

My data is coming from a CSV, which should be visualized in Tableau. However, the data contains the column category_list, which consists of values separated by a ...
2
votes
2answers
1k views

Pandas loads a csv file incorrectly, but without throwing an error [closed]

So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas load the data improperly. And it does so consistently. Let me show you ...
2
votes
1answer
5k views

Getting stock data in a discipline manner from Yahoo finance

I used the below code for downloading stock data from yahoo finance:- ...
2
votes
1answer
31 views

How to work with hundreds of CSVs with millions of rows in each?

So I'm doing a project on the COVID-19 Tweets dataset from the IEEE port and I plan to analyse the tweets over the time period from March 2020 till date. The thing is there's more than 300 CSVs for ...
2
votes
1answer
16k views

how to transform a folder of images into csv file?

I have a data set for segmentation project. The dataset have 2 folders containing train folder and annotated_train_data folder, both folders have images. input and label both are images. I want to ...
2
votes
1answer
212 views

alternatives to pyspark to query a csv file locally (no distributed computation needed)

I am reading a csv file with pyspark to extract some information from it. I am running pyspark locally and I do not need distributed computation. On the contrary I need some plotting library such as ...
2
votes
1answer
112 views

Creating a single number from a numpy array - Python [closed]

I am working on a gender classification project. I am extracting the pixels of an image using a Numpy array in Python, similar to the one below: ...
2
votes
1answer
44 views

Recommendations for Data Visualization solution for large CSV file on a non-profit website

I would like your advice and recommendations for a non-profit that is trying to turn a large CSV file into meaningful information. I am part of a non-profit that would like to turn a 23,000 row x 23 ...
2
votes
1answer
6k views

File converter: from CSV to HDF5

Can anyone recommend any command line tool for converting large CSV file into HDF5 format?
2
votes
1answer
3k views

day is out of range for month import from csv file in python

Here I have one csv file with different values and date_time. I wrote the code and run it. Then it gave me an error time data '8/6/2018 6:45' does not match format ' %d/%m/%Y %H:%M:%S' I want to plot ...
2
votes
0answers
21 views

Is there a corpus of toy datasets specifically designed for finding bugs in data science software?

I'm looking for a corpus of toy tabular datasets that can be used to test data profiling, machine learning, data manipulation, etc. software. Some example attributes: Strange column names (empty ...
2
votes
0answers
13 views

Is high school GPA included in IPEDs NCES databases?

I’m currently developing a computer tool which will sort through the US college and university NCES data provided in the IPEDs website to match users with potential universities. NCES shows university ...
2
votes
1answer
740 views

Whole time column in csv file convert into UTC (epoch) using python

I have a dataset with time and columns. I want to plot a graph with time and value. I tried many methods but didn't come proper graph. Because I have a time series. Then I thought I will convert time ...
1
vote
3answers
7k views

importing csv data in python

I have a csv file with around 130 columns and 6000 rows what is the best way to import them into python, so that I can later use them in a classification algorithm(columns are the labels and rows are ...
1
vote
3answers
893 views

What is a good way to store processed CSV data to train model in Python?

I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of ...
1
vote
2answers
12k views

Python - AttributeError: type object 'DataFrame' has no attribute 'read_csv'

I'm trying to create some charts using Python. I've this dataset in a CSV file: Banana Water Rice Rice Water Bread Banana Juice And I've this code: <...
1
vote
3answers
2k views

Creating data model out of .csv file using Python

I want to create a data model out of a .csv file using Python. I mean to create dependencies, for example the primary key and stuff such that I can check if the new ...
1
vote
2answers
112 views

Any tool to load large csv files quickly on windows 7

I am working with relatively large CSV files (1.5 MB+) and while testing the code, I have to frequently open up the CSV files to read a particular row, validate the data are there and then close the ...
1
vote
1answer
22k views

How to save API data into CSV format?

Is there an easy way to convert API data with python commands and output the results in CSV format? Here's what I'm doing now: ...
1
vote
2answers
3k views

how to take CSV file input in list of tuples

I have a .txt(data.txt) file containing csv data like: X Class 15.0001 Yes 18.00 NO 17.07 Yes I need to make a function to return a list of ...
1
vote
1answer
12k views

Orange: Data has no target variable error

I am trying to apply Random Forest algorithm on a data set using Orange. The target variable is not set in the data set. However, I know which column is the target variable. How can I specify the ...
1
vote
2answers
47 views

I am working on a pose dataset and want to convert this data into CSV format. How to convert this data into a single row of data

This is the data for one pose. I have a lot of data like this which i want to convert to CSV to apply machine learning algorithm. I am assuming I have to convert this data into a single row of comma ...
1
vote
2answers
625 views

Should I use MongoDB instead of storing data in CSV in python?

I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
1
vote
2answers
43 views

Removing # from header

I have a $(418,2)$ matrix and I want to convert it to csv. so I write: np.savetxt('titanic1.csv', Sol, fmt='%.2f', delimiter=",",header="PassengerId,Survived") ...
1
vote
1answer
378 views

Data scraping & NLP?

I'm scraping data from Bing search results for (non-commercial purposes, of course) on Python using BeautifulSoup. I've entered an Indian dessert name, called 'rasmalai' as the word that I am focusing ...
1
vote
2answers
135 views

Reading a wide dataset in R

I originally had a wide CSV dataset of about 18000 columns(and about 80 rows) that I am trying to read in R. It was stored in an Excel sheet,which unfortunately has a limit of only 16384 columns. ...
1
vote
1answer
755 views

How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names. what should ...
1
vote
1answer
331 views

How do I change the location of Google BigQuery

So, I'm trying to use Google BigQuery for the first time for a project of mine, and I'm a bit confused. The documentation isn't helping much, and it looks like all the Google employees are gone thanks ...
1
vote
1answer
54 views

Representing 2 dimensional data for facial recognition

My goal is to train a neural network to recognize faces based on a list of landmark points generated by Google Firebase ML-Kit. Since I just started familiarizing myself with ML, I only want my model ...
1
vote
2answers
3k views

How to create a parquet file from a query to a mysql table

Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic: ...
1
vote
1answer
530 views

R: reading big data files in R

In R, there are libraries which help speed up the process of reading huge data files. Examples of these libraries include sqldf and ff. What would be the disadvantage of using these packages to read ...