Questions tagged [csv]
Comma-Separated Values are a list of plain text values delimited by commas, or a file containing one or more lists in that format.
94
questions
0
votes
0
answers
19
views
read_csv, non-printing ascii delimiters, and multi index
I have some text data which uses the ASCII data characters 0x1e "group separator" and 0x1d "record separator"...
- 101
0
votes
2
answers
34
views
How to Insert a Row before Dataframe in Pandas
I have a csv file that is used as a pandas dataframe, now I only need to insert a dummy row before the dataframe starts like in the screenshot denoted as "Label". How can I do that?
0
votes
0
answers
11
views
How to generate csv file for custom metrics in apache storm?
I have generated custom logs for getting CPU utilization per topology in Apache Storm with this document Profiling Resource Usage. But, CSV file for that log is not generated, like other metrics. I ...
- 1
0
votes
1
answer
14
views
Pandas altering numerical data on import
I have a csv file where a column of data is a 19 digit number of which the last three digits are always 163, this is specific to identify that this is the correct id. If i open the file in sublime, ...
- 111
0
votes
1
answer
148
views
How to fix ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
I am using pytest to test a function that reads a csv into a dictionary of lists. In the test function it writes a sample dictionary to a csv then calls the function to read it back out. To accomplish ...
2
votes
1
answer
77
views
The single CSV created by combining a large number of CSV files is too large to process. What options do I have?
The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
- 21
0
votes
0
answers
19
views
Has anyone joined FHIR resources?
I have FHIR-formatted data (Fast Healthcare Interoperability Resources), that I've converted to CSV using the npm package ndjson-to-csv, and on first glance, this seems useable for ML experimentation, ...
- 101
0
votes
1
answer
273
views
multiple images inside one large CSV file
I'm very new to data science, and was admiring how people had made these massive open-source datasets, on places like kaggle. I noticed that all of the datasets where all in CSV format. I have lots of ...
- 1
0
votes
1
answer
71
views
Online Generating Dendrograms with imported CSV file
Is there any website out there that performs hierarchical clustering on an imported file like CSV to generate dendrograms or similarity matrix? Free or paid does not matter.
So far I found heatmapper....
- 1
0
votes
1
answer
239
views
Error: `raise ValueError( ValueError: Missing column provided to 'parse_dates': 'Date' [closed]
I am using a .csv with two columns. The first has dates and the second has temperatures. I would like to plot it with dates on the x-axis and temperatures on the y-...
- 103
0
votes
0
answers
20
views
NaNs not being ignored despite program being told to do so; Linear Regression
I have gone through the process of filtering a set of data that I have by removing excess numbers and values that are invalid. However, there is a lot of data that occasionally has ' ' in it, and ...
1
vote
1
answer
20
views
Defining a function in Pandas [closed]
Hello Data Scientists,
Disclaimer: I am a Beginner learning Python and at the same time just trying out a lot of things.
I am doing some data extraction on a file with with weekly deaths in ...
0
votes
0
answers
25
views
Save CSVs as hyper file
I have three separate CSV files (Points.csv, PointsSynonym.csv, and LocalDataPoints.csv). I made these files after extracting data from shape files and .gpkg files.
I would like to save them all a ...
- 181
1
vote
3
answers
385
views
Efficiently modify a large csv file in Pandas
I have a csv file and would like to do the following modification on it:
...
- 11
-1
votes
3
answers
5k
views
Convert .data file to .csv [closed]
I'm using a data called 'adults.data', I need to work with that data as a '.csv' file. Can anyone guide me on how can I change the format?
I tried opening the file in excel and then save it as csv, ...
- 11
2
votes
1
answer
39
views
How to work with hundreds of CSVs with millions of rows in each?
So I'm doing a project on the COVID-19 Tweets dataset from the IEEE port and I plan to analyse the tweets over the time period from March 2020 till date. The thing is there's more than 300 CSVs for ...
- 21
2
votes
0
answers
27
views
Is there a corpus of toy datasets specifically designed for finding bugs in data science software?
I'm looking for a corpus of toy tabular datasets that can be used to test data profiling, machine learning, data manipulation, etc. software. Some example attributes:
Strange column names (empty ...
- 21
2
votes
0
answers
20
views
Is high school GPA included in IPEDs NCES databases?
I’m currently developing a computer tool which will sort through the US college and university NCES data provided in the IPEDs website to match users with potential universities. NCES shows university ...
- 21
0
votes
1
answer
455
views
How to specify different columns stacked vertically within CSV using pandas?
I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
- 11
1
vote
1
answer
832
views
How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?
I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names.
what should ...
- 11
2
votes
2
answers
2k
views
Pandas loads a csv file incorrectly, but without throwing an error [closed]
So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas load the data improperly. And it does so consistently. Let me show you ...
0
votes
1
answer
37
views
How to filter in pandas based on string value? [closed]
I want to filter a pandas dataframe based on the id of the row. The id is a sequence of patient names. For example, I have columns in this format:
ID | A | B
And ...
6
votes
3
answers
259
views
How to get K most different rows in csv?
We have boring CSV with 10000 rows of ages (float), titles (enum/int), scores (float). How to select 1000 most different rows? I look for a general solution that ...
- 161
2
votes
1
answer
72
views
Recommendations for Data Visualization solution for large CSV file on a non-profit website
I would like your advice and recommendations for a non-profit that is trying to turn a large CSV file into meaningful information.
I am part of a non-profit that would like to turn a 23,000 row x 23 ...
0
votes
1
answer
6k
views
0
votes
1
answer
56
views
Scrapping data from a CSV file [closed]
So, I'm a rookie at web scrapping and Python in general, and I'm having quite a hard time learning it. I'm trying to get some data from a csv file, but so far I could only split lines and columns. I ...
- 1
0
votes
1
answer
46
views
CSVs: how to output missing data to make processing easier?
If supplying data (that can be either string or numeric) via CSVs, what's a good strategy for marking that a value is missing?
Some non-empty sentinel, like NA
...
- 103
0
votes
4
answers
116
views
How to view entire 5788 rows and 7 columns.?
I want to see entire 5788 rows and 7 columns of my CSV data. I used below code and it is giving little data by stating [5788 rows x 7 columns] but showing only 10 rows and 5 columns in python prompt. ...
- 13
0
votes
2
answers
1k
views
Getting some data from a csv file into a list of tuples using Python
I have a csv file of almost 2 million rows and about 20 columns. I am interested in about 40000 of these rows, and in 5 of the columns 'squat', 'bench', 'deadlift', 'bodyweight', 'total'. I'd like to ...
- 101
1
vote
1
answer
662
views
How do I change the location of Google BigQuery
So, I'm trying to use Google BigQuery for the first time for a project of mine, and I'm a bit confused. The documentation isn't helping much, and it looks like all the Google employees are gone thanks ...
- 263
1
vote
0
answers
474
views
Dask error when reading data from a large zip file
I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this:
...
- 11
1
vote
1
answer
73
views
Representing 2 dimensional data for facial recognition
My goal is to train a neural network to recognize faces based on a list of landmark points generated by Google Firebase ML-Kit. Since I just started familiarizing myself with ML, I only want my model ...
- 111
1
vote
2
answers
72
views
I am working on a pose dataset and want to convert this data into CSV format. How to convert this data into a single row of data
This is the data for one pose. I have a lot of data like this which i want to convert to CSV to apply machine learning algorithm.
I am assuming I have to convert this data into a single row of comma ...
1
vote
0
answers
16
views
Built strong base for Unsupervised Learning [closed]
I’m am new into machine learning, recently I have put a task upon my shoulders to Detect Outliers in Dataset.
The anomaly detection should be done using Unsupervised learning and preferably use ...
- 585
0
votes
1
answer
221
views
How to split a CSV into a dataframe without newline
Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same ...
- 109
1
vote
2
answers
872
views
Should I use MongoDB instead of storing data in CSV in python?
I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
- 579
0
votes
1
answer
71
views
Deep learning model gives random results
First I am new to machine learning if it is an obvious question, I am sorry.
...
- 157
0
votes
1
answer
531
views
How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]
I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are:
A <...
- 103
1
vote
2
answers
28k
views
How to save API data into CSV format?
Is there an easy way to convert API data with python commands and output the results in CSV format?
Here's what I'm doing now:
...
- 2,317
0
votes
1
answer
91
views
How do you see the element of a csv table with many columns (>30) which the names of its columns is more than 10 character in pandas?
How do you see in pandas the element of a csv table with many columns (>25) which the names of its columns is more than 10 character?
I have 5000 rows and 32 columns and the label of some columns ...
- 1,636
1
vote
2
answers
4k
views
How to create a parquet file from a query to a mysql table
Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic:
...
0
votes
1
answer
5k
views
Pandas - how to make further sub-groups after groupby( ) and get_group( )
Sample data in .csv format
...
- 281
4
votes
1
answer
2k
views
Any one read the over 100gb csv file and successfully concatenation?
I have been searching for the deal with large CSV file read method
Its over 100gb and need to know how deal with the chunk file processing
and make concatenation faster
...
- 483
2
votes
2
answers
135
views
File format where column names are repeated on each row
I have received a dataset in text file with the following format
...
- 133
2
votes
1
answer
6k
views
Getting stock data in a discipline manner from Yahoo finance
I used the below code for downloading stock data from yahoo finance:-
...
- 129
0
votes
1
answer
13k
views
np.loadtxt function showing error, could not convert string to float: '"Date"'
This is my stock market csv data:
...
- 129
1
vote
2
answers
43
views
Removing # from header
I have a $(418,2)$ matrix and I want to convert it to csv. so I write:
np.savetxt('titanic1.csv', Sol, fmt='%.2f', delimiter=",",header="PassengerId,Survived")
...
3
votes
1
answer
8k
views
File converter: from CSV to HDF5
Can anyone recommend any command line tool for converting large CSV file into HDF5 format?
- 659
3
votes
1
answer
324
views
Processing csv file with more than 700K rows of data
I have a .csv file (around 400MB in size) which contains 700K rows of structured data.
Table structure is:
...
- 81
3
votes
1
answer
6k
views
Pandas throwing "Error tokenizing data. C error" while loading data sets from URL
I am trying to work on the Titanic competition to get hands on experience with data science & machine learning. I tried to load up the datasets from GitHub but pandas threw the following error:
<...
