Questions tagged [csv]

Comma-Separated Values are a list of plain text values delimited by commas, or a file containing one or more lists in that format.

Filter by
Sorted by
Tagged with
1
vote
0answers
15 views

Built strong base for Unsupervised Learning [closed]

I’m am new into machine learning, recently I have put a task upon my shoulders to Detect Outliers in Dataset. The anomaly detection should be done using Unsupervised learning and preferably use ...
2
votes
1answer
35 views

How to split a CSV into a dataframe without newline

Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same ...
1
vote
0answers
22 views

Filter and plot values using matplotlib

In order to understand data a bit more, I want to plot one column from my csv file using matplotlib as a dotted graph This is how my my csv file looks like. I want to filter the csv with ...
3
votes
0answers
30 views

Sources to learn Anomaly Detection

Project Goal- Create an Anomaly Detection tool. As the first step I worked on some python code that would parse my XML files into a Dataframe in csv file. The next step is to go into machine ...
1
vote
2answers
26 views

Should I use MongoDB instead of storing data in CSV in python?

I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
0
votes
1answer
32 views

Deep learning model gives random results

First I am new to machine learning if it is an obvious question, I am sorry. ...
0
votes
1answer
28 views

How can I override the values of 1 column in a big dataframe using a small second dataframe? [closed]

I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are: A <...
0
votes
1answer
218 views

How to save API data into CSV format?

Is there an easy way to convert API data with python commands and output the results in CSV format? Here's what I'm doing now: ...
0
votes
1answer
53 views

how to see in pandas the element of a csv table with many columns (>30) which the names of its columns is more than 10 character?

how to see in pandas the element of a csv table with many columns (>25) which the names of its columns is more than 10 character? I have 5000 rows and 32 columns and the label of some columns are more ...
0
votes
1answer
90 views

How to create a parquet file from a query to a mysql table

Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic: ...
0
votes
1answer
298 views

Pandas - how to make further sub-groups after groupby( ) and get_group( )

Sample data in .csv format ...
4
votes
1answer
373 views

Any one read the over 100gb csv file and successfully concatenation?

I have been searching for the deal with large CSV file read method Its over 100gb and need to know how deal with the chunk file processing and make concatenation faster ...
0
votes
0answers
31 views

pd.read_csv() not reading /tmp folder AWS Lambda

I am using AWS Lambda to download a CSV from S3 and then upload each recod to a mongo. I download the csv to /tmp folder using the boto3 client and with pandas read it with pd.read_csv(). However the ...
2
votes
2answers
79 views

File format where column names are repeated on each row

I have received a dataset in text file with the following format ...
0
votes
1answer
222 views

Getting stock data in a discipline manner from Yahoo finance

I used the below code for downloading stock data from yahoo finance:- ...
0
votes
0answers
160 views

Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment

I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark). Question Need a ...
0
votes
1answer
2k views

np.loadtxt function showing error, could not convert string to float: '“Date”'

This is my stock market csv data: ...
1
vote
2answers
40 views

Removing # from header

I have a $(418,2)$ matrix and I want to convert it to csv. so I write: np.savetxt('titanic1.csv', Sol, fmt='%.2f', delimiter=",",header="PassengerId,Survived") ...
2
votes
1answer
2k views

File converter: from CSV to HDF5

Can anyone recommend any command line tool for converting large CSV file into HDF5 format?
2
votes
1answer
74 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
1
vote
1answer
2k views

Pandas throwing “Error tokenizing data. C error” while loading data sets from URL

I am trying to work on the Titanic competition to get hands on experience with data science & machine learning. I tried to load up the datasets from GitHub but pandas threw the following error: <...
-1
votes
2answers
47 views

How to calculate occurance of a name per hour basis of monthly csv data using python

I have csv file which have DateTime and application OS name. DateTime have for 1 month ,per hour basis. The csv looks as follows, ...
0
votes
0answers
27 views

Complex Value Neural Network with .csv

This is the part of my program in python: Path of input data and output data for learning. ...
2
votes
1answer
3k views

how to transform a folder of images into csv file?

I have a data set for segmentation project. The dataset have 2 folders containing train folder and annotated_train_data folder, both folders have images. input and label both are images. I want to ...
0
votes
1answer
52 views

Extract or subset hundreds of columns from a data frame

I need to extract many columns from a dataset. I have a very large csv file with thousands of columns and rows, and I read it into R using: ...
5
votes
2answers
30k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
1
vote
3answers
802 views

What is a good way to store processed CSV data to train model in Python?

I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of ...
1
vote
1answer
312 views

Whole time column in csv file convert into UTC (epoch) using python

I have a datset with time and column. I want to plot a graph with time and value. I tried many methods but didn't come proper graph. Because I have a time series. Then I thought I will convert time ...
0
votes
0answers
144 views

How to plot graph including y axis =value and x axis = datetime with timedelta (2 hrs) reading from csv file using python?

I have dataset import from csv file. Here I want to plot graph with datetime including time delta (two hours). In x axis datetime should be visible like %d/%m/%Y %H:%M. y is value. I wrote the code ...
2
votes
1answer
481 views

day is out of range for month import from csv file in python

Here I have one csv file with different values and date_time. I wrote the code and run it. Then it gave me an error time data '8/6/2018 6:45' does not match format ' %d/%m/%Y %H:%M:%S' I want to plot ...
1
vote
0answers
164 views

alternatives to pyspark to query a csv file locally (no distributed computation needed)

I am reading a csv file with pyspark to extract some information from it. I am running pyspark locally and I do not need distributed computation. On the contrary I need some plotting library such as ...
2
votes
2answers
4k views

How to transform a folder of images in a csv file

I have a folder with a lot of images that I want to use to bild a classificator using a SVM model in python with sklearn. I've always used csv file as train/test set with sklearn, how can I make it? (...
1
vote
2answers
2k views

how to take CSV file input in list of tuples

I have a .txt(data.txt) file containing csv data like: X Class 15.0001 Yes 18.00 NO 17.07 Yes I need to make a function to return a list of ...
0
votes
1answer
559 views

Find ID for the name in the list from csv file in python [closed]

The CSV file contains: ID,Name 1,AS 2,er 3,rtf 4,addfs The list contains the name for example (er,rtf) I want to find the ID's corresponding to these names mentioned above How to find the ID's using ...
2
votes
1answer
42 views

Creating a single number from a numpy array - Python [closed]

I am working on a gender classification project. I am extracting the pixels of an image using a Numpy array in Python, similar to the one below: ...
1
vote
2answers
25 views

Any tool to load large csv files quickly on windows 7

I am working with relatively large CSV files (1.5 MB+) and while testing the code, I have to frequently open up the CSV files to read a particular row, validate the data are there and then close the ...
3
votes
6answers
74k views

I got the following error : 'DataFrame' object has no attribute 'data'

I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result: Program: ...
3
votes
1answer
19k views

How to store strings in CSV with new line characters?

My question is: what are ways I can store strings in a CSV that contain newline characters (i.e. \n), where each data point is in one line? Sample data This is a ...
0
votes
2answers
19k views

Search and Replace in specific column of csv with python

I created a program that search and replaces over an entire csv file but I need to make so it is column specific. Is there a way to only search via a column. Here is my code, I am pretty new to python ...
1
vote
1answer
284 views

Data scraping & NLP?

I'm scraping data from Bing search results for (non-commercial purposes, of course) on Python using BeautifulSoup. I've entered an Indian dessert name, called 'rasmalai' as the word that I am focusing ...
0
votes
1answer
257 views

How can I transpose a high dimensional dataset?

I have a (.csv) file with more than 35,000 rows and 100 columns, where the rows represent the attributes and columns represent the instances. In Excel the maximum number of columns is 16,384. ...
0
votes
1answer
5k views

Orange: Data has no target variable error

I am trying to apply Random Forest algorithm on a data set using Orange. The target variable is not set in the data set. However, I know which column is the target variable. How can I specify the ...
1
vote
3answers
1k views

Creating data model out of .csv file using Python

I want to create a data model out of a .csv file using Python. I mean to create dependencies, for example the primary key and stuff such that I can check if the new ...
-1
votes
1answer
3k views

I have a csv file with time as the datetime format and want to edit the same column into miliseconds, how do I change that?

Input file is : time event earthAcceleration.x earthAcceleration.y earthAcceleration.z 2017-11-22T09:20:13.944 motion -0.006379993 -0.001029071 -0.010781396 2017-11-22T09:20:13.954 motion 0....
2
votes
2answers
1k views

How to deal with large data sets

So i'm very new to this, forgive my silly questions. I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job. I have .csv files containing data ...
0
votes
2answers
11k views

Python inserts newline by writing to csv

I am trying to scrape http://www.the-numbers.com/movie/budgets/all but when I write the table into a csv file, there is an additional line with the counter index written in between each movie row... ...
0
votes
1answer
15k views

Converting Json file to Dataframe Python

I have a json file which has multiple events, each event starts with EventVersion Key. The data looks similar to the following synthesized data. ...
1
vote
1answer
498 views

R: reading big data files in R

In R, there are libraries which help speed up the process of reading huge data files. Examples of these libraries include sqldf and ff. What would be the disadvantage of using these packages to read ...
2
votes
2answers
507 views

Breaking down a column in Pandas into a separate CSV for display in Tableau

My data is coming from a CSV, which should be visualized in Tableau. However, the data contains the column category_list, which consists of values separated by a ...
0
votes
1answer
2k views

How to shift rows values as columns in pandas?

Input: I have csv file like below as input.... ...