Skip to main content
19 votes
Accepted

How to store strings in CSV with new line characters?

Here is a simple solution: Replace all \n with \\n before saving to CSV. This will preserve the newline characters. ...
betelgeuse's user avatar
13 votes

importing csv data in python

For small data, I think pandas.read_csv is the way to go. For "medium" data, I recommend dask.read_csv And for big data, I recommend spark.read.csv
Emre's user avatar
  • 10.5k
10 votes

How to store strings in CSV with new line characters?

I assume that you want to keep the newlines in the strings for some reason after you have loaded the csv files from disk. Also that this is done again in Python. My solution will require Python 3, ...
n1k31t4's user avatar
  • 14.9k
8 votes
Accepted

AttributeError: type object 'DataFrame' has no attribute 'read_csv'

read_csv() is not available on DataFrame. to read csvs using pandas: ...
manohar amrutkar's user avatar
5 votes

Reading CSVs with new lines in fields with Spark

There is a recently-added feature in Spark 2.2.0... spark.read.csv(file, multiLine=True) https://issues.apache.org/jira/browse/SPARK-19610 https://issues.apache....
Aneel's user avatar
  • 151
5 votes

How to deal with large data sets

Welcome! I think you asked in the right place. If you have familiarity with Python, you might look at the numpy and Pandas libraries. Numpy implements fast array and matrix manipulations, and ...
tom's user avatar
  • 2,248
5 votes

What is a good way to store processed CSV data to train model in Python?

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second. Most of the time is going to be spent by dataframe runtime in parsing data and creation of ...
Shamit Verma's user avatar
  • 2,259
5 votes
Accepted

Getting stock data in a discipline manner from Yahoo finance

Use: data.stack().reset_index().rename(index=str, columns={"level_1": "Symbol"}).sort_values(['Symbol','Date']) Output: ...
Stef's user avatar
  • 166
5 votes
Accepted

File format where column names are repeated on each row

Using Miller (https://github.com/johnkerl/miller) mlr --ocsv unsparsify input.txt you will have this CSV ...
aborruso's user avatar
  • 176
5 votes
Accepted

Suitability of column store formats like Parquet for table data

Your question raises issues of both low-level storage format and high-level retention policy. tl;dr: Yes, keep using parquet. storage format Parquet offers several attractive features: binary ...
J_H's user avatar
  • 1,130
4 votes
Accepted

importing csv data in python

Use pandas library: import pandas as pd pd.read_csv('foo.csv') Pandas identify the headers automatically and is a great tool for data wrangling. 10 Minutes ...
Codezen's user avatar
  • 68
4 votes
Accepted

Breaking down a column in Pandas into a separate CSV for display in Tableau

If you don't need other columns, here is a solution. It splits the column, stacks in vertically and combines with "permalink" column ...
Andrey Lukyanenko's user avatar
4 votes
Accepted

Merging large CSV files in pandas

No, there is not. You will have to use an alternative tool like dask, drill, spark, or a good old fashioned relational database.
Emre's user avatar
  • 10.5k
4 votes
Accepted

How to deal with large data sets

As mentioned python's pandas library is good start. They have a lot of time series functionality, see e.g. the documentation here. You can load your data like so: ...
oW_'s user avatar
  • 6,367
3 votes
Accepted

Orange: Data has no target variable error

You can specify the target by choosing target in the feature column
Tarek's user avatar
  • 724
3 votes

Merging large CSV files in pandas

When faced with such situations (loading & appending multi-GB csv files), I found @user666's option of loading one data set (e.g. DataSet1) as a Pandas DF and appending the other (e.g. DataSet2) ...
vsdaking's user avatar
  • 236
3 votes

What is a good way to store processed CSV data to train model in Python?

You can find a nice benchmark for every approach in here.
Francesco Pegoraro's user avatar
3 votes

How to save API data into CSV format?

On the page you linked there is actually a Python example on how to get the data. It is in Python 2, but I will show you how to make it work in Python 3. ...
n1k31t4's user avatar
  • 14.9k
3 votes
Accepted

How to view entire 5788 rows and 7 columns.?

pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv ...
Craig's user avatar
  • 974
3 votes

How to get K most different rows in csv?

To select the most different rows, you would need to define first what you consider different. For ages and scores, subtracting values would work, for example: Row1 Age is 38 Score is 0.2 Row2 Age ...
Henrique Nader's user avatar
2 votes

Reading CSVs with new lines in fields with Spark

Augmenting Aneel's answer, I had to add escape='"' option get this working properly. Spark 2.3 ...
Thusitha's user avatar
  • 111
2 votes
Accepted

How to shift rows values as columns in pandas?

You want pandas.get_dummies. If you call get_dummies on a categorical column, it will output the binary dummy variables you're looking for. You should then be able ...
R Hill's user avatar
  • 1,105
2 votes

ML algorithm for determining CSV file header names based on content

Yes, it might not be exactly natural Language Understanding but CRF is an excellent algorithm to train Named Entity Recognition tasks and is the stamdard model used by Stanford NLP group. You can try ...
Himanshu Rai's user avatar
  • 1,848
2 votes

Output data from scikit learn logistic regression

Each line is treated independently at prediction, so you can be sure that the data is kept in the same order. For simplicity, you can keep your data in a pandas dataframe. Here's a short working ...
amyrit's user avatar
  • 256
2 votes

Search and Replace in specific column of csv with python

You can do it easily by using pandas, like this: ...
pcko1's user avatar
  • 3,950
2 votes

Data scraping & NLP?

NLTK A great starting point for keyword extraction is the NLTK (natural language toolkit) library. To extract keywords, you probably need to tokenize your data, breaking each word out into a token, ...
the_cat_lady's user avatar
2 votes
Accepted

Creating data model out of .csv file using Python

Adding on top of @El Burro's answer, most of the training/testing proof of concepts of model making happens on manipulating DataFrame which provides easy ...
Kiritee Gak's user avatar
  • 1,799
2 votes

Creating data model out of .csv file using Python

In my opinion this is probably achieved fastest with a rule set. You can easily load and manipulate data via pandas. Then you just have to determine what are rules. But without any details its ...
El Burro's user avatar
  • 800
2 votes

I got the following error : 'DataFrame' object has no attribute 'data'

"sklearn.datasets" is a scikit package, where it contains a method load_iris(). load_iris(), by default return an object which holds data, target and other members in it. In order to get actual ...
vipin bansal's user avatar
  • 1,272

Only top scored, non community-wiki answers of a minimum length are eligible