13
votes
importing csv data in python
For small data, I think pandas.read_csv is the way to go.
For "medium" data, I recommend dask.read_csv
And for big data, I recommend spark.read.csv
- 10.4k
11
votes
How to store strings in CSV with new line characters?
For anyone who's still facing the issue:
None of the other suggestions worked for me or were too much work to do. Simply replace all \n with ...
- 210
9
votes
Accepted
How to store strings in CSV with new line characters?
I assume that you want to keep the newlines in the strings for some reason after you have loaded the csv files from disk. Also that this is done again in Python. My solution will require Python 3, ...
- 13.9k
8
votes
Accepted
Recommendations for storing time series data
There are two solutions that are worth looking at:
InfluxDB is an open source database platform specifically designed for time series data. The platform includes many optimized functions related to ...
- 206
5
votes
Quick way to visually explore data?
When you hit the limits of an application like Apple Numbers, or Excel, you need to start using a programming language like Python, R or C. Using a programming language, you write your own application,...
- 731
5
votes
Reading CSVs with new lines in fields with Spark
There is a recently-added feature in Spark 2.2.0...
spark.read.csv(file, multiLine=True)
https://issues.apache.org/jira/browse/SPARK-19610
https://issues.apache....
- 151
5
votes
Accepted
Python - AttributeError: type object 'DataFrame' has no attribute 'read_csv'
read_csv() is not available on DataFrame. to read csvs using pandas -
import pandas as pd
data = pd.read_csv("file_name")
if you check type(data), it will be ...
- 121
5
votes
How to deal with large data sets
Welcome! I think you asked in the right place.
If you have familiarity with Python, you might look at the numpy and Pandas libraries. Numpy implements fast array and matrix manipulations, and ...
- 2,118
5
votes
What is a good way to store processed CSV data to train model in Python?
With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.
Most of the time is going to be spent by dataframe runtime in parsing data and creation of ...
- 2,181
4
votes
Accepted
importing csv data in python
Use pandas library:
import pandas as pd
pd.read_csv('foo.csv')
Pandas identify the headers automatically and is a great tool for data wrangling.
10 Minutes ...
- 68
4
votes
Accepted
Breaking down a column in Pandas into a separate CSV for display in Tableau
If you don't need other columns, here is a solution. It splits the column, stacks in vertically and combines with "permalink" column
...
4
votes
Accepted
Merging large CSV files in pandas
No, there is not. You will have to use an alternative tool like dask, drill, spark, or a good old fashioned relational database.
- 10.4k
4
votes
Accepted
How to deal with large data sets
As mentioned python's pandas library is good start. They have a lot of time series functionality, see e.g. the documentation here. You can load your data like so:
...
oW_♦
- 5,987
4
votes
Accepted
Getting stock data in a discipline manner from Yahoo finance
Use:
data.stack().reset_index().rename(index=str, columns={"level_1": "Symbol"}).sort_values(['Symbol','Date'])
Output:
...
- 156
4
votes
Accepted
File format where column names are repeated on each row
Using Miller (https://github.com/johnkerl/miller)
mlr --ocsv unsparsify input.txt
you will have this CSV
...
- 156
3
votes
Merging large CSV files in pandas
When faced with such situations (loading & appending multi-GB csv files), I found @user666's option of loading one data set (e.g. DataSet1) as a Pandas DF and appending the other (e.g. DataSet2) ...
- 236
3
votes
How to read file from user in Shiny and assign it to a variable in global.r?
I have added the following code to global.R file
data <- reactiveValues()
I used assign function in server.R to assign values to data in global.r
...
3
votes
Accepted
Merging repeating data cells in csv
Pandas is a python library that you will find very useful for these types of tasks.
Here is a stack overflow post that tells you how to do what you want to accomplish.
It boils down to three very ...
- 6,668
3
votes
Accepted
Orange: Data has no target variable error
You can specify the target by choosing target in the feature column
- 684
3
votes
What is a good way to store processed CSV data to train model in Python?
You can find a nice benchmark for every approach in here.
3
votes
How to save API data into CSV format?
On the page you linked there is actually a Python example on how to get the data. It is in Python 2, but I will show you how to make it work in Python 3.
...
- 13.9k
3
votes
Accepted
How to view entire 5788 rows and 7 columns.?
pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html
Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv ...
- 719
3
votes
How to get K most different rows in csv?
To select the most different rows, you would need to define first what you consider different. For ages and scores, subtracting values would work, for example:
Row1
Age is 38
Score is 0.2
Row2
Age ...
- 511
2
votes
How to read file from user in Shiny and assign it to a variable in global.r?
Sorry for the short answer, but if you use shiny with rmarkdown, it can be done as shown in this question:
https://stackoverflow.com/questions/29253481/data-specific-selectinput-choices-in-rmd-shiny/...
- 141
2
votes
How to read file from user in Shiny and assign it to a variable in global.r?
I did not use shiny but I tried in my gui code which I wrote using gwidgets to make a ariable global I used data<<-read.csv(inFile$datapath)
you can try this.
- 165
2
votes
Quick way to visually explore data?
Unfortunately, the highest number of rows in both Numbers and Microsoft excel is 1,048,576. So, if you have a file bigger than this, you are not able to open it.
Some ideas:
Connect your file in an ...
- 3,730
2
votes
Quick way to visually explore data?
You may install R - you are limited only with your RAM
...
- 1,103
2
votes
Reading CSVs with new lines in fields with Spark
Augmenting Aneel's answer, I had to add escape='"' option get this working properly. Spark 2.3
...
- 111
2
votes
Summarize and visualize a CSV in Java/Scala?
Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala.
I've managed to get an environment setup where this would just be a couple ...
- 269
2
votes
Accepted
How to shift rows values as columns in pandas?
You want pandas.get_dummies.
If you call get_dummies on a categorical column, it will output the binary dummy variables you're looking for. You should then be able ...
- 1,057
Only top scored, non community-wiki answers of a minimum length are eligible