13
For small data, I think pandas.read_csv is the way to go.
For "medium" data, I recommend dask.read_csv
And for big data, I recommend spark.read.csv
8
There are two solutions that are worth looking at:
InfluxDB is an open source database platform specifically designed for time series data. The platform includes many optimized functions related to time and you can collect data on any interval and compute rollups/aggregations when reporting. The company recently launched a query app called Chronograf. I ...
8
I assume that you want to keep the newlines in the strings for some reason after you have loaded the csv files from disk. Also that this is done again in Python. My solution will require Python 3, although the principle could be applied to Python 2.
The main trick
This is to replace the \n characters before writing with a weird character that otherwise ...
7
For anyone who's still facing the issue:
None of the other suggestions worked for me or was too much work to do. I simply replaced all \n with \\n before saving to CSV and it'll preserve the newline character.
df.Column_Name = df.Column_Name.apply(lambda x : x.replace('\n', '\\n'))
df.to_csv("df.csv", index=False)
5
When you hit the limits of an application like Apple Numbers, or Excel, you need to start using a programming language like Python, R or C. Using a programming language, you write your own application, which is not constrained by arbitrary limits like 220 rows by 214 columns. (Of course, you are still constrained by the physical memory and the way that a ...
5
There is a recently-added feature in Spark 2.2.0...
spark.read.csv(file, multiLine=True)
https://issues.apache.org/jira/browse/SPARK-19610
https://issues.apache.org/jira/browse/SPARK-20980
5
read_csv() is not available on DataFrame. to read csvs using pandas -
import pandas as pd
data = pd.read_csv("file_name")
if you check type(data), it will be pandas DataFrame.
5
Welcome! I think you asked in the right place.
If you have familiarity with Python, you might look at the numpy and Pandas libraries. Numpy implements fast array and matrix manipulations, and Pandas arranges numpy objects into tables. Along with scipy, they make the basis of the Python numerical computing stack.
Without more detail it's difficult to ...
5
With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.
Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.
4
Use pandas library:
import pandas as pd
pd.read_csv('foo.csv')
Pandas identify the headers automatically and is a great tool for data wrangling.
10 Minutes intro to pandas
4
If you don't need other columns, here is a solution. It splits the column, stacks in vertically and combines with "permalink" column
df.set_index('permalink').category_list.str.split('|', expand=True).stack().reset_index('permalink').rename(columns={0:'category'})
permalink category
0 /organization/-qounter Application Platforms
1 /organization/-...
4
No, there is not. You will have to use an alternative tool like dask, drill, spark, or a good old fashioned relational database.
4
As mentioned python's pandas library is good start. They have a lot of time series functionality, see e.g. the documentation here. You can load your data like so:
import pandas as pd
s = pd.read_csv("your_file.csv",header=None,index_col=0,names=['timestamp','value'],squeeze=True,parse_dates=[0])
This gives you a timeseries with the timestamps as index:
s
...
4
Using Miller (https://github.com/johnkerl/miller)
mlr --ocsv unsparsify input.txt
you will have this CSV
col1,col2,col3
datac1r1,datac2r1,datac3r1
datac1r2,datac2r2,datac3r2
datac1r3,datac2r3,datac3r3
datac1r4,datac2r4,datac3r4
3
When faced with such situations (loading & appending multi-GB csv files), I found @user666's option of loading one data set (e.g. DataSet1) as a Pandas DF and appending the other (e.g. DataSet2) in chunks to the existing DF to be quite feasible.
Here is the code I implement:
import pandas as pd
amgPd = pd.DataFrame()
for chunk in pd.read_csv(path1+'...
3
I have added the following code to global.R file
data <- reactiveValues()
I used assign function in server.R to assign values to data in global.r
output$contents <- renderText({
if(is.null(input$file1))return()
inFile <- input$file1
data2<-read.csv(inFile$datapath)
assign('data',data2,envir=.GlobalEnv)
print(summary(...
3
Pandas is a python library that you will find very useful for these types of tasks.
Here is a stack overflow post that tells you how to do what you want to accomplish.
It boils down to three very pythonic lines with a groupby and transformation followed by a drop_duplicates:
import pandas
df = pandas.read_csv('csvfile.csv', header = 0)
df['Total'] = df....
3
You can specify the target by choosing target in the feature column
3
"sklearn.datasets" is a scikit package, where it contains a method
load_iris().
load_iris(), by default return an object which holds data, target and other members in it. In order to get actual values you have to read the data and target content itself.
Whereas 'iris.csv', holds feature and target together.
FYI: If you set return_X_y as True in ...
3
You can find a nice benchmark for every approach in here.
3
Use:
data.stack().reset_index().rename(index=str, columns={"level_1": "Symbol"}).sort_values(['Symbol','Date'])
Output:
Date Symbol Adj Close ... Low Open Volume
0 2018-01-01 AXISBANK.NS 564.80 ... 560.50 563.80 6943234
7 2018-01-02 AXISBANK.NS 558.81 ... 556.35 567.00 6292268
14 2018-01-03 ...
3
On the page you linked there is actually a Python example on how to get the data. It is in Python 2, but I will show you how to make it work in Python 3.
import urllib
import json # Used to load data into JSON format
from pprint import pprint # pretty-print
url = "https://data.sa.gov.au/data/api/3/action/datastore_search?resource_id=...
3
pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html
Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv editor would be more useful.
3
To select the most different rows, you would need to define first what you consider different. For ages and scores, subtracting values would work, for example:
Row1
Age is 38
Score is 0.2
Row2
Age is 87
Score is 1.0
Difference by numeric feature:
Age Diff is 49
Score Diff is 0.8
Those values could be normalized or weighted to account for different ...
2
Sorry for the short answer, but if you use shiny with rmarkdown, it can be done as shown in this question:
https://stackoverflow.com/questions/29253481/data-specific-selectinput-choices-in-rmd-shiny/29255723#29255723
Create a reactive function that reads the file in!
2
I did not use shiny but I tried in my gui code which I wrote using gwidgets to make a ariable global I used data<<-read.csv(inFile$datapath)
you can try this.
2
Unfortunately, the highest number of rows in both Numbers and Microsoft excel is 1,048,576. So, if you have a file bigger than this, you are not able to open it.
Some ideas:
Connect your file in an SQL database and work with it from there.
Use another tool like delimitware which let you open files up to 2 billions rows.
Use Python Pandas with pandas....
2
You may install R - you are limited only with your RAM
## read csv file
df <- read.csv("l.csv")
## column names
> colnames(df)
[1] "a" "b"
> head(df)
a b
1 1 1
2 2 4
...
# quick overview
> summary(df)
a b
Min. : 1.0 Min. : 1
1st Qu.: 2500.8 1st Qu.: 6253751
...
2
Augmenting Aneel's answer, I had to add escape='"' option get this working properly. Spark 2.3
spark.read.csv(DATA_FILE, sep=',', escape='"', header=True,
inferSchema=True, multiLine=True).count()
159571
Interestingly, Pandas can read this without any additional instructions.
pd.read_csv(DATA_FILE).shape
(159571, 8)
2
Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala.
I've managed to get an environment setup where this would just be a couple of lines. I dont actually use ScalaLab, rather borrow some of its code, I use Intellij worksheets instead.
Only top voted, non community-wiki answers of a minimum length are eligible