Hot answers tagged

13 votes

importing csv data in python

For small data, I think pandas.read_csv is the way to go. For "medium" data, I recommend dask.read_csv And for big data, I recommend spark.read.csv
user avatar
  • 10.4k
11 votes

How to store strings in CSV with new line characters?

For anyone who's still facing the issue: None of the other suggestions worked for me or were too much work to do. Simply replace all \n with ...
user avatar
9 votes
Accepted

How to store strings in CSV with new line characters?

I assume that you want to keep the newlines in the strings for some reason after you have loaded the csv files from disk. Also that this is done again in Python. My solution will require Python 3, ...
user avatar
  • 13.9k
8 votes
Accepted

Recommendations for storing time series data

There are two solutions that are worth looking at: InfluxDB is an open source database platform specifically designed for time series data. The platform includes many optimized functions related to ...
user avatar
5 votes

Quick way to visually explore data?

When you hit the limits of an application like Apple Numbers, or Excel, you need to start using a programming language like Python, R or C. Using a programming language, you write your own application,...
user avatar
  • 731
5 votes

Reading CSVs with new lines in fields with Spark

There is a recently-added feature in Spark 2.2.0... spark.read.csv(file, multiLine=True) https://issues.apache.org/jira/browse/SPARK-19610 https://issues.apache....
user avatar
  • 151
5 votes
Accepted

Python - AttributeError: type object 'DataFrame' has no attribute 'read_csv'

read_csv() is not available on DataFrame. to read csvs using pandas - import pandas as pd data = pd.read_csv("file_name") if you check type(data), it will be ...
user avatar
5 votes

How to deal with large data sets

Welcome! I think you asked in the right place. If you have familiarity with Python, you might look at the numpy and Pandas libraries. Numpy implements fast array and matrix manipulations, and ...
user avatar
  • 2,118
5 votes

What is a good way to store processed CSV data to train model in Python?

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second. Most of the time is going to be spent by dataframe runtime in parsing data and creation of ...
user avatar
  • 2,181
4 votes
Accepted

importing csv data in python

Use pandas library: import pandas as pd pd.read_csv('foo.csv') Pandas identify the headers automatically and is a great tool for data wrangling. 10 Minutes ...
user avatar
  • 68
4 votes
Accepted

Breaking down a column in Pandas into a separate CSV for display in Tableau

If you don't need other columns, here is a solution. It splits the column, stacks in vertically and combines with "permalink" column ...
user avatar
4 votes
Accepted

Merging large CSV files in pandas

No, there is not. You will have to use an alternative tool like dask, drill, spark, or a good old fashioned relational database.
user avatar
  • 10.4k
4 votes
Accepted

How to deal with large data sets

As mentioned python's pandas library is good start. They have a lot of time series functionality, see e.g. the documentation here. You can load your data like so: ...
user avatar
  • 5,987
4 votes
Accepted

Getting stock data in a discipline manner from Yahoo finance

Use: data.stack().reset_index().rename(index=str, columns={"level_1": "Symbol"}).sort_values(['Symbol','Date']) Output: ...
user avatar
  • 156
4 votes
Accepted

File format where column names are repeated on each row

Using Miller (https://github.com/johnkerl/miller) mlr --ocsv unsparsify input.txt you will have this CSV ...
user avatar
  • 156
3 votes

Merging large CSV files in pandas

When faced with such situations (loading & appending multi-GB csv files), I found @user666's option of loading one data set (e.g. DataSet1) as a Pandas DF and appending the other (e.g. DataSet2) ...
user avatar
  • 236
3 votes

How to read file from user in Shiny and assign it to a variable in global.r?

I have added the following code to global.R file data <- reactiveValues() I used assign function in server.R to assign values to data in global.r ...
user avatar
3 votes
Accepted

Merging repeating data cells in csv

Pandas is a python library that you will find very useful for these types of tasks. Here is a stack overflow post that tells you how to do what you want to accomplish. It boils down to three very ...
user avatar
  • 6,668
3 votes
Accepted

Orange: Data has no target variable error

You can specify the target by choosing target in the feature column
user avatar
  • 684
3 votes

What is a good way to store processed CSV data to train model in Python?

You can find a nice benchmark for every approach in here.
user avatar
3 votes

How to save API data into CSV format?

On the page you linked there is actually a Python example on how to get the data. It is in Python 2, but I will show you how to make it work in Python 3. ...
user avatar
  • 13.9k
3 votes
Accepted

How to view entire 5788 rows and 7 columns.?

pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv ...
user avatar
  • 719
3 votes

How to get K most different rows in csv?

To select the most different rows, you would need to define first what you consider different. For ages and scores, subtracting values would work, for example: Row1 Age is 38 Score is 0.2 Row2 Age ...
user avatar
2 votes

How to read file from user in Shiny and assign it to a variable in global.r?

Sorry for the short answer, but if you use shiny with rmarkdown, it can be done as shown in this question: https://stackoverflow.com/questions/29253481/data-specific-selectinput-choices-in-rmd-shiny/...
user avatar
2 votes

How to read file from user in Shiny and assign it to a variable in global.r?

I did not use shiny but I tried in my gui code which I wrote using gwidgets to make a ariable global I used data<<-read.csv(inFile$datapath) you can try this.
user avatar
2 votes

Quick way to visually explore data?

Unfortunately, the highest number of rows in both Numbers and Microsoft excel is 1,048,576. So, if you have a file bigger than this, you are not able to open it. Some ideas: Connect your file in an ...
user avatar
  • 3,730
2 votes

Quick way to visually explore data?

You may install R - you are limited only with your RAM ...
user avatar
2 votes

Reading CSVs with new lines in fields with Spark

Augmenting Aneel's answer, I had to add escape='"' option get this working properly. Spark 2.3 ...
user avatar
  • 111
2 votes

Summarize and visualize a CSV in Java/Scala?

Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala. I've managed to get an environment setup where this would just be a couple ...
user avatar
2 votes
Accepted

How to shift rows values as columns in pandas?

You want pandas.get_dummies. If you call get_dummies on a categorical column, it will output the binary dummy variables you're looking for. You should then be able ...
user avatar
  • 1,057

Only top scored, non community-wiki answers of a minimum length are eligible