Overcoming "too many indices for array" error when loading file into Orange

Question

I am attempting to load the Stack Overflow 2017 developer survey results into Orange. The dataset has ~51.4K rows with 156 features. It is comma separated and is about 91MB in size.

When I attempt to load this file into Orange, I get the following error:

Cannot parse dataset H:/survey_results_public.csv: too many indices for array

How can I overcome this error?

Answer 1

This error basically means that your data is incorrectly formatted and Orange cannot read it. Somewhere there is either a redundant column delimiter or it might be absent. Either way, you should carefully check that all your columns are correctly separated by tabs or any other consistent separator.

Answer 2

I had the same issue, also trying to read a .csv file. Although the answer from @minerals would be technically correct, it assumes that your .csv file is not formatted correctly.

In my case, the csv file had the correct number of indices, and was validated as not corrupted (using the freeware software 'CSVed'). I was able to load my dataframe by saving the csv as a .xlsx (Microsoft Excel). It loaded fine. Interestingly, exporting back to .csv recreated the error!

Answer 3

Months later, after struggling with the same task, I've managed to load it!

TLDR version:

Replace all double-double-quotes ("") within the values in the file by something harmless (i.e. '). Orange then manages to load it.

Explanation:

The comment by "CodesInChaos" did indeed point in the right direction: This is a bug (in Orange or some library it uses) in parsing the quoted values. To be specific, it does not correctly handle correctly escaped double quotes inside the values of the columns.

RFC 4180 says:

If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote.

However, occurrence of this kind of escaping in the file causes the observed error in the Orange.

NOTE: Careful, simple search & replace may not cut it in general case - if the value itself starts with the double-quote, you would replace """ with '", which would produce invalid csv.

Also of NOTE: Weka seems to have this limitation too. Additionally, it does not tolerate line-breaks inside the quoted values - which again, according to the RFC are valid.

Moral of the story: CSV parsing is not easy.