In short, what would be the best method/tricks/techniques/tools for performing ad hoc sql (style) queries on 350TB of csv data? Would there be other options, tool wise that would do it faster if we dropped the "sql" requirement?
Is my best option Hive and as many servers I can't get my hands on? Would spark be of any benefit since this size of dataset wouldn't come close to fitting into memory? I have other related questions /ideas but don't want to bog this down.
The data is originally stored in a binary format that gets converted into an ASCII. The ASCII file is getting turned into CSV (actually tab separated but whatever). 1 binary file = 1 ASCII file = 1 CSV file. The data format is pretty simple, 200 to 500 header columns, each row is a sample of an attribute in 1-second intervals.
Example (first row is header): t,attribute1,attribute2...<attributeX> 1,val1,val2...<valx> 2,val1,val2...<valx> 3,val1,val2...<valx> ... X,val1,val2...<valx>
Example queries that could be ran.
1) Take t value (t is non-unique, as each file starts at 1 and ends at some random time) and compare different attribute values between t value of 100-1000 for all other attributes that were sampled at 100-1000.
2)There is UTC value as an attribute value (one of the columns), and same idea as #1 we might want to compare all attributes between X dates and compare those attributes to Y data that is from a different UTC range.
3)We might want to figure out a single particular attribute and trend that from the earliest time we saw that attribute for a different particular attribute.
These are my best examples as provided to me but other types of queries could exist. If you do have a question of why don't we cut out the middle man of the conversions of the binary data to CSV data, we could theoretically. However, the program that converts the files is quite old and is very involved.