By reading only a subset of the records of a file into R (or Python), one can speed up performance tremendously. I am curious about the mechanism behind this kind of file input. To read only a small part of a file, my guess is that I have to somehow access the file, load it in memory and filter out that small part that I am interested in (Otherwise how else would I be able to query the subset of the records that I want?). In otherwards that should then mean I have anyway had to read the whole file into memory. So if it was a 20GB file, it means I have had to read 20GB so as to filter off the 1GB that I want. Where do the performance benefits come from then?
There are multiple aspects to performance when it comes to reading a portion of the file. I am going to discuss the two most common one based on functions that are available in the base R.
What is R doing at the time of reading?
- Establish a connection with the file.
- Open the file
- Read the data sequentially
- Load the data in memory
- Close the connection
- dump the object into a variable if assigned, else render the output in some gui.
While it is doing all these steps, it is also trying to:
- Understand how big the data is so that it can allocate appropriate memory
- What are the data types or
classesso that it can (again) allocate appropriate memory.
2 is important here because unless
R reads the entire dataset, it cannot determine the datatype. Providing this information alone at the time of the read makes processing much faster.
By reading a subset of the data, R makes it easier to interpret the datatype and then assigns it to the subsequent sets without worrying about what the actual datatype is.
Does R really loads all the data every time?
Functions that allow filter at the time of read, really target the third step of reading data sequentially.
Each row is compared against the filter criteria and the result stored as a binary logical index. This index is then used to determine, what remains when the connection closes. I could not find specific reference in the documentation if the discard happens immediately or post entire read, but I will assume that the
R collects the garbage automatically as it reads with a filter criteria and makes the memory available to a subsequent line of data.
May be there are other experts who can throw more light on the real inner workings here and either validate or correct my assumptions, but hope my context helps.