By reading only a subset of the records of a file into R (or Python), one can speed up performance tremendously. I am curious about the mechanism behind this kind of file input. To read only a small part of a file, my guess is that I have to somehow access the file, load it in memory and filter out that small part that I am interested in (Otherwise how else would I be able to query the subset of the records that I want?). In otherwards that should then mean I have anyway had to read the whole file into memory. So if it was a 20GB file, it means I have had to read 20GB so as to filter off the 1GB that I want. Where do the performance benefits come from then?


There are multiple aspects to performance when it comes to reading a portion of the file. I am going to discuss the two most common one based on functions that are available in the base R.

What is R doing at the time of reading?

  1. Establish a connection with the file.
  2. Open the file
  3. Read the data sequentially
  4. Load the data in memory
  5. Close the connection
  6. dump the object into a variable if assigned, else render the output in some gui.

While it is doing all these steps, it is also trying to:

  1. Understand how big the data is so that it can allocate appropriate memory
  2. What are the data types or classes so that it can (again) allocate appropriate memory.

Sub bullet 2 is important here because unless R reads the entire dataset, it cannot determine the datatype. Providing this information alone at the time of the read makes processing much faster.

By reading a subset of the data, R makes it easier to interpret the datatype and then assigns it to the subsequent sets without worrying about what the actual datatype is.

Does R really loads all the data every time?

Functions that allow filter at the time of read, really target the third step of reading data sequentially.

Each row is compared against the filter criteria and the result stored as a binary logical index. This index is then used to determine, what remains when the connection closes. I could not find specific reference in the documentation if the discard happens immediately or post entire read, but I will assume that the R collects the garbage automatically as it reads with a filter criteria and makes the memory available to a subsequent line of data.

May be there are other experts who can throw more light on the real inner workings here and either validate or correct my assumptions, but hope my context helps.


It was not clear when you said "reading only a subset of the records of a file into R". If head(dataset, 100) is the subset that you intended, it doesn't read the entire file into memory. It just reads the first 100 rows into memory.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.