0
$\begingroup$

I have searched at all possible areas and haven't been able to find the solution to my problem. I am hoping that someone here would be able to help me out. If you feel that this question is already answered, I would appreciate the link.

I am using R3.2.2 64 bit on a windows 7 machine. I have a fixed width file of size 1.9 GB and has approx 18 million records in it. I need to read the file based on a filter condition RecordID = "Cast". All my attempts to read the file.dat has failed due to memory constraints. Any help on how to use package ff for this purpose?

I am open to other methods also.

$\endgroup$
2
$\begingroup$

Try the LaF package. See this: https://stackoverflow.com/a/24716798/1427069

Once you create a laf object, you can create a data frame or data table using your row filter condition.

library(LaF)
laf <- laf_open_fwf("foo.dat",
                    column_widths = colWidths, 
                    column_types=rep("character", length(colWidths)),
                    column_names = colNames)
castData <- laf[RecordID == "Cast", ]
$\endgroup$
  • $\begingroup$ - Thanks. I was able to successfully create the object as an ASCII file. However, filtering the records, always lead to the below error comparison (1) is possible only for atomic and list types.I have tried both index positions and subset() without much ado. I also tried input <- subset(source_input, source_input$Record.ID =="Cast") $\endgroup$ – Drj Sep 24 '15 at 20:00
0
$\begingroup$

This is a late answer. There are syntax examples elsewhere, but, for the sake of completeness, the syntax should be as follows:
library(LaF) laf <- laf_open_fwf("foo.dat", column_widths = colWidths, column_types=rep("character", length(colWidths)), column_names = colNames) castData <- laf[laf$RecordID[] == "Cast", ]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.