I am working on a use case, and I'm unsure of the best way to proceed: in order to analyze the behavior of users of a web-based music application, we retain all songs each has played since 2009. We store this information in flat files, each containing the songs played in a day. Each file contains 50M of lines, and we have 19M users. Our entire song catalogue consists of 35M tracks.
The format of these files is as follows:
id-user | country | id-artist | id-track
Question: I would like to represent each user by the songs he or she has played; this profile would be used by the production site. Does anyone have a suggestion for what would be the best way to process the whole chain?