1
$\begingroup$

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document where a "keyword" is more relevant, he also wants us to divide the project in 3 parts:

  • Data acquisition, preprocing (cleaning, transform, join, etc), loading. ETL PROCESS.
  • Data processing.
  • User friendly application.

I need to define what technologies or methods i'm gonna user for each of the parts of the project, but i have no idea what to do on the ETL part since the documents are gonna be written in legible English (they are books), i will appreciate any information you can give me on this but also about the other parts of the project.

Thanks a million for reading.

$\endgroup$
  • $\begingroup$ Apache Spark is your friend! I'm sure there're tones of tutorials and books you find with simple google search. $\endgroup$ – Ehsan M. Kermani Apr 16 '17 at 19:55
  • $\begingroup$ You might want to check out ElasticSearch. It's primary use case is to store log files, but it can store anything, including lots of text. works with key value pairs. $\endgroup$ – Placidia Apr 16 '17 at 20:04
1
$\begingroup$

I would suggest a data fabric. That would meet your need for data acquisition, preprocessing, data quality, master data management, etc.

Given I work for Talend, I would suggest our data fabric. =)

Here’s a case study with the Panama Papers. https://www.talend.com/blog/2017/01/17/talend-data-masters-2016-icij-decoded-panama-papers-talend/

The concept in that case study of combining a data fabric with analytic tools is the general concept to use regardless of which data fabric you use.

You can find the trial and open versions of Talend at https://www.talend.com/download/.

Edit: Here’s another example, which implements a backend and UI showing the ebooks of the Gutenberg project. It allows to import the whole gutenberg index using a camel route. https://github.com/Talend/tesb-rt-se/tree/master/examples/tesb/ebook

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.