Unfortunately, I didn't find much information surrounding this, no pdfs, no textbooks that discuss this in just enough detail. And I didn't see any forums posts about this. I just want to learn an example scenario of indexing in big data.

I've found this but I'd love some explanations and details about it rather than a picture alone.

enter image description here

  • $\begingroup$ Welcome to DataScienceSE. In general Google doesn't give the details of their methods, as a private company they don't want to and they don't have to. This picture gives a decent idea of what indexing basically looks like, but there are certainly a lot of tweaks to make things more accurate and more efficient. I'm not sure what kind of details you're looking for, is the concept of inverted index clear to you? $\endgroup$
    – Erwan
    Commented Jul 10, 2022 at 9:49
  • $\begingroup$ yeah it's somewhat clear to me. i don't want google's if they don't allow to tell, but in general the indexing techniques purposed for any big data. lucene is an example of indexing tool, but i want to learn like this figure rather than how lucene coding is done. high level overview. $\endgroup$
    – jewloa
    Commented Jul 10, 2022 at 10:03
  • $\begingroup$ I don't feel confident enough to give a full answer, but the main idea is indeed simply storing the inverted index usually together with something like TFIDF. The main issue is efficient storage/access, which is more a problem related to distributed database than an NLP. $\endgroup$
    – Erwan
    Commented Jul 10, 2022 at 11:30
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Jul 10, 2022 at 15:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.