0
$\begingroup$

I have zero to experience in data science or machine learning. Because of this I am not able to determine if building a corpus does apply to the problem I am trying to solve.

I am trying to build a reference site for cloud technologies such as AWS Google Cloud.

I was able to build structured data and identify primary entities with in a single ecosystem using standard web scraping and sql.queries.

But I wanted to have the ability to have a mechanism that can autonomously identify entities and related information that is relevant to that entity and other entities it has relationships with.

Given that a specific ecosystem documentation follows a certain style can I use few entities as training docs and then have it classify the information I mentioned above.

Is the starting point to this is to build a corpus? I tried it out nltk categorized corpus builder.

Is it fine to include a specific document in multiple categories? For example an instance in AWS can be in category ec2 and a general category Computing unit

Anyways is this problem I am trying to solve fit into the general NLP ML space?

$\endgroup$
  • $\begingroup$ Down voters care to explain the reason? $\endgroup$ – DevZer0 Feb 12 '20 at 7:55
  • $\begingroup$ I'm not the downvoter (and I gave you +1 to compensate!) but I'm guessing it's because your problem is not very clear: first is is text? can you maybe give a couple examples of what your documents and entities look like? What kind of information related to entities? Is there a link between the entities and the categories? $\endgroup$ – Erwan Feb 12 '20 at 15:45
  • $\begingroup$ Yes most certainly , lets take the example of aws ec2 group there is many resource types that you can create under that category. If i take their API documentation or CLI docs just by looking at commands or endpoints with Describe prerix or create prerix you can get a clear idea what they are. With the way i currently parsed their docs i am able to query that easily. Also by usimg their glossary i created a cross reference table when such resource glossary item has a reference to another glossary item and i established that they have some relationship. I am running out of space in the comment $\endgroup$ – DevZer0 Feb 12 '20 at 16:30
  • $\begingroup$ so yes i have manually built a data model with hand written rules for this. But i was wondering that its very well be possible that i missed important details because i didnt parse the user guides or admin guide docs. So i was wondering if i could have used nlp methods like classification or clustering to produce much richer data model from the documentation than i have done with hand written rules. So far on this topic i still realized while trying to build a corpus that i must specify a category for the data i am importing to the corpus. So that make sense. But can i expect ..... $\endgroup$ – DevZer0 Feb 12 '20 at 16:38
  • $\begingroup$ to be able to structure the data in such a way that it makes technical sensese. Or is this domain is for only solving problems related to general speaking language? Can it be also used to build corpus that are specialized topics? Or am i better off using my hand written rules and map the data to a Graph database like neo4j. Did my response gave you enough context? Basically can i expect NLP or ML to figure out how the subject matter should he tagged or labled etc? $\endgroup$ – DevZer0 Feb 12 '20 at 16:43
2
$\begingroup$

Anyways is this problem I am trying to solve fit into the general NLP ML space?

Generally speaking, feeding the source data in bulk to a ML system is unlikely to give the kind of structured output you expect. It's likely that you would have to somehow guide the process in the direction of what you want to obtain, and this might take a lot of time and effort (depending on the requirements).

That being said, there are indeed NLP methods which are meant to extract specific pieces of information from text, and it usually works quite well with domain-specific data (provided it's done correctly, caveats apply). I'm just going to list a few typical tasks which can be done:

  • Named Entity Recognition would be the most common and probably the most simple, since there are many existing libraries. Most libraries use a pre-trained model, but it's likely to give much better results when it's trained on the kind of data it's applied to (of course that usually means manually annotating your own training set).
  • Text classification can be used to automatically assign documents to a category (class) among a set of predefined categories. This is supervised so you would also need a training set containing labelled documents. Again there is a number of algorithms and libraries available.
  • Simple information retrieval methods based on measuring semantic similarity (see e.g here) between terms and documents can be used to search e.g. documents relevant to a term.
  • Topic modeling is an unsupervised approach which groups similar documents together (clustering). Since it's unsupervised it doesn't require any training data, but on the other hand what the algorithm finds as "topics" (groups) is often different than what a human would expect.
  • Extracting relations between concepts (typically between named entities) is a more advanced task which usually requires more work in order to capture the specifics of the job. I'm not aware of any general library for that.

Overall there are many things possible, but the first step would be to try to design the system precisely, typically using some of the existing tasks as building blocks.

Is it fine to include a specific document in multiple categories? For example an instance in AWS can be in category ec2 and a general category Computing unit

Yes it's fine, but if you want a ML classifier to be able to predict several classes you will need to use multi-label classification (the "standard" is single label).

Is the starting point to this is to build a corpus? I tried it out nltk categorized corpus builder.

I'd recommend building a corpus only once you have a clear idea of how you're going to use it. Also it's usually an experimental process with lots of back and forth, so try to progress iteratively rather than starting with strong assumptions/decisions which might later turn out not relevant.

$\endgroup$
  • $\begingroup$ Thank you very much for your detailed explanation and for your time $\endgroup$ – DevZer0 Feb 15 '20 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.