What is the NLP problem I am solving called and how should i go about solving it?

Question

I am working on a POC where I am required to write a NLP code after web scraping. The prompt to my code is

How good is the online Data Science degree offered by MIT?

I am required to do web scrapping and other information resources and generate a chatGPT type output. I am new to data science and understand that this is not a regression of classification task. I want to know what is this problem called so that I can do the relevant study and solve this problem.

Also, how should I go about solving this problem, like a block diagram is highly appreciated.

Answer 1

This problem is a combination of several tasks in Natural Language Processing (NLP) and Information Retrieval (IR). The tasks involved are:

Web Scraping: Extracting data from websites. This is not an NLP task per se, but it's a common step in many NLP projects where the data comes from the web.

Information Extraction: This is the process of automatically extracting structured information from unstructured and/or semi-structured machine-readable documents.

Sentiment Analysis: This is the interpretation and classification of emotions (positive, negative and neutral) within text data using text analysis techniques. Sentiment analysis allows organizations to identify public sentiment towards certain words or topics.

Text Summarization: This is the process of shortening a set of data computationally, to create a subset that represents the most important or relevant information within the original content.

Chatbot Development: This involves creating a chatbot (like GPT-3) that can generate human-like text based on the information extracted and summarized from the web.

Now to solve this, I am thinking of this approach that you might change whenever you want:

Web Scraping: Use a library like Beautiful Soup or Scrapy in Python to scrape information about MIT's online Data Science degree from various websites (like MIT's official website, online forums, review sites, etc.).

Data Cleaning: Clean the scraped data to remove any irrelevant information, HTML tags, etc.

Information Extraction: Extract relevant information about the degree like its curriculum, duration, cost, etc. You might need to use techniques like Named Entity Recognition (NER) for this.

Sentiment Analysis: Analyze the sentiment of reviews about the degree to understand public opinion about it. You can use libraries like NLTK or TextBlob in Python for this.

Text Summarization: Summarize the extracted information and sentiment analysis results into a concise summary. You can use techniques like extractive or abstractive summarization for this.

Chatbot Development: Use the summarized information to generate responses in a chatbot. You can use a pre-trained model like GPT-3 for this, or you can train your own model if you have enough data.

But, each of these steps is a big task in itself.