4
$\begingroup$

Given several wikipedia articles on different movies. What are the different approaches to implement a QA system to answer different quires related to movies.

Dataset : Wikipedia articles

Input   : natural language query, eg : who directed terminator ?

Output  : The Terminator is a 1984 American science fiction action film directed by James Cameron

Different approaches available are :

1 IR Based approaches

In IR based approaches the similarities between query and passaeges are considered and best suitable passages are selected as answer (as Google does)

2 Ontology based

In Ontology based prior domain knowledge is required about the domain. And documents need to be mapped accordingly.

3 Machine learning based

In ML based approaches large training set of documents is needed for training.

What are the other cons and pros of each approaches?

Which of these approaches suits the best for this use case ? Also is there any other methods ?

$\endgroup$
  • 1
    $\begingroup$ Mind adding more details? This is a bit broad (and/or) unclear atm :) $\endgroup$ – Dawny33 Jun 17 '16 at 9:54
  • 1
    $\begingroup$ I think this is comparable to visual question answering (VQA), where you pose a (textual) question about an image and the model gives an answer. Sreejithc321 would like to do that but with wikipedia articles about movies instead of images. See github.com/abhshkdz/neural-vqa or iamaaditya.github.io/2016/04/… for examples. Neural networks can be used for that - but you'll need LOTS of questions & answers & movie articles for training. $\endgroup$ – stmax Jun 17 '16 at 10:39
  • $\begingroup$ @Dawny33 Please check now $\endgroup$ – Sreejithc321 Jun 20 '16 at 5:39
  • $\begingroup$ @Sreejithc321 Looks fine now. Thanks for making it clear. [Voted to reopen] :) $\endgroup$ – Dawny33 Jun 20 '16 at 5:43
  • $\begingroup$ I have voted to re-open, but I still think it could do with some refining to make it less broad. I suggest drop the implementation question, and ask it separately once you have made an informed decision about the approach (with choice of approach made and some specific examples of data and goals). $\endgroup$ – Neil Slater Jun 20 '16 at 6:56
3
$\begingroup$

While the actual "best system" depends heavily on a number of factors including your goals and your resources, it is possible to discuss some general pros and cons to each system.

1. IR Based approaches: Information retrieval approaches allow the algorithm to make effective use of what we might call "explicit knowledge" or hardcoded facts. They can work very well when the answer to the query exists in some form already and where the algorithm simply needs to find this answer amongst a set of all the other possible answers. Such systems typically rely on either an explicit dataset of possible answers or facts, or by treating the web as such a dataset and querying it in some way. IR systems are often the only reasonable choice when the answer cannot be generated from abstract principles. For example, if we built a system to answer trivia questions about movie stars, we would need to have explicit knowledge of which features are associated with which movie stars (i.e. Brad Pitt has brown hair and starred in Fight Club.) No machine learning algorithm can be expected to generate this without the knowledge already present (although machine learning systems with access to such knowledge may serve as effective information retrieval systems.) Due to the external nature of the data with respect to the algorithm, IR systems tend to be very effective for large and rapidly changing data sets, which would otherwise require retraining of the machine learning model or manual reconfiguration of the ontology to handle. It is also worth noting that more complex information retrieval systems may include other systems such as NLP engines to translate the data (ex. natural language text) into a form usable by the IR system.

2. Ontology based approaches: Ontology methods take the most time to implement and are usually the most finicky. They require the iterative hypothesizing of a structured representation of the domain, testing of the representation, and modification of the representation based on discovered flaws. For complex real-world domains with many possible contingencies, this process can take a very long time and be quite frustrating. Because everything is hard-coded, even the best ontology-based systems display a relative degree of inflexibility compared to their machine learning and IR counterparts. Ontologies can be effective, however, when you do not have access to a data set for information retrieval or machine learning training, as the domain knowledge are hard-coded into ontology systems by hand. They can also be useful when the domain is relatively well-structured and where it is important to understand and plan every decision made by the system such as in automated attendant phone systems.

3. Machine learning based approaches: As machine learning approaches extract patterns from data, they are relatively robust to subtleties present in complex domains that were not explicitly built into an algorithm, model, or ontology. They work best when a prediction needs to be made about which "class" or category a given input (or query in your case) belongs to, or when a quantitative prediction needs to be produced from data. For example, if I have a large collection of animal pictured labeled with the type of animal present in these images, I can train a machine learning model to predict which animal type is present in future images, as long as the model has seen enough pictures of this animal that it can extract the patterns in the image that correspond to the image likely representing a lion or a tiger or a bear, etc. This works because there are properties in the image itself that predict which animal it represents and these patterns can be extracted from labeled data (there are also machine learning algorithms that pick out patterns in unlabeled data called unsupervised algorithms, but these are not relevant for your domain of interest.) Traditionally, machine learning systems are relatively poor at modeling domains where the answer depends not on recognizing a pattern, but having access to knowledge like in the movie stars example above. Machine learning systems are also great because they require relatively little manpower to implement. As you mentioned, they also require access to a dataset, which may need to be quite large depending on the complexity of the domain.

We can roughly summarize the advantages and disadvantages by ranking the methods according to a few criteria: enter image description here

All of the best systems (ex: IBM Watson) use a hybrid approach, taking advantage of each method for its relative strengths and substituting other methods to address their weaknesses. Depending on the performance you want, a QA system can be built by a single person with some knowledge of any of the above, or may require a team of upwards of 100 engineers.

$\endgroup$
  • $\begingroup$ Without training data (pairs of questions and their answers) you would need to work with unsupervised machine learning algorithms, which extract patterns in the data themselves. There aren't any out of the box algorithms for using these for question answering, but there are various approaches you might try including deep natural language processing techniques (which often require specialized knowledge to implement) and topic modeling techniques such as latent dirichlet allocation. Neither of these will get you all of the way there, but they will provide a basic language model that you can use. $\endgroup$ – Larry Muhlstein Jun 22 '16 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.