New Trends in MRC II

Multiple-Passage MRC

  • In MRC tasks, relevant passages are pre-identified, which contradicts how humans tend to approach questions and answering

  • Multi-passage datasets include MS MARCO, TriviaQA, SearchQA, DuReader, and QUASAR. The given context in multi-passage is a collection of documents rather than a single passage

  • The unique features of multi-passage MRC that makes it challenging are:
    • Massive Document Corpus
      • With each question link to multiple documents, the final performance of the MRC models is dependent on retrieving relevant documents accurately and quickly

    • Noisy Document Retrieval
      • Model may retrieve noisy document that coincidentally contains the correct answer span. This noise will affect the supervised ML task

    • No Answer
      • The retrieval component is key. If it doesn’t perform well, there will be no answers in the document. If the answer extraction module doesn’t include answer verification, then it will output the wrong answer

    • Multiple Answers
      • A question might have multiple answers but the correct answer is dependent on the context

    • Evidence Aggregation
      • Snippets of evidence can appear in different parts of the documents. This means that to form the correct answer, the model needs to aggregate all the evidence. More documents mean mode information and complexity

  • Methods to tackle multi-passage MRC task
    • “Retrieve then read” method
      • Retrieval component will retrieve several relevant documents

      • The reader will then use these documents to output an answer

      • DrQA by Chen et al.
        • Use TFIDF to select 5 relevant articles from Wikipedia for each question in SQuAD to narrow search space

        • For the reader module, they use richer word representations and a pointer module to predict the begin and end positions of answer spans

        • Both retrieval and reading are performed separately but retrieval error can easily affect the performance of the reader

      • To alleviate error propagation from poor document retrieval
        • Introduce ranker component
          • Reranks documents retrieved by the search engine

          • Htut et al. introduce two rankers
            • InferSent
              • Use FFNN to measure general semantic similarity between the context and the question

            • Relation-Networks Rankers
              • Use relation-networks to capture local interactions between context words and question words

          • Lee et al. proposed a Paragraph Ranker mechanism
            • Uses bi-directional LSTMs to compute representations of passages and questions and measures similarities using dot product to score each passage

        • Jointly train the retrieval and reading processes
          • Wang et al. proposed Reinforced Ranker-Reader (R3)
            • Use match-LSTM to compute similarity between the question and each passage to obtain document representations. These are fed into the ranker and the reader

            • Reinforcement learning is used to select most relevant passage while function of the reader is to predict answer span from selected passage. Both tasks are trained jointly

      • The computational complexity increases as the document corpus becomes larger

    • Das et al. propose an efficient retrieval method that represents passages independent from questions and stores outputs offline
      • When given the question, model measures similarities between passages and questions and feed top-ranked passages to the reader for answer extraction

      • Uses a GRU to recompute the query representations to take into account the state of the reader and the original query. The new query can be used to retrieve other relevant passages (mimic the reread process)

    • More than one possible answer
      • Pang et al. introduced three heuristic methods to avoid the model just selecting the first match span as answer
        • The RAND operation
          • Treats all answer spans equally and chooses one randomly

        • The Max operation
          • Chooses the one with maximum probability

        • The SUM operation
          • Assumes that there are more than one ground-truth span and sums all span probabilities together

      • Lin et al. introduced a fast paragraph selector to filter out passages with wrong answer labels
        • Use MLP / RNNs to obtain hidden states of passages and questions. Self attention is applied to the questions to illustrate the different importance

        • Compute similarity between passages and questions and feed the most similar ones to the reader

    • Evidence aggregation
      • Wang et al. propose strength-based and coverage-based re-rankers to make full use of multiple pieces of evidence
        • Strength-based
          • The answer with the most occurrences is chosen to be the correct answer

        • Coverage-based re-rankers
          • Concatenates all passages that contain candidate answers as a new context and feeds that to the reader to obtain the answer, which aggregates different evidence

Ryan

Ryan

Data Scientist

Leave a Reply