MARGOT: A web server for argumentation mining

Fri, 01 Jan 2016 00:00:00 +0000

Research setting

Argumentation mining is the automatic extraction of arguments from unstructured text, such as news, web pages, or social media. An argument has a claim, the conclusion that someone defends, and evidence, the premises that support it. A sentence is argumentative when it contains at least one claim or one piece of evidence. The boundaries of a claim or of a piece of evidence are its first and last words inside the sentence.

They also argue that economic inequality contributes to crime.Claim Dr. Gary Kleck, a criminologist at Florida State University, estimated that approximately 2.5 million people used their gun in self-defense or to prevent crime each year, often by merely displaying a weaponEvidence

Two sentences from the IBM corpus, the training data of MARGOT. The first contains a claim; the second is expert evidence, a testimony by someone with authority on the topic (Section 3 of the paper).

Motivation

Argumentation mining could serve policy-making, recommender systems, and the legal domain, but in 2016 no tool for it was open to people outside the research field. Existing methods solved single sub-tasks for single genres, and they relied on hand-made features tied to one genre. Methods trained on the IBM corpus also needed the topic of the debate in advance, because that corpus labels a claim only when it is relevant to a given topic.

System	Open to non-experts	No topic needed	Claims and evidence	Component boundaries	No genre-specific features
Context-dependent detection (Levy et al., 2014; Rinott et al., 2015)	no	no	yes	Claims only	no
Cross-genre mining with SVM-HMM (Habernal & Gurevych, 2015)	no	Not discussed	yes	yes	no
Context-independent claim detection (Lippi & Torroni, 2015)	no	yes	Claims only	no	yes
This paper (MARGOT) This work	yes	yes	yes	yes	yes

How MARGOT relates to the methods the paper discusses (Sections 1 to 3 of the paper).

Objective. Build MARGOT (Mining ARGuments frOm Text), a web server that finds claims, evidence, and their boundaries in any English text with no topic given, and test how it behaves across genres.

Approach

MARGOT is a pipeline of three stages. The Stanford parser first splits the text into sentences and builds the constituency parse tree of each one, a tree of its grammatical phrases. Two classifiers then score each sentence, one for claims and one for evidence. Last, for each argumentative sentence, a sequence labeler marks which words belong to the claim or to the evidence. MARGOT was trained on the IBM corpus: 547 Wikipedia articles on 58 topics, with 2,294 claims and 4,690 pieces of evidence.

Input text Plain text pasted by the user.
Parsing Sentences and their parse trees.
Sentence scoring Tree kernel classifiers score each sentence.
Word labels SVM-HMM marks claim and evidence words.
Output page HTML page with highlights.

The MARGOT pipeline, from a plain text query to an HTML page with the arguments highlighted (Figure 1 and Section 4 of the paper).

The classifiers are support vector machines (SVMs) with a tree kernel, a similarity that counts the fragments two parse trees share. They also use bag-of-words, the counts of the words in the sentence. The idea is that claims often share a structure, such as a clause after a verb like “argue” or “believe”, so no genre-specific features are needed. The boundary stage uses SVM-HMM, a sequence labeler that combines SVMs with hidden Markov models, over words, part-of-speech tags, lemmas, and named entities.

Results

On the IBM corpus, each of 39 topics is held out in turn as the test set, and scores are averaged over these topics. Precision is the share of predicted sentences that are correct, and recall is the share of true sentences that are found. Sentence detection reports F1, the harmonic mean of precision and recall, and AUROC, the probability that a positive sentence ranks above a negative one (1 is perfect). Boundary detection reports HT, the share of true claims that the prediction overlaps by at least one word. The paper then runs MARGOT on new Wikipedia pages, New York Times articles, and Reddit threads. There it counts the share of sentences with a claim (%C) and with evidence (%E).

0.816AUROC for claim sentence detection on the IBM corpus
85.9%of true claims hit by the predicted boundaries (HT)
34.2% vs 10.0%of sentences with a claim in controversial and non-controversial Wikipedia pages

Method	Claims, F1	Claims, AUROC	Evidence, F1	Evidence, AUROC
Random baseline	3.1	–	4.9	–
Bag-of-words SVM	16.9	0.805	13.9	0.671
Tree kernel (SSTK)	16.6	0.809	16.1	0.718
Tree kernel + bag-of-words (MARGOT)	17.5	0.816	16.7	0.724

Argumentative sentence detection on the IBM corpus, averaged over 39 held-out topics (Tables 1 and 2 of the paper). Less than 5% of the sentences are positive.

F1 stays low because fewer than 5% of the sentences are positive and MARGOT does not know the topic: many of its false positives are real claims that the IBM annotators skipped as off-topic. The tree kernel with bag-of-words gives the best F1 and AUROC for both claims and evidence. On boundaries, MARGOT reaches an F1 of 66.6 for claims and 90.7 for evidence (Table 4).

Controversial Wikipedia pages hold far more claims than neutral ones, and New York Times articles sit in between, with 22.4% of sentences holding a claim (Tables 5 and 6). On Reddit, the climate shift thread is richer in arguments than the New Hampshire primaries thread, with an evidence share (%E) of 0.88 against 0.48.

Reddit comment trees colored by the MARGOT prediction: blue for claim, red for evidence, yellow for both, gray for none (Figure 6 of the paper, post-print, CC BY).

Takeaways