Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification

Tue, 01 Jul 2025 00:00:00 +0000

Research setting

Claim verification decides whether a claim is true, given some evidence. A claim is a statement that has to be checked, such as a rumour circulating on social media. The evidence is a piece of text, here taken from Wikipedia or from news articles, that can confirm or contradict the claim. The verdict, called the veracity, is either supported or refuted, and a rationale is a short text that explains how the evidence leads to the verdict.

Top: an annotator reads a claim with its evidence and veracity, writes a rationale, and names the reasoning type. Bottom: a large language model (LLM) reads the claim and the evidence and returns a veracity, optionally with a rationale (Figure A2 of the paper).

The paper names two ways to reach a verdict. In deduction, the conclusion follows directly from the evidence, which states what the claim says or contradicts it. In abduction, the evidence is partial, so the verdict rests on the most plausible of several hypotheses. That hypothesis could be wrong.

Motivation

LLMs score well on reasoning tests in mathematics and coding, but whether they reason in other settings is still an open question. Fact-checking is a high-stakes setting where this question matters, because a verifier must give both a correct verdict and a sound rationale. The most prominent reasoning datasets also leave implicit which kind of reasoning each problem needs. No claim verification benchmark labeled each claim with the reasoning type required to verify it.

Work	Real-world claims and evidence	Evidence given with the claim	Reasoning mode not named in the prompt	Framework of atomic reasoning types
Claims only (Hu et al., 2023b)	Not stated	no	Not stated	Not stated
Fact verification benchmarks (Aly et al., 2023; Tang et al., 2024; Strong et al., 2024)	no	Not stated	Not stated	no
Logical reasoning evaluation (Xu et al., 2025)	no	Not stated	no	no
This paper (RECV) This work	yes	yes	yes	yes

How RECV relates to the claim verification work discussed in the paper (Section 9). Not stated means the paper does not say.

Objective. Measure how well LLMs verify real-world claims when the evidence calls for deductive or for abductive reasoning, and check whether asking them for a rationale helps.

Approach

The paper first defines a framework with three layers. The reasoning task is the overall problem, the reasoning process is one step or several, and an atomic reasoning type is the inference used at each step. A preliminary study on 90 hand-picked claims found that every claim needed either deduction or abduction, so RECV focuses on these two types. The authors then sampled 500 claim-evidence pairs from each of three existing datasets, using text-similarity thresholds so that abductive pairs would not be drowned out. Nine PhD students in Computer Science, three per dataset, labeled each pair as deductive or abductive.

Layers A framework of task, process, and atomic reasoning type.
Study A preliminary study of 90 claims finds only deduction and abduction.
Sampling Similarity thresholds pick 500 pairs per dataset.
Labels Three annotators per dataset label the reasoning type.
LLM evaluation Three LLMs predict the veracity, with or without a rationale.

How RECV was built and used (Sections 4 to 6 of the paper).

Dataset	Claims and evidence from	Pairs	Abductive pairs
VitaminC	Wikipedia	500	29 (5.8%)
CLIMATE-FEVER	Online claims and Wikipedia	500	102 (20.4%)
PHEMEPlus	Social media rumours and news articles	500	36 (7.2%)

The three RECV datasets, in order of increasing complexity. Source: Table 1 and Section 5 of the paper.

Three proprietary LLMs are tested: Claude V3 Sonnet, GPT-4, and GPT-4o. In the No-Exp setting, a model gives only the veracity; in the Exp setting, it first writes a rationale and then gives the veracity. Both settings use zero-shot prompts (ZS), without worked examples. Both also use manual chain-of-thought prompts (M-CoT), with worked step-by-step examples. The Exp setting adds zero-shot chain-of-thought prompts (ZS CoT).

Results

The paper scores veracity prediction with macro-F1, from 0 to 1. It also reports the error rate: the percentage of deductive, or of abductive, pairs whose veracity the model gets wrong. It also compares the generated rationales with human ones on 100 pairs per dataset.

48.58%average error rate on abductive pairs of CLIMATE-FEVER, against 15.58% on deductive ones
32%average error rate on abductive pairs of VitaminC, the simplest dataset, against 10.31% on deductive ones
14.76%average performance drop on abductive pairs of CLIMATE-FEVER when models write a rationale first

Error rate (%) by dataset and reasoning type (lower is better)

VitaminC, deductive 10.31%
VitaminC, abductive 32%
CLIMATE-FEVER, deductive 15.58%
CLIMATE-FEVER, abductive 48.58%
PHEMEPlus, deductive 20.06%
PHEMEPlus, abductive 44.68%

Average error rate across all models and prompt settings. Source: Section 6.1 of the paper.

Show as table

System	Error rate (%) by dataset and reasoning type
VitaminC, deductive	10.31%
VitaminC, abductive	32%
CLIMATE-FEVER, deductive	15.58%
CLIMATE-FEVER, abductive	48.58%
PHEMEPlus, deductive	20.06%
PHEMEPlus, abductive	44.68%

Model and setting	VitaminC F1	CLIMATE-FEVER F1	PHEMEPlus F1
Claude ZS No-Exp	0.85	0.80	0.73
Claude ZS Exp	0.89	0.74	0.74
GPT-4 ZS No-Exp	0.86	0.87	0.69
GPT-4 ZS Exp	0.88	0.78	0.73
GPT-4o ZS No-Exp	0.88	0.84	0.72
GPT-4o ZS Exp	0.89	0.79	0.74

Veracity macro-F1 with zero-shot prompts, without (No-Exp) and with (Exp) a rationale. Source: Table 2 of the paper.

On every dataset, abductive pairs are much harder than deductive ones: the gap is about three times on average. Writing a rationale first raises macro-F1 on VitaminC but lowers it on CLIMATE-FEVER, so its effect depends on the dataset and the model. Against human rationales, the generated ones score from 0.81 to 0.94 on evidence appropriateness and factual consistency. They match human rationales better when the verdict is correct. For abductive pairs, models tend to write firm assertions instead of hedged ones, and they often attend to only part of the claim or of the evidence.

Takeaways