Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises

Sun, 01 Dec 2024 00:00:00 +0000

Research setting

Language teachers use multiple-choice cloze (MCC) exercises to train grammar. An MCC exercise has a body, a sentence with a gap; a key, the correct answer for the gap; and distractors, plausible but incorrect alternatives. Good distractors are homogeneous (same syntactic category as the key), plausible, and unambiguous (none of them could replace the key). Writing many such exercises by hand takes time and expert knowledge, so the paper asks a large language model (LLM) to write them.

Thanks to high technology, doctors can betterKey assess patients’ conditions.

An MCC exercise on comparisons from the fine-tuning dataset (Listing 2 of the paper), shown with its key filled in. The learner sees a gap in place of the key and chooses among better, best, good, and well; the last three are the distractors.

Motivation

Most earlier systems take sentences from existing texts and place the gap with fixed rules, such as the first verb or an adjective. Systems that write new sentences still constrain the model, for example with a fixed part-of-speech (POS) sequence, and their output tends to lack creativity. The only English MCC dataset, SC-Ques, has limited diversity, underrepresented topics, and frequent mistakes. Lastly, the field has no agreed automatic metric for the quality of a generated exercise.

Approach	Writes its own sentences	No fixed gap rule or POS sequence
Rule-based pipelines (Sumita et al.; Lin et al.)	no	no
Gap detection with conditional random fields (Goto et al.)	no	yes
Neural gap prediction (Bitew et al.; Matsumori et al.)	no	yes
Generation from a POS sequence, keyword, and topic (Chomphooyod et al.)	yes	no
This paper This work	yes	yes

How the paper relates to the prior work it discusses (Sections 1 and 3).

Objective. Find out how well an LLM can write accurate English grammar MCC exercises without predefined constraints or POS sequences, and measure their quality with an automatic metric checked against a human expert.

Approach

The base Llama 3 model, given a detailed prompt, struggled with some grammar topics and wrote poor distractors. The authors therefore fine-tuned it on a cleaned dataset of exercises, built from SC-Ques in the steps below.

SC-Ques About 300k English sentence completion exercises written by teachers.
Cleaning Drop items with several gaps, fewer than two distractors, or noise.
Topics A rule-based Pattern Matcher links each key to one of 19 grammar topics.
Filtering Remove similar exercises per topic and cap how often one key appears.
Balancing Add synthetic WH-question exercises written with GPT-4.

Building the fine-tuning dataset (Section 5 of the paper). After the removal of similar exercises, 30% of the original data was left.

Each training example pairs a grammar topic and a number of distractors with an exercise in JSON: the full sentence, the sentence with the gap, the key, and the distractors. Two metrics then score the output. Structural compliance (SC) is a rule-based check of the exercise form. The gap must sit in the intended place, the key must have the right grammatical form, and every distractor must be homogeneous with the key. Self-BLEU measures how much the exercises of one topic repeat each other’s word sequences (2 to 5 words long); lower means more diverse.

1. Fine-tune

Grammar topic Data
Llama 3 weights Frozen
LoRA adapters Trained
Exercise Data
Structural check

Llama 3 8B Instruct is quantized to 4-bit precision and kept frozen. Small low-rank (LoRA) adapters on its attention layers learn to write the exercise for a given topic, for 3 epochs, in two hours on one GPU.

2. Generate

Grammar topic Data
Llama 3 weights Frozen
LoRA adapters Frozen
Exercise Data
Structural check

For each of the 19 topics, the model writes 50 exercises with one distractor each, sampling at temperature 0.7.

3. Check

Grammar topic
Llama 3 weights
LoRA adapters
Exercise Data
Structural check Frozen

The structural check tests the gap and key (pertinence) and the distractor (homogeneity). For inflectional topics, the distractor must share the key’s lemma; for free morpheme topics, whose options form a small closed set, it must come from a list of admitted words.

Trained with gradients Kept fixed Data

Training, generation, and checking (Sections 6 to 8 of the paper).

Results

The 950 generated exercises were scored with the automatic metrics and reviewed by a computational linguist with a background in language teaching. The expert judged each exercise correct (EC) only if it was plausible, unambiguous, coherent with common sense, and acceptable (free of stereotypes and inappropriate content). The expert also judged structural compliance, which serves as the gold reference for the automatic check (SC_A).

79%of the 950 exercises judged ready for learners by the expert
95%F1 of the automatic structural check against the expert (precision 98%, recall 91%)
0.07average self-BLEU, a sign of low repetition

Exercise correctness (EC) per grammar topic (higher is better)

articles 0.74
comparison adjectives 0.72
conditional statements 0.66
future simple 0.90
modal verbs 0.70
infinitive and gerund verbs 0.86
passive tenses 0.74
past continuous 0.88
past perfect 0.82
past simple 0.82
personal pronouns 0.74
possessive adjectives 0.72
prepositions 0.72
present continuous 0.88
present perfect 0.84
present simple 0.86
quantifiers 0.84
relative clauses 0.74
WH- question 0.90

Share of exercises judged correct by the expert, per grammar topic. Source: Table 1 of the paper (50 exercises per topic).

Show as table

System	Exercise correctness (EC) per grammar topic
articles	0.74
comparison adjectives	0.72
conditional statements	0.66
future simple	0.90
modal verbs	0.70
infinitive and gerund verbs	0.86
passive tenses	0.74
past continuous	0.88
past perfect	0.82
past simple	0.82
personal pronouns	0.74
possessive adjectives	0.72
prepositions	0.72
present continuous	0.88
present perfect	0.84
present simple	0.86
quantifiers	0.84
relative clauses	0.74
WH- question	0.90

Average over the 19 topics	Value
Structural compliance, automatic check (SC_A)	0.85
Structural compliance, expert (SC_H)	0.92
Self-BLEU (lower is better)	0.07
Exercise correctness, expert (EC)	0.79
Wrong exercises that break common sense	0.75
Wrong exercises with an ambiguous distractor	0.19
Wrong exercises that are not acceptable	0.05
Wrong exercises that are not plausible	0.01

Averages from Table 1 (first four rows) and from the error analysis on wrong exercises in Table 2 of the paper.

Most generated exercises are well formed and varied, and the automatic check agrees closely with the expert. By the structural scores, the model does better on free morpheme topics, where keys and distractors come from a small set of words. On verb tense topics, the Pattern Matcher sometimes mislabels verbs, so the check can wrongly reject an exercise. Among the wrong exercises, a sentence that makes no sense is the most frequent error, and an ambiguous distractor is the second. Biased or trivial exercises are almost absent.

Takeaways