AlmaSDG: A Dataset of Scientific Articles' Contributions to the UN Sustainable Development Goals

Research setting

The UN Sustainable Development Goals (SDGs) are 17 goals adopted in 2015 by 193 UN states, such as no poverty, gender equality, and climate action. Universities and research institutions want to know which of their articles contribute to which goal. The paper frames this as multi-label document classification. The input is the title, abstract, and keywords of a scientific article. The output is one yes or no label for each of the 17 SDGs. An article can contribute to several goals at once, or to none.

Motivation

Tools that map research to the SDGs already exist, but there is no gold standard to evaluate how accurate they are. There is also no shared definition of what it means for an article to contribute to an SDG. Existing SDG datasets are labeled automatically by queries, by crowd workers who see one goal at a time, or by the authors of the documents themselves. Single-label data can miss an article’s contribution to other goals and count it as a negative.

Objective. Define annotation guidelines that say when a scientific article contributes to an SDG, use them to build a dataset labeled by multiple annotators on all 17 SDGs, and validate it with agreement measures and existing models and tools.

Approach

AlmaSDG is the result of a 12-month joint effort of stakeholders, academic experts in SDGs, and computer scientists. A first pilot showed that the official SDG targets are too specific and too open to interpretation to serve as annotation instructions. With five academic experts in sustainability, the team wrote new guidelines. A direct contribution studies, develops, or applies methods and technologies. An indirect contribution lays the groundwork for them, for example with a review or new data. A cursory mention of an SDG topic is not a contribution. The guidelines list 165 contribution areas across the 17 SDGs, with positive and negative examples.

The source is a publication repository of the University of Bologna, chosen because large databases cover the humanities less than the sciences. A random sample would contain mostly negative articles. Instead, the team queried the repository with fragments of the SDG descriptions and kept 139 articles that look relevant, trying to cover all goals. Eight NLP researchers formed four pairs, and each pair labeled all 139 articles on its own subset of 4 or 5 SDGs. Each annotator worked independently, and each pair then discussed its disagreements to fix the final labels.

Results

The paper validates the dataset in two ways. First, it measures inter-annotator agreement for each SDG with Cohen’s kappa, where 1 means perfect agreement and 0 means agreement at chance level. Second, it prompts three LLMs (GPT, Gemini, Llama) zero-shot with the guidelines, and it runs three existing SDG classifiers (Aurora, Elsevier, OSDG). The metrics are precision, recall, and F1 score, macro-averaged over the 17 SDGs.

• 2,363labels over 139 articles and 17 SDGs
• 0.54average Cohen’s kappa, moderate agreement
• 0.68best macro F1, GPT

The dataset is sparse: on average 14.24 articles are positive for an SDG, from 5 for SDG 1 to 32 for SDG 10 (Table 3 of the paper). Only 13 of the 139 articles contribute to no SDG, and about half contribute to several (Table 4 of the paper).

Agreement depends on the goal, even within one pair. SDG 5 (gender equality) reaches 0.86, “almost perfect agreement”, because gender equality is often stated explicitly as a goal of the article. SDG 8 (decent work and economic growth) and SDG 16 (peace, justice and strong institutions) reach only 0.23 and 0.24, “fair agreement”, because their scope is broad.

Zero-shot evaluation on AlmaSDG (Table 6 of the paper, rows reordered). GPT, Gemini, and Llama are prompted with the guidelines; the other rows are existing SDG classifiers. Elsevier-S keeps one label, and Elsevier-M keeps every SDG with probability above 0.1. Best value in bold, as in the paper. The paper reports no statistical significance test.

GPT has the best macro F1 (0.68), followed by Gemini (0.63), while Llama is the weakest LLM (0.48). Among the existing tools, Elsevier-M is best (0.54) and Aurora is lowest (0.35). Per SDG, Gemini reaches 0.96 F1 on SDG 5 (Table 7 of the paper). GPT is best on SDG 17 with 0.60, while Aurora, Elsevier, and OSDG all score 0.00 there. Models often score lower where humans disagree more, such as on SDGs 8 and 9. On SDG 5, the goal with the highest agreement, four of the seven systems reach at least 0.80 F1.

Takeaways

1. Guidelines define what a contribution is.AlmaSDG comes with guidelines that separate direct and indirect contributions and list 165 contribution areas, so every label rests on a written criterion.
2. Some goals are harder to judge.Agreement ranges from 0.23 on SDG 8 to 0.86 on SDG 5, with an average Cohen’s kappa of 0.54.
3. A benchmark for SDG tools.On the 139 articles, a guideline-based GPT prompt reaches 0.68 macro F1 and the best existing tool 0.54, and the existing tools score 0.00 on SDG 17.

@article{Bolognini_2026,
  title = {AlmaSDG: A Dataset of Scientific Articles’ Contributions to the UN Sustainable Development Goals},
  volume = {13},
  ISSN = {2052-4463},
  url = {https://doi.org/10.1038/s41597-026-07452-4},
  DOI = {10.1038/s41597-026-07452-4},
  number = {1},
  journal = {Scientific Data},
  publisher = {Springer Science and Business Media LLC},
  author = {Bolognini, Luca and Palmieri, Elena and Donati, Nicolò and Grundler, Giulia and Pappacoda, Gianmarco and Ruggeri, Federico and Galassi, Andrea and Torroni, Paolo},
  year = {2026},
  month = {June}
}

• Dataset
• Sustainable Development Goals
• Scientific Papers
• Annotation Guidelines
• Inter-Annotator Agreement