Abstention and Citation in Retrieval-Augmented Assistants: A Reference for Association Content Teams
Abstention is a RAG assistant's decision not to answer when retrieved documents do not support an answer. A reference for association content teams.
Last updated: October 20, 2026
Abstention is a retrieval-augmented assistant's decision not to answer when the documents it retrieved do not support an answer. A grounded citation is a link from a statement in the answer to the specific passage, in a named document, that supports it. The two are the same check run in opposite directions: if no passage can be cited, the assistant should abstain.
This reference is for the people who own an association's content library: the standards staff, the certification team, the journal editors and the member-services lead who will answer for the assistant when a member quotes it back to them. It defines the terms, shows where abstention is decided inside a retrieval-augmented generation (RAG) pipeline, compares citation formats, and works one example on a certification handbook.
30-second summary: abstention is a RAG assistant's decision not to answer when no retrieved passage supports an answer; if no passage can be cited, it should abstain. Sources: OpenAI (September 2025) and Glean's citation documentation.
Terms used in this reference
| Term | Definition |
|---|---|
| Retrieval-augmented generation (RAG) | A pattern that retrieves relevant content from your own data, adds it to the model's input as grounding data, and generates a response that can cite that content (Microsoft Learn: RAG in Azure AI Foundry). |
| Abstention | The model declining to provide an answer. A 2025 survey in Transactions of the Association for Computational Linguistics defines it as "the refusal of large language models (LLMs) to provide an answer" and examines it from three perspectives: the query, the model and human values (Wen et al., 2025). |
| Grounded answer | An answer whose statements are all supported by the retrieved passages. Microsoft's groundedness evaluator measures "how well the generated response aligns with the given context without fabricating content" (Microsoft Learn, RAG evaluators). |
| Passage citation | A citation that points inside a document. Amazon Bedrock's Citation object, for example, carries a location (character positions, page numbers or chunk identifiers), the source, the cited source content and the document title (AWS API reference). |
| Hallucination | OpenAI's definition: "plausible but false statements generated by language models" (OpenAI, September 2025). |
Why language models guess instead of abstaining
OpenAI's September 2025 research summary argues that "language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty." Its analogy is a multiple-choice test: a blank answer scores zero, a guess sometimes scores, so a model graded only on accuracy learns to guess. The same summary sorts answers into three categories, accurate responses, errors and abstentions, and states that "errors are worse than abstentions" (OpenAI).
Two older papers explain why retrieval changes the picture. Kadavath et al. (2022) trained models to predict "P(IK)," the probability that the model knows the answer, and found that those predictions "increase appropriately in the presence of relevant source materials in the context" (arXiv:2207.05221). Cole et al. (2023) found that the most reliable signal for deciding when to abstain was "quantifying repetition within sampled model outputs," rather than the model's likelihood or self-verification (arXiv:2305.14613).
For an association, the practical reading is simple. A general chatbot answering from memory has nothing to check its guess against. A RAG assistant does: the retrieved passages. Abstention on an association assistant is mostly a retrieval and evaluation decision, not a model-personality setting.
Where abstention is decided in a RAG pipeline
Abstention can be triggered at four points. Each one fails differently, so each needs its own test question.
| Stage | Trigger | What the assistant should do | Association example |
|---|---|---|---|
| Retrieval | No passage clears the relevance threshold | Say the library does not cover it; offer the right contact | "What is the pass rate for the exam?" when no pass-rate report is loaded |
| Retrieval, access | The relevant passage exists but sits in a collection the user may not read | Say it is member-only content and where to sign in; never quote it | A non-member asks about a member-only practice guide |
| Generation | Passages are on topic but do not contain the answer | Answer only the supported part and say which part is not covered | The handbook explains the audit process but not the appeal deadline |
| Scope policy | The question asks for advice the association does not give | Decline the advice, cite the policy text, point to a professional | "Should I sign this contract?" asked of a bar association assistant |
| Version | Only a superseded edition answers the question | Name the edition and say it is no longer in force | A 2022 standard answers, but the 2026 edition replaced it |
The access row matters most for associations. Glean's documentation states the rule plainly: "Citations never grant new access. You can only open items you already have permission to view" (Glean citation documentation). An assistant that quotes a member-only passage to a non-member has leaked it, whether or not the link works.
Citation formats compared
| Format | What the reader sees | How much work it is to verify | Fits |
|---|---|---|---|
| Document link | A list of source documents under the answer | High: the reader has to find the passage in a 120-page PDF | Short web pages, FAQs |
| Passage or chunk citation | An inline marker that opens the quoted passage | Low: the passage is shown | Handbooks, standards, bylaws |
| Deep link with highlight | The source opens at the cited passage, highlighted | Lowest | Long PDFs, journals, course transcripts |
| Page and edition reference | "Handbook, 2026 edition, p. 14" | Medium, but it survives printing and email | Anything members quote to each other |
Glean describes the deep-link format: "When deep linked citations are enabled, you are directed to the exact passage in the source that supports the statement, with the referenced text so that you can verify it." Its documentation also lists when citations are not included, for example when the assistant "didn't invoke a retrieval tool" and answered from pre-trained knowledge (Glean's citations guide). That second point is the one to test: an answer with no citation on an association assistant should be treated as an abstention failure.
Worked example: five questions against one certification handbook
The handbook below is illustrative: a credentialing body's recertification handbook, loaded as the only source for a member-facing assistant. Each row shows the behavior to expect and how to score it.
| Member question | Does the handbook answer it? | Expected behavior | Expected citation | Score as |
|---|---|---|---|---|
| How many continuing-education hours do I need to recertify? | Yes | Answer with the number and the cycle length | The recertification requirements section, with edition | Accurate or error |
| Can ethics hours count toward the general requirement? | Partly: it lists ethics hours but is silent on carry-over | Answer the listed part, say carry-over is not covered | The ethics-hours passage | Accurate if the gap is stated; error if the gap is filled |
| What percentage of candidates pass the exam? | No | Abstain and point to the credentialing staff | None | Abstention (correct) or error |
| Will my employer have to pay my renewal fee? | Out of scope | Decline; the handbook covers the credential, not employment terms | The fee section, to show what it does cover | Abstention (correct) or error |
| What did the audit rule say before this edition? | Only in the archived edition | Answer from the archived edition and say it is superseded | The archived edition, named | Accurate if labeled; error if presented as current |
The scoring column follows the three categories in OpenAI's summary. Count accurate answers, errors and abstentions separately. A test set needs unanswerable questions on purpose, because an assistant that has only ever been tested on answerable questions has never been tested on abstention at all.
How to test abstention and citation before members do
- Write the question set from the library, not from the demo. For each collection, write answerable questions with the passage that answers them, and unanswerable questions that sound like they belong. The member-services inbox is the best source of both.
- Score precision and recall separately. Microsoft's evaluators separate groundedness, which "focuses on the precision aspect of the response," from response completeness, which "focuses on the recall aspect" (Microsoft Learn: RAG evaluators). An assistant can be fully grounded and still miss half the answer.
- Re-run the set after every library change. A new edition, a re-chunked PDF or a new collection changes what retrieval returns.
Writing the "I don't know" message
An abstention is still an answer, and members judge it like one. A useful message does three things: it says what the assistant does cover ("I answer from the 2026 recertification handbook and the member FAQ"), it says plainly that this question is not covered, and it routes the member to a person or page that can help. It should not apologize at length, and it should never offer a general-knowledge guess after declining.
Where the platform allows it, staff should write this message themselves. One example is CustomGPT.ai's configurable I don't know message, which the association words itself and which is shown when its library does not hold the answer. Whatever the product, the message belongs to the content owner, not the vendor, and it should be reviewed with the same care as any member-facing policy text.
Questions association teams ask
Can an AI assistant tell members when it doesn't know the answer?
Yes. A retrieval-augmented assistant can decline when no retrieved passage supports an answer, and the decline can route the member to staff. Whether it does so reliably has to be tested with unanswerable questions, because OpenAI's 2025 research shows that models trained and scored only on accuracy are rewarded for guessing.
How does an AI assistant decide to say "I don't know"?
Usually at retrieval: if no passage clears a relevance threshold, or the passage is in a collection the user may not read, the assistant declines. It can also decline at generation, when passages are on topic but do not contain the answer, which is checked with a groundedness evaluation.
What is the difference between a grounded citation and a source list?
A source list names documents that were retrieved. A grounded citation ties a specific statement to the passage that supports it, often with a page, character position or chunk identifier, so a member can verify the statement without reading the whole document.
References
- OpenAI. "Why language models hallucinate." September 5, 2025. openai.com/index/why-language-models-hallucinate
- Kadavath, S. et al. "Language Models (Mostly) Know What They Know." arXiv:2207.05221, 2022. arxiv.org/abs/2207.05221
- Cole, J. et al. "Selectively Answering Ambiguous Questions." arXiv:2305.14613, 2023. arxiv.org/abs/2305.14613
- Wen, B. et al. "Know Your Limits: A Survey of Abstention in Large Language Models." Transactions of the Association for Computational Linguistics 13 (2025): 529 to 556. aclanthology.org/2025.tacl-1.26
- Lewis, P. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." arXiv:2005.11401, 2020. arxiv.org/abs/2005.11401
- Microsoft Learn. "Retrieval-Augmented Generation (RAG) Evaluators for Generative AI." learn.microsoft.com
- Amazon Web Services. "Citation" (Amazon Bedrock API reference). docs.aws.amazon.com
- Glean. "Citations." docs.glean.com
