Superintelligence Tool Pagination: Fetch Every Page or Flag It
A practical method for AI agents reading paginated APIs: follow every cursor, verify completeness, and flag any incomplete pull for human review.

An AI agent should follow every page of a paginated API response by using the vendor's documented continuation signal, not by guessing. It should record what each page contained, confirm the pull ended on a clear end signal, and compare its counts against any total the source reports. If a page fails, a cursor is rejected, or the counts disagree without explanation, the agent should mark the result incomplete and send it to a named person before anyone relies on it.
Key takeaways
- Follow the API's own continuation signal, such as a next cursor, a next link or a has-more flag, until it clearly ends. Never infer the end of a list from a short page alone.
- Treat an expired cursor, a repeated token or a timeout as a question to answer, not as the end of the data.
- Store a completion receipt for every pull: pages read, records received, the final token, retries and any gaps.
- When a pull cannot be proven complete, label it incomplete and block any downstream action until a reviewer decides.
- Design the tool so pagination state arrives as structured fields, so the agent does not have to interpret free text.
What makes pagination risky for an agent?
Most business systems do not return a full list in one response. They return a slice of records plus a token or link that points to the next slice. A person clicking through a screen notices when the list stops. An agent that reads the first slice and writes a summary can easily treat that slice as the whole answer, and nothing in the text it receives will announce the gap.
Pagination creates four specific failure modes. The first is silent truncation, where the loop stops early because of a timeout, a page limit or a misread field. The second is drift, where records are inserted, removed or reordered while the pull runs, so one record appears twice or another is skipped. The third is expired or reused cursors, which some APIs invalidate after a period of time. The fourth is partial failure, where a rate limit or server error interrupts the pull after many pages have already been read. Retry rules matter here, and they are covered in AI Agent Rate Limit Handling: Retry, Backoff and Queue Rules. For pagination, the key point is that a retry must resume from the last confirmed token, not restart silently.
How should an agent follow a paginated API?
The agent should follow the documented continuation mechanism exactly. The steps below can be executed by a tool wrapper or an agent harness, which keeps guesswork out of the parts where it causes harm.
- Send the first request with an explicit sort order and the largest page size the vendor allows, so the request can be reproduced.
- Read the documented continuation signal in each response, and do not treat a short page as proof that the list has ended.
- Fetch the next page using only the token from the previous response, keeping every query parameter identical to the first request.
- Before moving on, save the page's record identifiers, the token used to fetch it and the time of the request.
- Stop only when the continuation signal is absent or false, and record which signal ended the loop.
- If a page fails, retry from the same token under the vendor's rate rules. If the token is rejected, mark the pull incomplete rather than restarting from the first page without a note.
Restarts matter for writes. A pull that is reprocessed must not create duplicate actions, so any downstream write should key on a stable identifier. The pattern is described in Idempotent Tool Writes for AI Agents: Stop Duplicate Records.
How does an agent confirm it read every page?
Finishing the loop is not the same as proving completeness. A completeness check should be a short set of facts the agent can state and a person can audit. Use this checklist before any result is described as complete:
- The loop ended on the vendor's documented end signal, and the agent can name that signal.
- Each request used a token that came from the previous response, no token repeated, and no page sequence moved backward.
- The number of unique record identifiers matches the number of records received, so duplicates across pages were noticed.
- If the vendor reports a total, the unique count matches it, or the difference is recorded together with the reason the vendor gives.
- The sort key was fixed in the first request, and the agent confirmed that returned records follow it.
- A completion receipt was stored with the query, start and end times, pages read, final token and any retries.
A reported total can change during a live pull because records are created or removed on the vendor side. A mismatch is therefore information, not an automatic failure. The agent should record it and apply the flag rules in the next section, rather than adjusting its count until the numbers agree. Treating each pull like a traced operation, with a clear record of each step, follows the general logic of [OpenTelemetry's observability concepts](https://opentelemetry.io/docs/concepts/observability-primer/). Behavior like this can also be tested before release with a set of known cases, as described in [OpenAI's guide to evals](https://platform.openai.com/docs/guides/evals).
When should an agent flag an incomplete pull?
A flag is an instruction to stop relying on a result and to ask a named person what happens next. A flag costs a short delay, while a confident summary built on a partial list can cost much more. Use this decision guide:
- Flag when any page after the first fails with a server error, a timeout or a rate limit that the retry rules cannot clear.
- Flag when a token is expired, rejected or returned twice, because the sequence can no longer be trusted.
- Flag when the unique count and the reported total differ and the vendor offers no explanation.
- Flag when the pull feeds a payment, a message to a customer, a deletion, a status change or a financial report.
- Flag when the loop reached a page ceiling set for cost or time, because a stopped loop is not a finished one.
- Do not flag, but still store the receipt, when the pull ended on the documented signal, the counts agree and the output is a read-only internal summary.
A flag should state four things: which query ran, which pages were confirmed, where the sequence broke and which action is now blocked. The review route should be fixed before go live. For example, the team's chat channel can deliver the flag to a named operations lead with clear choices such as retry, accept as partial or cancel. The approval pattern is described in Slack Approval Buttons for AI Agent Actions: Decide Before Any Write.
What should a tool return so pagination stays honest?
Many pagination problems begin in the tool design, not in the model. If a tool returns only a list of records, the agent has to infer the state from whether data appeared. A better tool returns pagination as named fields: the records, a next cursor or null, a boolean for whether more results exist, the count returned in this page, and a total only when the source system provides one.
Errors deserve the same care. An expired cursor, a rate limit and a permission problem each need a distinct code, because each leads to a different action. The tool should also accept an explicit maximum number of pages, and when that ceiling is reached, the response should say the pull is incomplete instead of returning a clean-looking partial list. Descriptions and error messages are part of the contract, and the design choices are explained in Tool Schema Design for AI Agents: Names, Descriptions and Errors. Guidance on writing tool descriptions for agents is also available in [Anthropic's notes on writing effective tools for agents](https://www.anthropic.com/engineering/writing-tools-for-agents).
Illustrative scenario: a supplier catalog pull
The following scenario is illustrative and does not describe a real client. An operations lead asks an agent to compare a supplier's current catalog with the prices recorded in the purchasing sheet. The supplier API returns the catalog in pages and gives a next cursor on each one.
The agent reads the first pages without trouble, saves each page's identifiers and keeps every cursor. Partway through, the connection drops, and the retry returns a cursor the agent has already used. Under the rules above, the agent does not continue from that point or restart on its own. It saves a receipt listing the confirmed pages, marks the pull incomplete and sends the flag to the purchasing lead, noting that the comparison is blocked.
The lead decides to run the pull again with the same sort key during a quieter window, and to compare only the records that appear in both receipts. The second pull completes, the counts agree, and the comparison is released with its receipt attached. Nothing was summarized from a partial list, and nobody had to reconstruct the history from memory.
How Be Human handles pagination in managed operations
In a first Be Human engagement, pagination is treated as part of the tool contract rather than as a detail hidden in a prompt. The team defines which continuation signals each source uses, what the completion receipt contains, which conditions trigger a flag and who the named reviewer is, all before the agent is allowed to act on a pull. In managed operations the work continues: receipts are monitored, flags are reviewed when they fire, and the tool is adjusted when a vendor changes how it pages results. The goal is a plain record of what was read and what was not, so the business decides on evidence. To find the first task worth handing to an agent, take the free Be Human AI diagnostic.
Frequently asked questions
How should an agent behave if the API has no next cursor and no total? The agent should not conclude that a result is complete just because a response came back. It should look for another documented end signal, such as a next link or a count field, and if none exists, record that completeness cannot be proven from the response. Such pulls should be flagged for review or replaced by a second pull whose receipts are compared.
Should an agent ever estimate the number of pages in advance? No. An estimate based on a total or a page size can be wrong if records change during the pull, and it hides the real stopping rule. The agent should follow the continuation signal and report the pages it actually read, using any reported total only as a check on the result.
Can an agent summarize a partial pull? It can, but only if the summary is labeled partial and names the pages confirmed and the point where the sequence broke. Presenting a partial list as a complete answer is the main risk this article is meant to prevent. For anything that drives a payment, a message or a status change, the partial result should stay blocked until a person accepts it or a complete pull replaces it.
Who should review a flagged pull? The reviewer should be the person who owns the business process, not only the engineer who built the connector. For purchasing data that is usually the purchasing lead, and for financial data it is the person responsible for the books. The review route should be set before go live, with a short list of choices such as retry, accept as partial or cancel, so the decision stays with the owner.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


