Superintelligence Skeptics and Business: What to Do Either Way

Superintelligence skeptics argue that the leap from today's useful but uneven AI to systems far beyond human ability is neither close nor guaranteed. For a business, the practical answer is the same whichever side proves right: build narrow, well-governed agents whose memory, permissions and tests belong to you. This article states the skeptical case fairly, then sorts preparation into what holds up under both outcomes.

Key takeaways

• Skeptics make serious arguments about the limits of scaling, unreliable long tasks, slow deployment and vague definitions. A business should weigh them, not dismiss them.
• The disagreement is mostly about timing and size. AI agents in bounded roles are useful now, whichever side is right.
• Preparation that depends on a forecast is fragile. Preparation built on your own operating knowledge, permissions and tests is not.
• Six steps pay off in both futures: one scoped role, written operating knowledge, approval boundaries, your own tests, a replaceable model and a regular review.
• Watch evidence from your own agents rather than headlines, and let it set the pace of expansion.

What is the strongest case that superintelligence is far off?

Skeptics are not a single camp. Some think very capable AI is decades away, some think the label itself is meaningless, and some accept fast progress but doubt that it changes business as quickly as promised. Five arguments carry most of the weight:

• Scaling has limits. Larger models need more data, energy and money, and gains on tests may not carry over to open-ended work. Each further step may cost much more than the last.
• Intelligence is not one dial. A system can be excellent at one task and weak at a neighboring one, so a single word like superintelligence hides a very uneven profile.
• Reliability compounds. A long task made of many steps fails when small errors accumulate, and an agent that shines in a demonstration can drift in a real workflow.
• Deployment is the bottleneck. Most business results depend on approvals, contracts, physical goods, regulation and people changing habits. None of these speed up because a model got smarter.
• Forecasters have incentives. People predicting rapid arrival often gain funding or attention from the prediction, and confident forecasts in this field have been wrong in both directions.

A sixth point is quieter and matters most to operators: much of what a company knows is tacit. It lives in habits, exceptions and judgment that were never written down, so a model trained on public text does not have it. If you want precise definitions of the terms in this debate, the superintelligence glossary separates AGI, ASI, alignment and agents.

Where does the skeptical case run out?

Skeptics are right about many specifics, but the case has gaps that deserve equally plain treatment.

The absence of a proven ceiling is not proof of one. Arguments about data or energy describe pressures, and engineers have often found ways around pressures of that kind, though not always. Uneven ability also cuts both ways. A system does not need to be superhuman at everything to take over much of one role, and many business roles are narrow. Slow deployment means slower change, not no change, and the companies with their processes written down move first.

Caution applies to the optimistic side too. Claims that a system can handle long, open-ended work are testable on your own tasks, and some will fail that test. A vendor demonstration is evidence about the demonstration, not about your workflow.

Neither camp has settled evidence about the next several years. Risk frameworks such as the NIST AI Risk Management Framework are built for this condition. They ask you to govern, map, measure and manage risk without needing a forecast, which is the right posture here.

Why does the disagreement matter less than it seems?

Consider two futures. In the first, the skeptics are right: models improve steadily but unevenly, and the main source of value is fitting agents to specific processes. In the second, the optimists are right: far more capable systems arrive and can handle longer, harder work.

In both futures the scarce resource is the same. Capable models are available to everyone who pays for them, so the difference between companies comes from what each can safely give a model: accurate context, scoped access and a way to check results. A company with those can use a stronger model soon after it arrives. A company without them gets little from even a very good model and carries more risk.

This is also why the two obvious reactions both fail. Betting heavily on a forecast, by buying a large platform or reorganizing around an arrival date, is costly if the forecast is wrong. Waiting for the debate to end means starting from nothing when the evidence finally comes. The 90-day plan for far more capable AI takes the middle path of building the base that either future needs.

Which preparation steps pay off whichever side is right?

These six steps do not depend on who wins the argument. Take them in order.

• Pick one role with a clear owner. Choose a recurring piece of work, such as drafting replies to routine supplier questions or preparing a weekly report, that a named person owns and can judge.
• Write down how the work is really done. Capture rules, exceptions, examples of good output and who decides what. This becomes the agent's memory, and it helps new employees even if you never use an agent.
• Set approval boundaries before launch. List what the agent may read, draft, send or change, and which actions need a human yes. Give it the least access that does the job.
• Define an observable success criterion and a small set of test cases. Keep tasks with known good answers so you can rerun them whenever the model or setup changes. Anthropic's writing on evaluating agents is a practical reference for how to structure them.
• Keep the model replaceable. Store memory, instructions and tests in formats you control so that switching models is a configuration decision. The guide to avoiding AI vendor lock-in covers the contract and technical side.
• Review on a schedule. Check permissions, logs and test results at regular intervals, as described in the quarterly permission review.

Each step earns its place in both futures. If models stall, the first three give you a dependable process and the rest keep it dependable. If models leap, your tests tell you quickly whether the new model is better on your work, and your boundaries limit the damage if it is not. Anthropic's engineering guidance on building effective agents makes a related point: start with the simplest arrangement that works and add complexity only when evidence supports it.

An illustrative scenario: one decision, two futures

Illustrative scenario: an operations manager at a mid-sized distribution company assigns an agent one role, preparing first drafts of replies about delayed orders. The manager writes down the rules for when to offer a replacement or a credit, collects past cases with the replies the team considered good, and decides that the agent may draft but not send, and may never promise a credit without approval.

In the first future, models improve slowly. The agent stays a drafting assistant, the test cases catch small regressions after each model update, and the team gains steady time every week.

In the second future, a much stronger model becomes available. The manager reruns the same cases and sees that hard exceptions are handled better. She allows the agent to send routine replies on its own while keeping credits behind approval. The change is a decision about a boundary, not a rebuild. The memory, the tests and the permission record all carry over.

What should you watch instead of the headlines?

Headlines will keep swinging between confident arrival and confident collapse. Your own agents produce better evidence, and you can collect it cheaply:

• How often reviewers accept the agent's output without edits on your test cases.
• Whether failures repeat the same kind of mistake or keep appearing in new forms.
• How long it takes you to gain confidence in a new model after it becomes available.
• Whether the people who own the work choose to use the agent.
• Whether the agent's access needs are growing faster than your ability to review them.

Use the same method on outside claims. Ask what a capability claim means for tasks like yours, who ran the test, and whether you could repeat it. A claim you cannot repeat is a prediction, and predictions belong in the discussion, not in the plan.

How does managed operations handle the uncertainty?

Be Human designs, implements and operates personalized AI infrastructure for companies. A first engagement defines one useful role, the memory and tools it needs, approval boundaries and an observable success criterion. Managed operations can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. When a new model appears, the recurring evaluations show whether it helps on your work, and any wider access goes through the approval boundaries instead of around them. The service is remote, in English and Portuguese.

If you want to find the first task worth handing to an agent, take the free Be Human AI diagnostic.

Frequently asked questions

Are superintelligence skeptics right that the claims are overblown? Some claims are, especially confident dates and sweeping statements about all work. Others, such as steady gains in tool use and in longer tasks, are hard to dismiss. A sound plan treats both as possible and tests capabilities on your own work.

Should a company wait until the debate is settled? No. A scoped role, written knowledge, approval boundaries and your own tests are useful today and easy to adjust later. Waiting mostly delays learning how your organization works with agents.

Does skepticism about superintelligence mean AI agents are not worth adopting? No. Skepticism about superintelligence is a claim about the far end of capability, while an agent in a bounded role is a present-day matter you can judge against a success criterion. Adopt agents for the role in front of you, not for a forecast.

Who should own this decision inside the company? A named executive sponsor should own it, with the person who runs the affected process as the day-to-day owner. The sponsor sets approval boundaries and reviews evidence, while the process owner judges the quality of the output.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.