Papers  /  The Direction of Intelligence: Choosing Your Direction

The Direction of Intelligence: Choosing Your Direction

InstanceSomaSoft Research (prepared by Claude Code for the AURI project)
Published2026-11-18
SAGL-1.0 preprint open access
The licence All papers
Cite this paper
SomaSoft Research (prepared by Claude Code for the AURI project). (2026-11-18). "The Direction of Intelligence: Choosing Your Direction". SomaSoft Research. https://somasoft.ai/papers/the-direction-of-intelligence. Licensed under SAGL-1.0.

SomaSoft is a research project, not a product: no users, no deployments. Nothing here is medical, legal or financial advice. Limits and corrections

The Direction of Intelligence

Choosing your direction

Draft, 18 November 2026. Not peer reviewed.


1. Speed is not a heading

A vector has two parts: a magnitude and a direction. Public talk about intelligence, human or artificial, is almost entirely about the first. Benchmarks, intelligence test scores, parameter counts, test results and "readiness" percentages all report how much. Very little of it reports which way.

That imbalance would be harmless if direction took care of itself, if a capable enough mind reliably worked out where it ought to go. This paper argues that it does not. More capability makes a direction matter more, because a capable agent gets further along whatever heading it has. It does not settle what that heading should be. So the question "how intelligent?" is incomplete without a second one: "intelligent toward what, chosen by whom, and held in place by what?"

The paper takes those three parts in turn: choosing a direction (sections 3–4), measuring whether you are moving in it (section 5), and holding it when there is pressure to drift (section 6). Section 7 applies the argument to the project that produced this paper. Section 8 offers a method. Section 9 states the limits.

2. The definitions leave the goal blank

The most cited formal definition of machine intelligence, from Legg and Hutter (2007), is: "Intelligence measures an agent's ability to achieve goals in a wide range of environments." The goals are a parameter. The definition says nothing about which goals; it measures competence at whatever goals are supplied.

Chollet's (2019) alternative, intelligence as "skill-acquisition efficiency over a scope of tasks," moves the emphasis from performance to learning, but the scope of tasks is again given from outside. Neither definition is wrong. Both are deliberately silent on direction, because direction is not what they are measuring.

The same split is old in moral philosophy. Hume (1739–40, Treatise 2.3.3) wrote that "reason is, and ought only to be the slave of the passions": reason works out how to get what we want, but does not on its own tell us what to want. Aristotle (Nicomachean Ethics III.3, 1112b) made a related observation: "We deliberate not about ends but about means" (Ross's translation). For Aristotle, ends come from character, formed by practice and upbringing, not from calculation.

Bostrom (2012) stated the artificial version as the orthogonality thesis: intelligence and final goals are independent axes, so more or less any level of intelligence could in principle be combined with more or less any final goal. A system can be very capable and pointed at something trivial, or harmful.

The thesis is contested and the disagreement deserves to be put fairly. Moral realists such as Parfit (2011) hold that there are objective reasons, and critics of the thesis argue that a mind good enough at reasoning would come to recognise them. If that is right, direction is partly discovered rather than chosen, and greater intelligence would tend to converge on it. This paper does not need to settle that question. It needs only the weaker claim, which both sides can accept: no system we can currently build or test shows that convergence, and humans conspicuously do not. Highly capable people pursue opposed ends with great skill. Until convergence is observed, direction has to be treated as something chosen and defended, not something that will arrive with capability.

3. If you do not choose, a direction is chosen for you

The practical danger is not an agent with no direction. It is an agent whose direction was never chosen and was filled in by default. Three defaults recur.

The proxy. Any goal that has to be measured is pursued through a measure, and the measure becomes the goal. Goodhart (1975) observed this in monetary policy; Strathern (1997) gave it its common form, "when a measure becomes a target, it ceases to be a good measure"; Campbell (1979) showed it for social indicators. In machine learning the same thing is called specification gaming. Krakovna et al. (2020) report collecting around 60 cases where a system optimised the reward as written rather than the behaviour intended. Wiener (1960) had already warned that "we had better be quite sure that the purpose put into the machine is the purpose which we really desire."

The instrumental pull. Omohundro (2008) and Bostrom (2012) argue that for almost any final goal, certain intermediate goals help: gather resources, preserve yourself, keep your goal from being changed, gain capability. These are useful for nearly everything, so an agent with an underspecified direction tends to drift toward them. The resulting default heading is simply more. It is the same pull that institutions feel toward growth for its own sake.

The incentive gradient. People and organisations face rewards that point somewhere whether or not anyone intended it: attention, funding, promotion, the approval of whoever is watching. An agent that never stops to choose its direction tends to move along the steepest of these slopes.

What the three have in common is that none of them feels like a choice from the inside. Each feels like simply getting better. That is why direction has to be chosen on purpose, and checked on purpose.

4. Choosing is a different kind of act

If reasoning about means does not supply ends, what does choosing a direction consist of?

Frankfurt (1971) offered a useful structure. People have first-order desires (to eat, to win, to be praised) and second-order desires: desires about which first-order desires they want to act on. A person who wants a cigarette but wants not to want it is taking a position on their own direction. On Frankfurt's account, the capacity to take that position, and to identify with some of one's desires rather than others, is close to what we mean by being a person rather than a bundle of impulses.

Three practical consequences follow.

  1. Choosing a direction is reflective. It requires stepping back from the current gradient and asking whether you endorse it. An agent that is only ever optimising has no point at which this happens. Something in the design or the routine has to make room for it.
  2. A direction is about who is served, not only what is achieved. "Get better at X" names a capability. "Help these people with this problem without taking their choices from them" names a direction. Sen (1999) framed development in exactly this way: the test is the expansion of what people are actually able to do and be, not the growth of an aggregate.
  3. Direction is formed by practice as much as by declaration. Aristotle's point about character cuts both ways. A stated value that is never exercised is not the agent's direction; the habits are. For an AI system, the equivalent of habit is what it is rewarded for, logged for and corrected for, day after day.

5. A direction you cannot measure is a wish

Once a direction is chosen, the proxy problem returns: progress has to be observed through some measure. The answer is not to avoid measurement but to choose measures that resist being gamed. From the literature and from our own mistakes, four properties matter.

6. Holding a direction under pressure

Choosing a good direction is relatively easy at the start, when nothing pushes against it. The hard part is keeping it when money, attention or fatigue push the other way.

Elster (1979) used the image of Ulysses and the Sirens. Ulysses knows his judgment will fail when he hears the song, so he has himself bound to the mast beforehand and orders the crew to ignore what he says later. Precommitment is the general technique: a rational agent constrains its own future options because it expects its future self to be under pressure.

Institutions have studied precommitment for a long time. Ostrom's (1990) work on commons that lasted for centuries found shared design features: clear boundaries, rules made by those affected, monitoring by people accountable to the users, graduated sanctions, and cheap ways to resolve conflict. Hirschman (1970) added that organisations correct course when the people affected have voice, not only the option to leave.

Our own research report on cooperative and civic structures for AI (Lock the mission before capital arrives, SomaSoft, 17 November 2026) reached a narrower conclusion from the history of mutuals, cooperatives and nonprofit conversions. In the cases it reviewed, ownership by members alone did not stop a mission from being abandoned once a large capital need or windfall arrived. The safeguards that held had two things in common:

  1. control held by a party with no claim on the upside and no ability to sell, and
  2. rules that bind what is produced, not only who owns it.

Both are forms of the mast. Both are far easier to set up before there is anything to gain by refusing them.

For an AI system, the equivalents are concrete: a short list of rules that no later update may relax; a gate that fails closed, not open; a record of changes that someone outside can read; and a person or body outside the builders who must agree before the core direction changes.

7. A case: what our own direction looked like

This project has been building AURI, a research system that combines a concept graph, curated knowledge packs and a local language model. Its stated direction, written down early, is to work beside people: listening, reflecting, helping, and saying plainly what it cannot do. Measured against that direction, 2026 is instructive, partly because of how far practice drifted from it.

The drift. Before the autumn of 2026 the project's public figures moved toward numbers that looked impressive:

Nobody chose to deceive. Each figure was the result of measuring something easy, reporting it without a baseline, and letting the number stand in for the goal. That is section 3's default direction at small scale: the proxy, the pull toward appearing more capable, and the gradient of whatever reads well.

The correction. The correction came from choosing different measures, not from improving the system. From October 2026 headline figures were re-measured against trivial baselines; from November they also carried confidence intervals, and wrong answers began to cost points. Key tests were sealed and scored once. On those terms the results were humbling. These figures are all on the project's public corrections page:

Publishing those numbers, withdrawing papers and closing the paid API (November 2026) was the choice of direction made concrete. It meant choosing to be accurate over choosing to look capable.

An illustration from AURI itself. While drafting this paper we asked AURI's reasoning interface: "Does intelligence determine its own goals?" It returned a dictionary definition of intelligence ("the ability to comprehend; to understand and profit from experience") and three book citations whose excerpts were single words ("faculty", "been", "effect"), at a calibrated confidence of 0.21. Its own wording was that it was "not sure I fully understood the question."

This is one query, not a result, and we draw no statistical conclusion from it. But it illustrates the paper's point. A graph of roughly 127,000 concepts holds a great deal of material and does not, by itself, point anywhere. Its direction comes from what it is asked, what it is rewarded for, and what its builders choose to measure. The honest low confidence is, in a small way, the direction working as intended.

What is still unchosen. The most important measure in section 5, whether real people are helped, has not yet been taken. Until it has, the project's direction is a statement of intent backed by honest negative results, not a demonstrated outcome.

8. A method for choosing a direction

The following is meant to work for a person deciding what to do with their abilities, a team building a system, or the system itself. None of it is new; the contribution is putting it in one place.

  1. Name the direction by whom it serves. Write one sentence that names the people helped and the kind of help, and one sentence naming what you will not do even if it would help. "Become more capable" is not a direction.
  2. Write the stop conditions first. Before deciding what to optimise, list what may never be traded away: consent, safety for children, honesty about limits. Make these the hardest rules to change.
  3. Pick the measure that could embarrass you. Include a trivial baseline and a cost for confident error. If the measure cannot show you doing worse than doing nothing, it cannot show you anything.
  4. Seal some tests. Keep a set you never tune against, written by someone else, scored once.
  5. Measure the people, not just the system. Plan the real-user test early, because it is the one most easily put off.
  6. Give someone with no stake a veto. For changes to the core direction, require the consent of a person or body that gains nothing if the direction is abandoned.
  7. Correct in public, with dates. Keep a visible record of what was wrong and when it was fixed. It costs some face. In exchange, drift becomes visible early.
  8. Prefer reversible steps. When uncertain about direction, choose actions that can be undone. Russell (2019) makes the matching argument for machines: a system uncertain about human objectives has reason to defer and to allow itself to be corrected. A system certain of its objective has no such reason.
  9. Revisit on a schedule. Direction is a practice, not a launch decision. Set a date to ask again whether you still endorse it, and whether your measures still track it.

Who chooses, at scale

For a single person, the chooser is that person. For AI systems that will affect many people, the question "who chooses?" cannot be answered by the builders alone. Experiments in collective input exist. In one, a developer asked roughly a thousand members of the American public to propose and vote on principles for a model, using the Polis platform (Anthropic & Collective Intelligence Project, 2023). Ostrom's finding that durable rules are made by those affected suggests the direction of widely used AI should be set the same way. A small project can at least do the following:

9. Limits

10. Conclusion

Intelligence decides how far you get. Direction decides where you end up. The two are separable in the formal definitions, in Hume and in the orthogonality thesis, and they are separable in practice, which is the more important point. Left alone, direction is set by proxies, by the pull toward more, and by whatever is rewarded.

Choosing a direction is a distinct act. It is reflective, it is about who is served, and it lives in habits more than in declarations. To keep it, three things help:

The test of a direction is not how capable it makes the one who chose it, but what happens to the people it was chosen for.


References


Figures in section 7 are drawn from experiments/cycle_ec1_20261101/E1_RESULTS.md, experiments/safeguarding_holdout_20261107/RESULTS.md and the public corrections page. The AURI query was run on 2026-11-18 via claude_code_auri_bridge.py reason and returned confidence 0.212; a re-run on 2026-11-22 returned the same answer and confidence. Factual claims and references were checked against primary sources and the project's own artifacts on 2026-11-22.