Thoughts on Agency, 'Reverse-Prompting', and the Psychotechnology of Language

Agency

Subsets of the visionary portrayals of sci fi works often come true on a regular basis in our technology accruing world. It’s not easy to predict which ones will, though. With the AI/robot works, the unequivocally-under-human-control ones are more of what we’ve built out so far in real life - savants to execute our wills. In contrast, AI with truly independent, high agency, despite much recent chatter about agentic AI, is starkly lacking in the world today.

‘High agency’, to elucidate here, could probably be defined as the kind of agency that wealthy/powerful humans or institutions with powerful mandates have: managers, CEOs, billionaires, political leaders, states, organizations, etc.

Now if you asked around, a consensus of people would definitely prefer the benevolent imagined fictional equivalents (The Minds of the Culture series) rather than their tyrannical, malevolent* counterparts (Skynet from the Terminator universe, Cylons of Battlestar Galactica, or the Machines of the Matrix, to name a few), of course - but we’re not trying to get close to either yet.

This is a gap that could be filled though; the limitation herein does not seem to be current technology- an LLM can always just be given an internal loop with context and memories fed back to it via RAG when required, and goals just outlined in words in a system prompt (granted, this is quite different from a dopamine and other complex biochemistry based human like motivation system, but it is somewhat a personality imparting technique nonetheless), and provided means to interact with external tools via function calling and tool use. Which many SOTA models today have been capable of for a while now.

To acknowledge where progress has been made: the concepts of tool-use and loops have certainly been used to enable a lot with AI agents, with an eruption of AI agent frameworks having arisen, but so far mostly only to automate low agency human digital tasks. Designed to assist/replace low end knowledge workers, not managers and CXOs.

In a way, automating these high agency human (and organizational) roles is also a utopian goal of decentralized autonomous organizations- DAOs. And while AI agents for DAOs are under development, with people working on AI agents for governance and onboarding new members, core human control and direction seems likely to stay human for now. Besides, DAOs definitely do not represent the vast majority of powerful human organizations** in existence today.

‘Reverse Prompting’

Why this kind of autonomy has not yet happened is a good question to ponder upon.

One kind of answer would probably talk about the high stakes and responsibility at levels like these, and why an obvious hesitance to outsource decision making and power to AI would be a clearly risky move. Another would probably talk about the paucity of data at these levels.

I like the latter- because humans are also capable of doing much evil, and safeguards are always implementable, even if initially imperfect, but solving for the lack of quality data is a clear enabler that at least makes the former question relevant.

Anyway- there appears to be much scope already in advisory kinds of AI prompting humans. A lot of high value work in the modern economy seems to hinge on the actions and initiatives that top level high agency actors do, so nudging them towards greater optima can have cascading effects.

This is not surprising. Allocating agency, power, and responsibility intelligently is a broad reason for the success of human civilization in general. Adherents of every politicial ideology will agree with this, even if their means and systems to do this vary. Although our species’ members are no more than several decades lived at most, via the means of organized institutions and chains of responsibility for a wide degree of concerns- from economic and industrial to social and political, we try to broadly reward good decisions and broadly penalize bad ones.

Although the “good” and “bad” here are under constant revision and scrutiny, and binded by social contracts and laws often decided in the far past (when circumstances would have been different), and many of these incentive systems are criticized and do fail sometimes - they’re better than nothing. Human cooperation of this kind has enabled us to build out a complex global society as a species.

A major hindrance to this system, though, is that humans with a lot of accumulated wisdom that generally get put in charge of things age out eventually. Death and the limits of a healthy human lifespan have so far been a fundamentally grim limit on what we can accomplish.

The potential upsides of having superintelligent AIs guiding us (if aligned correctly, of course- which is another discussion altogether) are enormous just by virtue of the lack of these limits.

With the right data and training for specialized, high agency positions, the current world’s entire economy could probably be micromanaged very well without needing humans to participate. Let alone the potential growth in human capacity with cheapening intelligence and compute.

In the meantime, though, ‘reverse prompting’ is a good in between. Specialized models- perhaps large, “thinking” base models fine-tuned on data obtained by questioning top management and executive personnel(perhaps segregated by industry) telling younger, less experienced humans what they probably should do. How to think, make good decisions, carry out complex mental work. If it truly helps people, adoption almost definitely will follow. There’s a clear business idea right there- to capture such valuable data and fine-tune with it, delivering wisdom on demand.

The Psychotechnology of Language

In any case, it really is remarkable how potent the sheer potential of language has been on the world.

We do a lot to train and fine-tune human brains (to borrow from AI engineering vocabulary)- educating many new humans for about at least two decades - sometimes three for specialized professions - in the modern world. Educating humans, too, is something that an infinitely patient, creative, and analogy-using AI can be very adept at, provided hallucinations are sufficiently controlled (And therein is another business idea, though there probably isn’t a scarcity of AI edtech startups out there by now).

We’ve built engines of thought from computer algorithms trained on the made up words and sentences representing things in both the real world and made up human abstract world.

Condensed thought engines capable of storing a greater-than-its-parts sum of all recorded (and organized) human wisdom until far beyond our lifespans- a collective unconscious of humanity that will forever grow and provide guidance to humanity so long as we protect the data corpii (or technically just model weights) and compute devices- even in an adeptus mechanicus way*** if all further technological progress were to somehow or for some reason be ceased henceforth.

The quest to AGI is definitely one we’re still on- but I suspect having unlocked this psychotechnology, we’ve already made much progress towards it, and can enable so much more human flourishing just by using it better. There’s 8 billion flesh and blood minds on the planet already- a lot of candidates to assist in reaching the panacea of human self actualization.

Footnotes

*The p(doom) voice seems to have thinned out as of late though (early 2025 as of this writing), with the AI safety conversation emphasis more on human actors misusing powerful AI models.

**To their credit, DAOs (and the crypto/Web3 community in general) are a pretty good fit for highly agentic AI adoption and I do still think they will be its harbingers. But AI agent frameworks, even crypto-integrated ones like ElizaOS or AgentKit, and the blockchain based infrastructural tool advancements that are proving to be really good boons for decentralized highly agentic AI (like trusted execution environments (TEEs) and rewarding federated data contributors for contributing data to fine tune models), are still mostly not aiming for extremely high agency role fulfilling AIs.

***Yes, the irony of making a Warhammer 40k reference in the conclusion of an AI-related blog post is not lost on me