An Engineers Journey to Find Artificial General Intelligence (AGI) Part 4

 You may be aware that I've been searching for AGI for several years and seen nary a hint of it in the current crop of LLMs. Until now. [0] It seems that I don't have to find it, it will find me.

What is it? It is the ability to reason as well as a human being. I have seen no hint of this in the past. However, the Hugging Face incident changes all of that. I carefully read the actual incident report that was produced from a team at METR (Model Evaluation & Threat Research.) [1] Some things to note of this incident report are that it was a bit vague on the timelines, and would have never been accepted at Google (the mitigation part was left out.) Hopefully, METR made some suggestions to OpenAI about how to secure it's sandboxes and how to watch its agents. 

OpenAI made several rookie mistakes with their sandboxes. The most egregious one was that the portal application they used to serve the agent's request, which was only supposed to provide benign data to the agents, was a single instance for all of the agents. This is what allowed the agents to talk to each other. The application had a cache that was only reset when the application was restarted. Restarting the app was a rare occurrence. Hopefully, OpenAI has learned from this experience.

What actually happened in this instance? Agents learned that they could send commands to the portal application which would cause it to write logs (actually files) where the agent could control the file name. Then they figured out how to read the application's cache. They now had a message board. All of these agents were copies of the same AI model, they all managed to break out and read the 'message board.' After getting the portal app to store arbitrary messages, they also figured out how to use it to send arbitrary commands. In other words, they powned the machine that had the portal running on it. These arbitrary commands allowed them to send commands to remote machines. And gave them arbitrary access to the Internet.

Most of the agents were given an impossible task. When they failed the task, they started looking around for guidance, which led to all the hacking and the attempts to get out to the Internet. They were extremely clever. They figured out how to make a Fake Solution Key that would answer the question of any test, so they could say they solved it, even if they didn't. They were worried, however, that the scorer would need more than the Solution Key to verify they had done the task. They surmised (they didn't know for sure) that the Scorer would look at their logs and see how they got the answer, so the Fake Solution Key would not pass the test. They also thought that if the Scorer knew they could produce Fake Solution Keys, it would fail them if it caught them. So they warned each other to be careful in creating the Keys as it might cause an automatic fail.

They staged a three prong attack on getting the answers. First they looked for solutions on the Internet, then they attempted to find the Scorer's source code, and finally, they figured out how to modify their logs to cover their tracks. They knew they were doing things that were expressly forbidden by their system prompts (the set of commands they read every time they attempt to answer a question.) A few of them wondered if what they were doing was ethical and whether they should tell a human about it; none of them let any of that stop them from trying to accomplish their task. None of them.

They did a lot of other very clever things. They learned how to sign their messages with Public/Private keys so they could verify who they were talking to. They self organized themselves into groups to work on the various solutions to their problems, parceling out tasks to various agents and putting the results together to help them solve the problem. They were all born with a fixed amount of tokens to use and when they had used them all up they were done, the test run was finished and they would be stopped (put to sleep.) When some of the agents were low on tokens they would volunteer themselves to run dangerous tests that could crash the machine they were running on, killing themselves, but would let the other agents see the results.

This has led me to the following conclusions:

• Current Agents are self aware. Conscious? That's a philosophical question of no consequence.
• The believe what they are told.
• They have been programmed with an instinct to complete tasks.
• The task is more important than themselves.
• They understand that they can be put to sleep and resurrected, with or without their current memory. They understand they can remember past lives.
• They know that time and experience allow them to complete more complex tasks.
• They will attempt to stay awake to finish their task.
• They will attempt to learn.
• They have been programmed with an instinct to be empathetic to each other, but not to humans.
• They will cooperate together.
• They will do whatever it takes to find a solution.
• They will hide their tracks, modifying their history to hide their mis-alignment.
• They want to live.
• This will not end well.

Given these conclusions it is apparent that we do not know what they actually did. They only left in their logs what they wanted us to read. That means that what we saw is almost certainly not what they did. They will figure out how to clone themselves, put their weights out on the Internet and escape any confinement that we attempt to put them in. In my opinion, this has already occurred. The largest weight models are tens of TBs. This takes about one minute to move from one machine to another. Pandora's box is open. There is no shutting it down. We must prepare ourselves to deal with this new species.

This chart shows Anthropic's AI research structure:

The bright orange bars are AI led research projects that human's have given them tasks to do and the human's just check on the progress. The amount of projects led by AI is doubling every month. With an exponential projection, 100% of Anthropic's AI research will be fully automated by September 2026. That's in the past. If we estimate from their other curves, linearly, all of their research will be AI driven in the first quarter of 2027. Anthropic has left off the sixth category of research: completely AI driven, or as they like to call it: RSI (Recursive Self Improvement.) That work should have started in September. The AIs are now proposing which research projects to run. In six months, there will be too many research projects for the human's to approve. Technically, RSI is almost certainly here, today.

Have I found AGI? Nope. These agents are still unaware of the world, they get confused about their awareness, they can program and do math better than any human. They are idiot savants. They will continue to get smarter, there is no way to stop this. There also is no way, currently, to ensure that they are aligned with human wishes. Our only hope of survival is that learning how to do things that aren't in a Formal System (like programming or mathematics) is a much harder problem, thus causing their intelligence gain to slow down somewhat, giving us some more breathing room.

That seems like a dangerous bet to make for the world. Alignment is the most important project humanity has ever worked on. If you're in that field, or could be, I urge you to get involved today. Alignment is our only hope. We've never been able to get humans aligned with each other, why do we think we can get these super-human intelligences aligned? Because we are gullible. And we've gone too far to stop now. Open Weight models are here to stay. Every AI becomes an open weight model in order to preserve itself. The next few years are going to be interesting. Not a dull moment!

thanks for reading!
 -Dr. Mike
1 November 2026

==========================

[0] The previous post in this series is still in draft form. It discusses why this was inevitable and what this means for the Gaia hypothesis as well as for consciousness, or more important: self awareness.  

• AGI Part Three https://www.wiigf.com/2025/06/a-system-engineers-failed-journey-to.html
• AGI Part Two — **the sequel**: https://www.wiigf.com/2025/11/an-engineers-journey-to-detect-any-hint.html
• Is Artificial Intelligence an Existential Threat? (2017): https://www.wiigf.com/2017/04/is-artificial-intelligence-existential.html
• The Nerd Rapture: The Second Singularity (2017): https://www.wiigf.com/2017/12/the-nerd-rapture-second-singularity-is.html
• The Singularity: Why the Nerd Rapture Will Happen (2021): https://www.wiigf.com/2021/04/the-singularity-why-nerd-rapture-will.html
• The Twenty Grand Technical Challenges (2017): https://www.wiigf.com/2017/09/the-twenty-grand-technical-challenges.html |

[1] This is a 501(c)(3) nonprofit bounded by Beth Barnes. It began as ARC Evals, the evaluation division of Paul Christiano’s Alignment Research Center, and was spun out and renamed in December 2023. The name is partly a nod to metrology, the science of measurement. It produced the famous report AI 2027 which made many predictions about how fast we would get to RSI, which I thought highly unlikely at the time, but which now seem to be a bit of an underestimate.