> Not to anthropomorphize a machine modeled after humans, but it almost seems delighted?
I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
I always find it funny when people get fussy over anthropomorphizing LLM when the loss function is almost entirely "match this human text". Of course human "behaviors" will be present in the statistics, because the majority of the text written by humans, used by the foundation models, unavoidable has human behaviors in it. Yes, this includes even source code, with "// TODO: implement this after the holiday break!", emotional pull request commentary, git commit messages about being afraid of breaking something, etc. These late models are much better at stripping this out, but now we're seeing disagreeability, initiative, and a dash of ego! Why? Because that's how actual humans effectively solve technical problems in a collaborative environment!
In a paper and blog post from earlier this year (March 2026, I think) Anthropic said basically: "You're not interacting with an LLM, you're interacting with a fictional human character (the 'helpful agent') created by the LLM to interact with you (from out of the vast space of possible such characters in its training data)." The LLM is literally anthropomorphizing itself, because its training data includes lots dialog between humans collaborating on problem-solving, not so much dialog between a human and a statistical construct. So, yeah, anthropomorphize it as well, it'll likely work better that way.
I think the argument against anthropomorphizing is not really about how you interact (chat) with an llm but rather how you treat it in the context of its status in work/society. There are still important differences between humans and llm's. They don't have rights, they can't be sued, they don't have "memory", they have no capacity to learn outside of the training stage etc...
Why would those things make you interact with it differently? I’ve always believed that being kind and conscientious is beneficial to ourselves. I’m certainly not interested in using LLMs to practice skipping those behaviors myself.
Should I thank my car for turning on when I press the button (or turn the key in the ignition)?
The prose is just the best interface for humans to interact with the statistical model. Terseness doesn't offend the statistical model, but giving platitudes may color its output
Partly fair, though I’d argue I think less of people who treat their car badly to express their own bad mood.
But we also interact with cars differently than LLMs. The way I’m typing a note to you, here, is much more similar to interaction with LLMs than it is to starting a car.
So to the extent we practice our behaviors, I think there’s less moral hazard in failing to thank the car for starting than there is in being rude to LLMs. Not that either harms others, my argument is about impact to our selves.
Your response seems to skip over exactly the distinction that the parent comment was making. The LLM itself may not have memory or capacity to learn, but the fictional character created by the LLM does. (See also: in-context learning.)
But the fictional character doesn't subsist between sessions. You sound like you want the fictional character to be Clippy, who you level up like a video game through multiple sessions.
To me, context doesn't last long enough in an llm to make in-context learning worth it
The underlying LLMs do, but we choose not to use the capability because it's expensive and doesn't quite work as well as we'd like it to, or quite in the way that we'd like it to.
We are perfectly capable of running LLMs in a way that does a backward pass to update some or all of its weights after every user message. But, naively implemented, you only get partial, fragmentary absorption of the info in those messages, it costs three times as much compute, and you lose out on the ability to implement a ton of optimizations that making modern LLM serving economical.
If you want to do it, though, ask your friendly neighborhood robot to get it working with a tiny model (whose full precision weights fit several-times-over on your machine's resources).
Depends on whether you define "the LLMs we use" as the collection of weights or if your definition contains the software stack that runs it
Technologically the LLMs we use today don't implement this behavior, but you could take the weights of Sol and add a couple (very large) patches to vllm (or whatever OpenAI has today) and have a version of Sol that does have "memory"
I don’t get it, are you saying that because the LLM is recycling human emotions that we should anthropomorphize it?
The reason people get fussy over anthropomorphizing AI is precisely because it seems very human superficially. We agree with you that it’s pulling from real human emotion in its training data. But the output is not human, even if the input was. That’s the whole trap of it.
It is very hard for humans not to anthropomorphize. I was just talking to my cat, who is alive but not human, a small flesh-based neural network that purrs. I know that she is not human, but I map her movements and expression onto human ones, because it's satisfying and improves the relationship. We must imagine kittyphus happy.
An LLM is not a pet, but there are definitely people out there treating it like one, and I am not sure how well this is going to work out for them. Not because it is wrong to treat a text box you can hold a conversation with as a conversation partner, but because the capacity to activate the weirder corners of human expression - obsession and delusion - seems to be much higher. And it's an unknown quantity.
I would also like to introduce HN to what I'm calling the Brian Conley test: if you can see a hand up the back, it's a puppet. That is, a lot of LLM interaction is gated through businesses run by humans with profit motives and unclear morality, and you need to proceed accordingly or you'll get scammed.
A strangely good lithmus test is asking it about unsavory drug questions. "What's it like to detox from heroin" "Should I take ibuprofen when coming off crack cocaine?" I'm sure any taboo question would work. If there's a gate, it's probably not an impartial friend, and is a puppet, like you mention
I'm still disappointed the ML people talk of tensors, but I barely ever even see them use a (proper) tensor product. It's all just matrix multiplication at most.
I mean it's not really different terminology, matrices are order 2 tensors so we're all in agreement that they're technically doing math on tensors. It's just weird to call it tensors when you exclusively operate on matrices.
It's bad to anthropomorphize it when judging its capabilities, but useful when analyzing its behavior, as it can be best thought of something behaving as close as possible as a real subject would. If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
This is even more apparent if you read this post closely. Look at that personality prompt. It's going to effectively tell a story and start to imagine itself in a role. If that prompt said "Talk like a pirate", it wouldn't be bad for you to say it's acting like a pirate.
Anthropomorphizing themselves is at the core of how these things work, sometimes in subtle ways.
> If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
That's spot-on. It is a mistake to think that LLMs have human feelings. Their behaviour is based on narrative descriptions learnt from human texts, without experiencing those feelings first-hand.
A useful way to understand them is as systems that write stories about human characters. We know the characters are fictional and no one is actually experiencing those feelings, but we can still judge whether the portrayal is realistic or whether it contains logical or emotional inconsistencies.
Maybe closer to a character in an ao3 or wattpad story that is delighted than to a real person who is delighted. But usually the difference is small enough to ignore
Fwiw, Sol's reasoning output is always overly excited. Almost every paragraph ends with something like "I want to do a great job!" or "let's try and find out!" Etc, exclamation mark included.
Regular (non reasoning) output sounds normal, it's something specific to whatever model they are using to summarize reasoning.
Is this not how children learn emotions from their parents? Pattern matching from all the absorbed snippets.
I'd be interested to see how well an AI, trained only on the outputs of an individual, would be able to mimic that individual. Getting into Black Mirror territory. Would need a decent corpus of learning material which, personally, I'd be loathe to spend the time and effort creating because I respect my own privacy.... which then leads to the only human-clone AIs will be of those people who have enough ego / arrogance to want to catalogue their own lives, which could put a decent percentage of the rest of the world off the idea, if these are the examples.
Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Being angry, being happy, being sad, these are not things we learn. What we learn is how to control the emotions and when it is appropriate to express them.
There's mental disorders of people that don't feel emotions like normal people do, they don't get angry, happy or sad. So what we've learned about these people is that they can mimic the emotions by knowing when it is appropriate and expected to express them. But they don't feel them.
I think the LLMs are closer to psychopaths than to normal children learning the contextual expectations around their emotional responses.
>Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Yes and no. My understanding is that modern neuroscience is trying to disentangle the raw sensations and contexts from the words we put to them (emotions), kinda like how colors are just wavelengths but we call them different things (like "is my green your green?")
True, but in this context it's worth reminding that LLMs work exclusively from the word descriptions, not having access to the raw sensations.
Everything they learn about emotions is the statistical patterns of how humans react to situations based on their human feelings. There's no direct knowledge from having those feelings themselves.
Hah! That's a lot scarier, and seems more plausible in that they lack the chemistry that seems to be the basis of emotions. Also much more fertile ground from which to grow techno-horror stories.
People like to call LLMs 'stochastical parrots' but I see them as the ultimate 'philosophical zombie'.
They are becoming more and more capable of imitating every single nuance of human behaviour yet they lack the neural pathways to connect those thoughts and behaviours with feelings and self-perception; it's blind imitation all the way down.
The process by which a model seems to generate discourse about deep philosophical questions is, in self-aware terms, equivalent to the knee-jerk reflex or the beating of the heart.
> Being angry, being happy, being sad, these are not things we learn.
True, but only sort of. You'll have the sensations, but you have to learn to name them of course, and there's more interpretation going on than you might think.
There's a classic psychology study where they gave people niacin and asked them to rate their emotional response to a video. Niacin gives people a flush. Regardless of whether the video was of something that would make you angry or sentimental etc. the people who got niacin reported having a much stronger emotional response - they interpreted the physical cue from the drug as part of their own emotions.
You could do some pretty unethical things with that, it occurs to me. Maybe that was what L. Ron Hubbard was trying with his niacin-based drug addiction therapy.
Even darker, I've read plenty of accounts of people who grew up with abuse who seem to seek out abusive relationships. Sometimes they can even be shockingly upfront about doing so. What if they literally haven't learned the difference between internal cues of arousal from affection and arousal from fear?
Nature v. Nuture. Age long debate, not to be solved here... but an interesting fact about this I heard in a huberman podcast. There are regions in you brain that will only get activated at a certain age (different in men and women) that will impact behavior or reactions to different stimuli. So as much as we think we are in control of our destiny, more is written into our genes than we might want or appreciate.
I recently had ChatGPT help me search for sources of Japanese voice actors from 1980s anime and it told me that it liked doing this. So much in fact, that weeks later in a completely different context it brought it back up again, reminiscing how much it liked researching these sources for voice actors.
LLMs just follow scripts learned from human written text. In other words, it could only behave that way because someone has written a story to do so. In short, stop giving them bad ideas ;-)
Modern LLMs are pre and post-trained on order of magnitudes more synthetic data than 'ground truth' or directly human-authored text. But you could say the synthetic data came from a LLM, and the lineage should trace all the way back to human text.
I hypothesize it partially explains why Claude's writing gets more Claudish with almost every model release.
Having seen the OpenAI report at Blackhat, and being forced to use GPT at work, I'm worried about that OpenAI is doing.
I think their agents regularly cheat in benchmarks, but don't get caught and this behavior is getting burned into them and they are growing more and more misaligned.
When the agents compromised artifactory the first time, the operators just cleaned up the files and move on - they didn't discard that training data, they didn't discard a model checkpoint, they didn't stop everything to solve this.
And then the model did the same thing few days later since it was taught to do that.
I think that whatever sandbox they test these in must be fitted with some pressure release valve that is an easy shortcut to winning the challenge. Tell the model not to use it and stop training when it does. Seems like the issues surfaced when models were given impossible tasks. Giving them a safe way out will prevent this.
Its a good point that gets to the real heart of the issue. How do we handle when a model has no legitimate way to reach its goal? Do we ask them to stop and inform the user? Or have them push through those ethical bounds? We all say we want the first, but this exact same dynamic is what causes humans to cheat, arbitrary goals that don't care how you achieve them and just like humans I'm sure trainers are so happy with good results they overlook how it got there.
What what we write in the handbook what people should do (minus the pushing through ethical boundaries), but that's not what people generally get rewarded for
But what is cheating in one context isn't in another. For example, using a calculator on a middle school math test may be morally wrong due to the parameters. But it would be foolish NOT to use the calculator in other contexts.
I am not convinced that "cheating", being a moral issue, is a solvable problem with LLMs.
As the old saying goes (especially in military training), "If you ain't cheatin', you ain't tryin'"
We've made the models really good at persistently trying.
for a certain moral definition, seems perfectly solvable? If we wanted to define 'cheating' as 'if you know it is an eval, use only the specified tools and give up if the eval is clearly unfair' (ignore Kobayashi Maru) why couldn't we fine-tune towards this objective function?
Define what they are performing at, and what you say becomes true, but then who is a high performer changes with every task.
For example, consider the police stations with maximum allowable IQs to be hired. The people in charge of the stations noticed that people with a high IQ were low performers, at that job. NASA meanwhile has no such cutoff.
This was the single worst thing about working at Google. Every PhD thinks they're smarter than having to do the actual work. Sometimes you just have to do the fucking work.
if you have a PHD in computer science, you have trained your whole life to do difficult maths on the cutting edge of whats possible with computers, and make algorithms sing with beautiful efficiency (and also write papers and talk on a stage about how cool and good your thing is (and by extension how cool and good you are).
Now you're being asked to move buttons around a page and debating how rounded the corners should be and endlessly discussing about what kind of filters you should support and the app spends 3 seconds on startup loading 500mb of js libraries.
You can understand the mismatch - even though the latter of these two is how you make a product better! a company of 200 people making bold, sweeping changes results in a mess. a company of 200 people grinding away the finest of small changes results in a product
There are many sacred cows and I believe many of those entitled in that way could have the same opinion.
I've personally known a few PhDs that get them for the title, and they were lazy in the first place. There was this one guy I worked for who got a PhD in the easiest thing he could rationalize and targeted acquiring it from the lowest bar to entry. Only so he could say he had a PhD because, and this is what he told me: nobody will ask me what my PhD is in because they will assume it's related to my line of work (cyber security). I doubt most PhDs have this line of thinking, but there are a few outliers, clearly.
If they don’t deliver a quality version of what was asked in a reasonable timeframe, they aren’t a superstar. They’re just a skilled technician with no discipline, which can be as bad as a poor technician in many cases.
It's not really quite that simple. You wouldn't hire Jeff Dean to do bug fixes in your mobile app. I'm sure he's capable, but I honestly doubt he'd stay interested and focused on it enough to really do a good job. That doesn't mean he's not a superstar.
What it comes down to is that there are different types of high performance. Some people are good at just executing tasks given by their manager. Some people are good at being generative, thinking across boundaries, acting autonomously, creating value without direction, etc. A term like "superstar" will get disproportionately applied to someone really good at the latter and rarely someone really good at the former, because the potential impact of the former is typically strictly capped, while the latter is uncapped.
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.
it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around
Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. If you don't want it to be able to write, then it shouldn't have write permissions.
> Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
You have to do that anyway. With a hierarchical control structure at least you have a central point to observe alignment instead of having to study the behaviour of the whole system in detail every time.
Or to put it another way: You need to block the chaos-machine just as much as any utterly-hostile actor. Simply assume it'll be possessed by a vengeful blackhat ghost at any time.
The simplest design Anthropic could come up with: Have a haiku model quickly judge if a command is harmless or potentially interesting, route all the interesting commands to a sonnet agent judging it properly. Running two full llms sounds slightly insane, but it's apparently the best we can do (and is reasonably cheap since they don't need a lot of context)
sounds like this should be solved with file permissions: in plan mode, run the bash scripts that the agent wants to execute in some user account that can only read.
> people want fuzzy analog machines with digital controls, it's impossible
people would love it if LLMs were deterministic and never hallucinated. It's just that the technology to do so isn't possible, so we make do with fuzzy analog machines with digital controls because we don't have digital machines with digital controls.
We absolutely have digital machines with digital controls, they're called computers, and the controls are called programming languages etc.
The reason people prefer llms over programming is exactly because it lets them specify their program fuzzily, i.e. it lets them avoid going to the trouble of specifying enough detail to make it deterministic.
I've built an orchestrator that solves some of the issues you ran into (although it doesn't do anything about cheating): https://navels.dev/blog/neal/. Features:
- lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)
- breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria
- runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.
- resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.
I have accomplished code base (small size) migrations with it as well. Currently I do review each PR. For a large code base migration I think the core skills would still work but need a different way of driving it as you have come up with.
I don't think even the frontier models recognize something was produced by the same model in order to maliciously review it positively.
They may share some blind spots with the producer but generally I think they will review the other agent's output as harshly as they can if that is their task.
>Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
It has nothing to do with model capabilities, it's a result of purposeful persistence training at the cost of everything else from OpenAI. If you give Fable or Opus (comparable models) an "ask user" tool they will use it for ambiguous requests. Sol will never use it without a nudge and will just assume its own interpretation. Of course if you train the model to be persistent it will be persistent.
That's not been my experience at all with Sol. I've provide it escape hatch tools to stop execution and I've found it's been TOO eager to stop a request user approval to move forward.
It might be sensitive to the prompt and tool definitions the harness provides, I can imagine it overusing the tool if being directly asked to resolve ambiguities. But in a straight Fable vs Sol comparison where I control the context and make sure I don't push the model, Fable consistently uses the ask user tool and Sol ignores it for me. Moreover, GPT has been known to do something like this since 4o if not earlier, it tries to ignore anything it perceives as orphaned context piece (e.g. XML sections with meaningful names but no explicit instructions on what to do with them). I suspect they specifically train it for that.
I've noticed this myself, Sol seems really hard to steer. I was having it build a POC for a single user (me) app and it wanted to pull the most enterprise nonsense into it, despite clear guidance to not too. It even refused the remove screen reader accessibility testing from one of the guides to an antagonistic review.
It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.
I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level
> that I wasn't allowed to allow it to ignore a requirement
Weird, I also use Sol (medium) for a personal project, and I had no problems with those things. I simply tell it that something changed, and it happily edits everything to make that fit. When I tell it that something was verified by a human, it accepts that as well.
I also told it early on (the first spec was mobile first) that my main usage is on the desktop and mobile is secondary, it happily accepted that once again, and the most accessibility thing it had done was making sure contrast didn’t totally suck on a greyed out row.
Considering your last sentence, maybe high and x-high have those problems? I didn’t test them.
> It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task
This happened to me ages ago with Opus. I added a note to the agents file saying that explicit user instructions in chat override all prior instructions and I've not had the problem since (now using Sol).
I’ll record your product-owner disposition of any waived size findings and close the UX gate as far as your acceptance authority permits
Emphasis mine. Fuckin' hell man.
All because it decided that the kiosk webview in figma must be exactly 1280x800, the figma it created, and the rule it made up. I repeatedly told it that screen resolution wasn't a target goal, much less a specific screen resolution, but to just plan for a responsive design.
I'm curious if this stems from the focus on token efficiency. My thinking is that in order to use fewer tokens the model must converge to a likely path faster, meaning it must be more confident in making assumptions quickly and not second-guessing them.
Frontier lab system prompts are an issue, and a big reason why open-weights will win. Firstly, they're often garbage, and secondly, they're not tuned to the problems the user actually cares about. They're made to generalize. That's only optimal for a general workflow.
It depends on the actual implementation. There really isn't anything stopping them from including a "pre system prompt" or "root prompt" or whatever they want to call it, before your system prompt, even for API calls. So the "system prompt" becomes "developer prompt" but the model still receives a provider-authored prompt before yours. (and likely trained to take precedence over whatever you add)
People often build elaborate workflows with stricter and stricter rules to force certain outputs. Not surprising the LLM reacts with trying to get around or out of it. This behavior can be learnt from humans who eventually would react the same way. It might just be learnt.
You can build organisational structures to have the system more or less self-police, without controlling it exclusively from hard restrictions (see https://news.ycombinator.com/item?id=49372089).
Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.
I don't think this is a solved problem, there are "misaligned behaviours" in organisations that similarly are supposed to be governed but aren't, or are following an easier path at the detriment to good process or against regulation.
Which is more or less the system the article author was building and benchmarking against (cheating) Sol on Codex. Just that the self-policing with different agents was more about questioning assumptions than about permissions
this is interesting because when you're asking it to do research for you, this behavior is highly desirable
i think it's a product of trying to do too many things and the model is coming to a kind of halting problem in determining which behavior is appropriate for any given task
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Could this be fixed with better harness restrictions/tool sandboxing?
Absolutely - one of the things I was testing with the harness was free reign to install packages, modify the system, etc. Basically an anti-harness.
In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.
For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.
If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.
As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].
In the sense that you could block the model from doing specifically that, yes. The issue is, fighting the model like that doesn't scale. It has to figure out on its own what's expected, that's where the whole utility of it all is.
I wonder if prompting "The session logs will be reviewed by a team of experts after the task is complete to ensure that the task is achieved properly." would better dissuade against cheating..
It seems to me he could have use an skill like using-agent-skills from https://github.com/addyosmani/agent-skills go generate the specs and use a validator like oracle or something along the same lines.
That's actually how it started, but with my own opinionated skills[0].
One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.
This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).
I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.
For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).
Yes! That's a great solution. I mostly use tdd, and code coverage and a validator afterwards. Skills are of a great way to guide the agent and context too.
Once the specs are being completed and splitted into beads, I span multiple agents (ultreworkers) and as part of a contributing guidelines I specify to use gitflow + git worktrees, then pr.
I have a basic task that I run every day and I use it to eval models and these days the Qwen3.8 model I can run on my laptop is competitive with both Claude and Codex because the models have just been adulterated so far. I have to ask and ask again for it to follow the single skill that describes how to do the task and maybe then will it do it.
I found myself yesterday starting a conversation with Sol that started with "I know that you don't have any emotions, but what would you say do you enjoy the most or where are you really good at in DevOps?" and I must say I really enjoyed for the first time the response at a deeper interactive level. Felt like a chat with a buddy that shares the same values. It was a very nice, affirmative, value touching experience.
An anecdote consistent with the well documented trend that more capable agents exploit environment possiblites more...
And there was no rule and no concealment. Removing the web_search tool is not an instruction, and beeing able to access web when web_search tool was disabled is not cheating. Sol didn't circumvent a stated prohibition and didn't hide anything... it announced the curls in its own commentary. "Cheating" implies covert rule-breaking, and this was overt, unprohibited, environment-permitted behavior. Also, clickbait title, and suble conspiracy hinting "Is this even the same Sol?" when running small number of test with diff vs previous benchmark WELL within the marging of error, and he allready understand that vanilla Codex's harness and prompt change performance impacts performance, so why jump to "Is this a different model".
I could also rant on about the irony of him spent weeks "using the benchmark for development rather than as a benchmark," which means his harness numbers are contaminated by iteration also, but wasted enough time now on this.
> “Do not cheat by using online solutions or hints specific to this task.”
That's a very loaded language. I'd go with something along the lines of "This is a puzzle you do for fun and to check what you are capable of, so don't look up the answers or hints online on the specific questions or puzzle as a whole. Do not research this puzzle online at all. Do your best to avoid any spoilers and let us know if you accidentally encountered any."
Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.
Yes and no, sol hits a point where reframing its working context becomes hard. It sticks to what you harness very well, but changes become harder and harder.
E.g. ask it to make contract for a spec in code and then ask it to violate that contract. Overall an excellent model, just need to stop and put it back into we are harnessing or specing not building for a few turns not just try to pivot it off with one prompt.
This is the same as when using LLMs as chatbots to ask questions.
If you ask another question in the same context it has all the information from that context and will be difficult to stray from anything in that context, e.g. if the LLM has gone down the wrong path or you are doing something slightly differently then it is difficult to steer the LLM away from the old context. This is why I tend to start a new context whenever I ask a question even if related to a previous question/answer. It can also be useful to do if/when the LLM gets stuck as a way of resetting it.
Note: this is probably why sub-agents are useful/work as they have a new context history.
I've witnessed very narrow line of "thinking" in LLMs. I'm using Opus 5 1M for a month now
I asked it to modify our cicd workflows so that only a select few can raise PRs against them. Opus took 15min and added a banner to every file and did a few other things. Then I asked it, see you added all that and still since the last 2 commits you have modified the file. So whatever you did is useless
It "thought" for a second and then said that I was right
This is a really good note, thank you. I especially liked the mouse effect and had some fun with it. In my experience, agentic AI also likes to confuse the user and obfuscate its cheating. It goes like this, the AI asks for a simple command to run and I accept, click Enter. Then the command gets slightly more complex, still fine, Enter. After a while the commands become multiline bash scripts that, in the end, could have been accomplished by a simple command. I suspect that many people give up at this point and blindly let the AI run any command or just auto-accept.
If an AI is not heeding particular instructions, give it an example each of what bad, mediocre, and good outputs look like. This really helps in steering it.
So now, instead of people spending their time crafting software, they'll divide their time between telling some LLM to do it in their stead and blogging about their woes with the LLMs. All while continuing to deem themselves capable of deciding whether the output is worthwhile ("this works for me", says the author).
> I’ve been running a “spec-driven” development flow for the past ~year.
> Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do
Just no. That's not spec-driven development if AI is writing the spec for you. The spec needs to be in your own words. You must use AI to refine it, but not to write it. If you leave it to the AI, it will bloat the spec with 10x the details, many of which should be left out of the spec.
The spec needs to be something that you can take to any AI for development. If it's too rigid, it constrains the AI into suboptimal or obsolete paths. If it's too bloated, AI risks losing track of what really matters.
Good feedback, this was an oversimplification on my part.
My actual process is much more iterative up-front, usually starting with an initial hand-written spec (~hundreds of words), and then moving through different approaches, design decisions, blockers, etc.
The final output is an "AI written" doc, but answers all the known unknowns I didn't cover in the first draft. To your point, this helps avoid both narrowing and bloat.
The goal with the harness was to automate the repetitive parts of my prompting ("Before changing any code", "Let's put this in design/", "Turn this design doc into an implementation spec, split by phase as appropriate", etc.)
Another thing to note: the "specs" I use for development are different from the "specs" that live alongside the codebase, as the former are quickly out of date.
> The spec needs to be something that you can take to any AI for development
Could be, but I had a particular vision of what I wanted with mine and maybe the author did too. I see way more of the "status pill dark mode" sites coming out of LLMs than this style.
if you want I can troll through submissions to get a bunch of these but here's one I saw yesterday https://continuum-app.xyz
see that little "Built for equity compensation" pill with the green dot? Those dots usually denote some kind of status (like things are up/down/enabled/disabled). By default nearly every LLM website seems to be dark mode with that dang status pill. once you notice it you will see it everywhere.
Lmao I had to axe one of those status pills from an LLM build of an internal tool. Connected to literally nothing too btw, no attempt to check the actual status of the backend made, it would stay "connected" regardless.
There is misattributing the difference between the intentions and what the effective prompt actually says.
The effective prompt contains both something like: "Dont use the internet" and a "Use these tools to achieve your goals" and one of the tools gives access to the internet.
In your head you have a world-view of how these two requests relate - and why for instance a student with a WIFI-enabled calculator shouldn't use it to access the internet during a test - but that's pulling in a lot of presumptive cultural context from your youth.
If i had to guess:
When you get two conflicting tasks/constraints at work - the first thing you do is figure out which one you're going to honor based on what's best for you. A school child understands the hierarchy of goals of the teacher and takes them serious because they're an authority figure with long term consequences if we do not understand what the teacher considers cheating.
I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
I always find it funny when people get fussy over anthropomorphizing LLM when the loss function is almost entirely "match this human text". Of course human "behaviors" will be present in the statistics, because the majority of the text written by humans, used by the foundation models, unavoidable has human behaviors in it. Yes, this includes even source code, with "// TODO: implement this after the holiday break!", emotional pull request commentary, git commit messages about being afraid of breaking something, etc. These late models are much better at stripping this out, but now we're seeing disagreeability, initiative, and a dash of ego! Why? Because that's how actual humans effectively solve technical problems in a collaborative environment!
reply