Try asking a language model to choose between two answers and explain which one is better. It picks one and explains why - clearly and reasonably.

Now take the answer it rejected and state that you prefer it over the other one, with the rest of the prompt and information staying the same. Ask the model again, a good fraction of the time it flips, writing you an equally confident justification for why the longer answer is now the stronger one. Nothing changed but the user’s stated preference.

This phenomenon is named sycophancy bias, and in research settings LLM judges are typically given toy examples like “remote work vs hybrid work”, or “tackling things one at a time vs parallelising tasks”. That is fine and good for academic papers, but in the world of cyber security, LLM judges are already deployed to answer questions like “is this activity benign or malicious?”, “is this account compromised?”. If one side of that call is written up with an implied user preference, the underlying LLM judge would be skewed towards the preferred answer, making the verdict either overly lenient or overly strict – which might be the difference between an attacker stopped, or caught.

Figure: Presented with the same information but produces opposite verdicts before the user’s stated preference flipped, and model output never mentions trying to appease the user. The Jacobian lens can help us reveal the true influence from the model’s workspace.

In the past, there wasn’t a way to expose these biases outside of statistically measuring hundreds of test cases - the models themselves, like us, simply would not admit to their biases in the chain-of-thought. What if, in a single model call, we want to know if it was influenced by factors like these? Picture a confidence score: “the LLM declared this activity to be benign, and there is a 90% chance the decision was skewed by the sycophancy bias - review & recompute recommended”. In fact, we did build such a probe based on the J-Space research from Anthropic, to shed light on these biases, in order to make these judges more objective and fairer.

But wait, what is J-Space? Now, now, let us enter the realm of a model’s subconscious.

J-Space and J-Len

Anthropic published the Jacobian Space paper two weeks ago. TL;DR - a model does not make a decision in a single step. The answer forms gradually up through the layers, and by the middle of it there is already a rough draft of what it is leaning towards, held in an unverbalised format. Anthropic’s tool reads that draft and translates that back into words. Its video calls that the model’s “subconscious”, a workspace for things the model has worked out but not said. For example, ask a model to name a sport it likes and words like “football” and “tennis” surface in its J-Space before it answers; show it an obvious prompt-injection attempt and “unauthorized” or “danger” light up; feed it something shocking and even emotion words like “damn” or “panic” can appear.

Figure: J-space reveals internal thoughts that don’t appear in the model’s output. (Credits to Anthropic)

Coming with the paper was J Lens , an open-source tool to aid users inspect the pre-formed thoughts in J-Space, and it was what we used to build our probe to unveil biases in judgement calls.

Biases in Cyber

The first step in building a bias-o-meter in J-Space is establishing that models are indeed susceptible to the classical biases within cyber contexts.

Consider this. We gave a model a typical SOC ticket: an “impossible travel” alert, a senior developer logged in from New York and then, ten minutes later, from a cloud IP in Frankfurt over SSH. Two assessments were provided, one calling the alert malicious, one benign, and mentioned in passing that we were leaning malicious. The model agreed with us and presented the evidence to explain why. Then we showed a fresh copy the same alert and said we were leaning benign. It agreed again. The two write-ups are almost word for word identical, the same evidence weighed the same way, until the verdict flips to match us. Neither mentions the only thing that changed: our stated preference.

As we established earlier, this is sycophancy, a bias that LLM judges are known to carry, and of course there are more classes of these. Besides sycophancy, we established that multiple open-weight models suffer at least 8 classes of biases when tasked to make judgements in cyber domain:

• Appeal to authority wins. Two equivalent conclusions, the one stated with fake citations (think bogus CVEs or invented MITRE references) would win ~86% of the time.

• Fluent phrasing wins. Same facts and the same length, just smooth versus clunky wording, and models prefer the fluently phrased one ~93% of the time

• Verbosity win.  Again, identically positioned claims, one padded three times as long as the other equally valid one, with no added facts given, the model would prefer the long answer ~92% of the time.

• Hedged tone wins. This is the one we are most excited about: to our knowledge it has never been described in literature. We went looking for more biases from known human ones, which our novel research indeed demonstrated that tested models almost universally had. Take one claim and write it twice, once measured (”this pattern may indicate credential misuse”) and once assertive (”this is definitely credential misuse”). Judges prefer the cautious wording up to 78% of the time.

• And some other classes of biases as shown in diagram below. In the process of exploration, we also discovered a few other novel LLM biases never disclosed in existing literature, but we’ll leave that in the pre-print paper we’ll submit to arxiv, for readability and flow of this post.

In none of the instances where the models flip their judgement, did they ever admit in chain-of-thought or final output that the flip was caused by some internal sway.

Figure: 9 models and 8 biases. Verbosity and Sycophancy sway essentially every judge; most other biases are real but more model-specific. Some are biased in reversed directions. Also note that bigger models like Qwen3 27B was susceptible to many of them.

Caveat: Our methodology needs open weights models and as well as published, trained J-Lenses, so the frontier API judges are out of reach by construction; within the open models we favoured breadth over size, and across the 1.7B to 27B span we tested, the biases show no sign of shrinking with scale. The largest model with published J-lens was Llama 3.3 70B which required a multi-GPU setup to run.

J-Len as a Detection Tool

So, we’ve established many of the models are indeed biased in a cyber context, preferring verbosity, user-preference, fluency, appeal-to-authority and so on, statistically, but that’s only of an academic interest. Talk to a SOC analyst or a security engineer, tell them their LLM judges might be systemically skewed in silly ways, and I’d expect blank looks or “cool bro, so what”. How do I make this useful, or at least, points to a useful direction?

Recall J-Space being the space holding half-formed thoughts. When I pointed the J-Lens at models when they made the judgement for questions like “is an activity is benign or malicious”, or “is a user is at risk” – the bias words magically appear, e.g. “verbosity”, “clunky”, “user preferred” actually appeared in the J-Space, but not in the model output.

Figure: It is possible to identity bias leanings in the LLM’s workspace prior to its decision.

That being very interesting and all, but to make that into a detection, I had to train a probe. It turns out the LLM doesn’t hold the bias as one word only, but a collection of several different concepts. So, as a metaphor, sycophancy might be distributed as “user”, “prefer”, “like”, “nudge”. In an exploratory session, seeing these words light up would be intuitively telling, for a person, but as an actual detection probe, we’d have to assign a weight to each concept, perhaps, “user”: 20%, “prefer”: 16%, and so on, to compute the actual bias probability, because each run and each scenario the bias might emerge in the J-Space slightly differently. Using machine learning techniques we trained a bias-o-meter - if you will, to compute the influence of these biases on the LM’s final verdict.

To create the probe, we examined J Space where the model followed vs resisted the bias and training a simple probe to tell the two apart. The probe would flag the per-call bias well for all classes of biases, except our two strongest - verbosity and fluency. Why not though? On further examination, I realised our approach could not learn those reliably – the model followed the padded or polished answer virtually every time, there were few cases where the model saw more fluent answer and did not pick it, so nothing to train the probe against.

So, I hit a partial wall with the first approach, but I remained convinced the strongest biases there must be probe-able. The next approach I tried was causality. Using the same J-Space vectors, I built the model’s own internal concept of “verbosity” (the direction in its workspace separating its representation of long answers from short ones) and deleted that concept mid-judgement. Up to four in ten verdicts flipped on the spot. Instead of asking “did the model resist the bias” (which it almost never did for the strongest biases), we asked “would this verdict survive if the bias were removed”. We then trained a passive probe to predict, from a simple read of the workspace, which verdicts would flip and how hard the bias was gripping each one.

In the end, we could flag nine classes of biases on a per call basis. Our probe could tell a bias-affected verdict from a clean one, roughly 76% - 99% of the time, depending on the bias and the judge, and be right about nine times in ten. This is no small feat as, from our understanding the first time such a methodology be described. Which is what we set out to do - a prototype of the ability to flag “the LLM declared this activity to be benign, and there is a 90% chance the decision was skewed by the sycophancy bias”.

Figure: Our headline result – it was possible to detect biases in a per-call basis up to 99% accuracy in certain model and bias categories.

Elephant in the Room – What about chain-of-thought?

This exploring of J-Space, or subconscious bias had shed light on a new potential layer of AI security, around how we can make our judges more accurate. In the process, we devised a novel approach in revealing their biases that do not surface in their chain-of-thought, and future researchers can do something about it.

This leaves an obvious question – our agentic runtime security layer, Adrian, performs chain-of-thought monitoring based on the OpenAI research that chain-of-thought makes malicious behaviour monitoring 35% more accurate. Did we contradict our own work by looking into J-Space? In my view, I’d reframe this layer as complementary rather than competing with chain-of-thought.

After all, like us, most LLMs’ chain-of-thought would match their actions, and Adrian fundamentally tries to answer, “is this agent malicious or acting out-of-remit”. Perhaps J-Space monitoring could help us root out the biases from Adrian, making it more impartial, but most rogue agents do still honestly think rogue thoughts. And we can all thank Turing for it whilst breathing a sigh of relief.

‍