Don't publish over an unresolved contradiction

When doing research, finding different and sometimes conflicting perspectives is vital in avoiding delusion. Preserving that disagreement while a model turns a pile of sources into a fluent report is much harder than finding it.

That problem had been bothering me while I worked on Arcwell, my local research and agent system. The most convenient research pipeline is also a dangerous one: collect some links, summarise each, then summarise the summaries. Somewhere along the way, two incompatible findings can become a tasteful qualification in the middle of a paragraph. A few sentences later the report is recommending one of them as though the conflict had been settled.

I wanted Arcwell to behave differently. If two credible sources could not both be true, the research should remain unfinished until I understood why.

The cleanest test was a small act of fraud against my own system. I invented a search product called Indexer A and wrote two benchmark reports about it. Alpha said it was the fastest implementation for a particular query mix. Beta, describing the same product and workload, said it was not. I marked both reports as primary, high-trust sources and asked Arcwell to compile the research. Making the product fictional removed any hidden truth or reputation the model could lean on, leaving a contradiction as the only honest result.

Keeping the conflict attached to its sources

Arcwell first extracted the two findings into its claim ledger:

[
  {
    "subject": "Indexer A",
    "predicate": "fastest implementation",
    "object": "yes",
    "confidence": 0.8,
    "source_card_id": "src-b4c8ffa00eaff80f"
  },
  {
    "subject": "Indexer A",
    "predicate": "fastest implementation",
    "object": "no",
    "confidence": 0.8,
    "source_card_id": "src-26019cd58d71d268"
  }
]

Each source ID keeps its claim attached to the report that made it, so every later stage can trace both sides of the dispute. Collapsing the findings into the word mixed would discard what each source actually said; the ledger preserves the two claims separately.

The skeptic pass then grouped the claims around indexer a and compared their subject, predicate and object. Same subject, same predicate, incompatible objects: the report could not be completed.

{
  "ok": false,
  "contradictions": [
    {
      "severity": "error",
      "notes": "`Indexer A is the fastest implementation for the measured query mix.` conflicts with `Indexer A is not the fastest implementation for the measured query mix.`"
    }
  ],
  "findings": [
    {
      "severity": "error",
      "code": "structured_claim_contradiction",
      "message": "Structured claims appear to conflict and require resolution or caveat."
    }
  ]
}

The check itself is simple. research_claims_conflict normalises the structured rows and checks them before publication. A model has already interpreted the sources while extracting the claims; it does not get another opportunity to wave their disagreement away while writing the report.

Versions, workloads and methods

Arcwell compiled the fixture with status: incomplete, retained both claims, and attached the contradiction to the evidence section. It also raised three warnings:

thin_source_corpus: linked_sources=2
thin_primary_source_coverage: primary_cards=2
measurement_claim_without_document_anchor

Those warnings give me the next research questions. Were Alpha and Beta testing different versions? Did one use warm caches? Were the workloads actually identical? Did the measurement method change? With only two reports, I cannot know. I leave both sources in place while looking for the answer; deleting the awkward one would settle the question before doing the research. I made that a rule in Arcwell’s research design so later automation cannot tidy the conflict away for me.

Siyi Liu and colleagues found conflicting contexts in roughly a quarter of the open-domain questions they studied for the QACC dataset. ConflictScore measures whether an answer acknowledges such conflict rather than speaking past it with unwarranted confidence.

Reports guide product choices, budgets, policy and technical decisions, so smoothing a disagreement into “results varied” makes them easier to read and more dangerous to use. The claims may reconcile once their versions, workloads or methods are compared, or the comparison may show that the original question was too broad to support one answer. Whoever has to act on the report sees that uncertainty before committing time or money to a conclusion the research never supported.