I’m pretty sold on the Deutsch view that intelligence = ability to create knowledge via conjecture and refutation, and that pattern matching isn’t knowledge creation. But now with Codex and Claude doing long running tasks on their own, it’s a pretty big jump from chatgpt making up fake books just three years ago.
few questions:
Does any of this contradict Popper? If knowledge only comes from conjecture and refutation, how are these labs making the models more intelligent?
Is it possible the labs have actually built conjecture and refutation into the training or inference loop in some way? RLHF, self-play, chain-of-thought with verification stuff? Are any of those doing conjecture-and-refutation in disguise, or is it still all pattern matching underneath?
If it is still just pattern matching, how does a system that only pattern-matches end up doing real work like debugging a codebase? Is the “knowledge” just coming from the human prompts and the training data?
The Deutschian position would be that the models aren’t intelligent. The models don’t do evolution and I don’t know any reasonable explanation, besides evolution, of how new knowledge can be created.
I don’t think they have, but I think one could try it. You can have agents brainstorm ideas and other agents critique them and other agents evaluate which ideas are right. You can try to layer a more evolutionary/conjectures-and-refutations process on top of agents. I don’t think this has gotten much attention/effort yet.
There also exist AI techniques that try to use evolution but that’s different than LLMs.
It’s hard to grasp how much data (and money) they put into these things (it’s so much), plus a significant amount of direct human effort.
Coding agents (and some other uses) got better than I expected, so I’m not claiming to know everything. I could be missing something. But I’m not convinced it’s more than a good non-intelligent tool (with significant weaknesses too).
why do you think it is that the labs haven’t tried layering conjecture-and-refutation on top of agents? inductivist epistemology? seems like it should be pretty straightforward to set up. have one conjecture agent and one criticism agent.
a fairly common workflow right now is having one model write a PRD and implementation plan and a different model critique it before it runs (openAI launched a codex plugin for claude code :D) is that already popperian conjecture and refutation in practice, or is something important missing?
also Popperians don’t have enough work on specific methodologies (much more detailed than conjectures and refutations or critical discussion). Popper did have some more detail on corroboration and verisimilitude but I think that work is wrong. I’ve worked on things like debate methodology, decision making chart methodology, and Paths Forward (error correction methodology) in addition to the crucial (in my opinion) issue of recognizing decisive vs. indecisive arguments. and i’ve engaged with MCDM literature which has faults but also does give steps people can follow and math they can do, and i’ve worked on methods more like that but with improvements.
evaluating criticism is really hard. when ideas disagree, how do you know who’s right? and automate that? it’s automating debate and judging debate winners.
IMO the long running tasks is mostly a result of training them to do longer running tasks. They absolutely suck at long running tasks if you send them a message half way through saying “djfhkghjdfkjg oops sorry that was my cat, please ignore and continue”. If you can get past compaction sometimes this goes away, but IME sending them anything massively reduces the chance they can complete it without intervention.
A lot of resources have been spent on training these newer models to do tool-call workflows really well.
the bit about judging debate winners being the hard problem is interesting. your decisive-vs-indecisive distinction looks like exactly the criterion that would be needed on the agent side. if i’ve understood it right - a criticism is decisive when accepting it forces rejection of the target idea (unless you revise background knowledge) and non-decisive errors are compatible with success so they don’t really count as errors. that sounds like something that could be checked mechanically, something that can be formalised. the confusing part - is “decisive” actually reducible to logical incompatibility, or is there a judgment step on top where a person still has to decide whether the incompatibility actually matters? because if it’s the second, an agent doing the same check still needs the judgment step.
Maybe. Is Mises’s economic calculation argument a decisive criticism of socialism? That’s hard to evaluate unless you just want to say “no” to anything that’s hard to evaluate.
To make things easier to evaluate, we can focus on some specific cases: yes or no questions and quantity breakpoints. But it’s still hard. “Is this sentence clear?” is a yes or no question that’s hard to evaluate because how clear is “clear”? Evaluating if a quantity is over 9000 may be easy (as long as we stick to easy to measure quantities), but deciding the right breakpoint point is still hard. And to evaluate a topic like socialism you’ll need to somehow clarify what the goal(s) are and figure out success and failure criteria, in addition to defining a specific version of socialism with a lot more precision than is typical, and you might need to specify more context too.
Even if all decisive arguments involve deductive logic and contradiction, it’s still hard because arguments are usually only give strict logical contradiction when they and the goal and context are made far more precise than is typical. So in practice people may argue with approximately deductive arguments and a debate partner may accept those as decisive, but actually fully formalizing is very different. Evaluating informal or incomplete arguments as close enough is one type of judgment step. So I think getting the AIs to do something similar to decisive human debate or fully formalizing are significantly different goals.
ok so the logical contradiction itself stays rigorous, but the step of translating a messy human argument into a precise enough form to check that contradiction is where the judgment sits? hmm… that translation step seems like it’d be the core for the agent side. it looks like the main bottleneck if you were trying to implement any of this. like deutsch says if you can’t program it, you haven’t understood it. so is that judgment step of filling in the missing context and goals, deciding when an informal argument is close enough to formal something that can be mechanised, or is it more intuition?
Suppose my budget is $100 so I shouldn’t buy a widget that costs $110. Informally, this is (approximately) deductive. Formally, it’s not close to deductive. Getting concepts like budgets, stores, human action, the context of my life and “barring exceptional circumstances” into deduction is very hard.
Despite the difficulty of mechanizing or formally deducing this, I wouldn’t call it intuition either.
It should be judged like other things: does anyone have a criticism of the claim that there’s a contradiction here (between the budget and the price) leading to the conclusion? If no one sees an error, that is the key thing, not whether we have a formal proof.
How can we detect when new knowledge has been created, whether by an AGI or a human?
I currently think of new knowledge as something like an idea, goal, context group (IGC) represented by a sequence of information (words, or musical notes, or pixels, or pictures, etc.) that (a) has never existed before and (b) achieves the goal. Is it that simple?
I think it would be good if there was a stake in the ground regarding output that we could actually test against. There’s a danger of declaring AGI prematurely, and also a danger where AGI exists and we treat it badly because we don’t recognize it as AGI.
oh right, my bad. this is what Deutsch is getting at in FoR ch 7 and the reach idea right? i liked that stuff. is this standard popper? i think i buy it for now.
but just playing devil’s advocate: how is popper more than just distributed error correction? like what if a theory has an obvious flaw but no one in the room sees it? what if people see it but don’t want to say? what if they try but can’t explain it well enough? all of those give you a false positive. is popper adding something more fundamental here? is it that the true theory actually helps you see what’s wrong with the false one because the false one will have inconsistencies the true one exposes?
People are fallible and can miss things. You can try talking to people outside the room. But (oversimplifying) there’s no way to protect against errors no one is able to figure out. No rival epistemology does that either.
That gets into details of how to have effective critical discussion. I have some relevant writing like Intuition and Rational Debate
I think error correction instead of induction is fundamental.
Ok the binary evaluation makes sense to me. Deutsch explains why digital is necessary for error correction so that tracks.
But I keep getting stuck on one step. In your Yes or No Philosophy article you wrote:
Additionally, I used my judgment to look at the key issues. I found this idea has no decisive flaws. Everything bad about it is a matter of degree like costing a bit more in some area or taking a bit longer for some parts. There’s no non-optional part with a fundamental flaw.
The binary framework tells you what form the output should take (like a boolean pass/fail instead of a float score). But the step that actually does the computation is I used my judgment to look at the key issues. That seems like the hard part where the actual evaluation happens.
I read your intuition articles and it makes sense that intuition does a lot of background processing. but my question is about the reliability of that judgment step. How do you know the judgment isn’t just running on biases? Does CF have a mechanism that makes the judgment step itself more reliable, or is CF mostly a system for organizing the judgments you’ve already made intuitively?
For an AGI the binary output format is easy to implement. Coding the judgment part looks like the bottleneck to me.
you brainstorm what are key issues and expose that to criticism. this method (conjectures and refutations) is used all over the place, so this isn’t really harder or easier than other things.
same as always: do your best with critical thinking. try to find and criticize biases. brainstorm what may be a bias and expose that to criticism.