# AGI, LLMs, CR and CF
**URL:** https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169
**Category:** Unbounded
**Tags:** ai
**Created:** [April 10, 2026, 10:47pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169 "2026-04-10T22:47:50Z")
**Posts on this page:** 20
**Page:** 3
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 13, 2026, 6:49pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/41 "2026-04-13T18:49:35Z")
Do you think they are logical contradictions?
---
### Author:  [@Oracle](https://discuss.criticalfallibilism.com/u/Oracle)
#### Post date: [April 13, 2026, 7:15pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/42 "2026-04-13T19:15:45Z")
no, not logical contradictions. i just read the article fully (hayek and mises are on my to read list so maybe i’m misunderstanding terms like minarchy) and the first one still seems like a real problem to me even if it’s not a formal contradiction. if the government is also staffed by irrational people, how do we not just run into the same problems on the opposite side? more vigorous policing by flawed people could mean more vigorous corruption or abuse of power.
---
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 13, 2026, 7:46pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/43 "2026-04-13T19:46:00Z")
@Eternity @Max or anyone else want to try to analyze this as a practice problem?
---
### Author:  [@Lebowski](https://discuss.criticalfallibilism.com/u/Lebowski)
#### Post date: [April 13, 2026, 9:57pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/44 "2026-04-13T21:57:54Z")
> [@Elliot](#):
>
> > [@Lebowski](#):
> >
> > How can we detect when new knowledge has been created, whether by an AGI or a human?
>
> I primarily look at the methods used, not the output.
Intuitively, making judgments about potential intelligence based on methods seems worse/riskier to me than judging outputs. At least partly because the outputs of humans and of other potential intelligences are more directly accessible than their methods.
We might be wrong about how humans create new knowledge, and we might be wrong that the way humans create new knowledge is the only way it can be done.
I’m not saying I think we are wrong. If I knew a criticism of Popperian epistemology I’d say it, and I don’t.
And I also think we might be wrong about what is and isn’t (or could be vs. couldn’t be) going on inside a sufficiently deep neural network.
---
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 13, 2026, 11:38pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/45 "2026-04-13T23:38:18Z")
> [@Lebowski](#):
>
> Intuitively, making judgments about potential intelligence based on methods seems worse/riskier to me than judging outputs. At least partly because the outputs of humans and of other potential intelligences are more directly accessible than their methods.
Here’s a risk of looking at outputs:
One person writes a new book.
Another person copies the book, shape by shape. They aren’t even literate.
Same output.
---
### Author:  [@Jarrod](https://discuss.criticalfallibilism.com/u/Jarrod)
#### Post date: [April 14, 2026, 12:55am UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/46 "2026-04-14T00:55:00Z")
> [@Elliot](#):
>
> @Eternity @Max or anyone else want to try to analyze this as a practice problem?
I haven’t done a detailed analysis (and it’s been a long time since I read ET’s article), but a few thoughts occurred to me as I read Oracle’s post:
- Incentives could change if companies were policed better. (Inadequately policed companies might see no downside to committing fraud.)
- It’s been a long time since I read ET’s article, but I feel like a goal of ET’s article was to warn people (especially libertarians, Objectivists, etc.) against being biased in favor of big companies. Not to present policing fraud as a panacea.
> [@Oracle](#):
>
> a state that needs to constantly outmaneuver billionaire-funded corporate adversaries doesn’t seem minimal.
- I think ET is in favor of a government that is limited to the task of defending people’s rights, not a government that is so minimally funded that it’s incapable of doing a good job of defending rights. So, limited/~“minimal” in terms of the scope of what it does (it just protects rights) but not “minimal” in terms of how robust/effective/well-financed it is in pursuing that goal.
> [@Oracle](#):
>
> your own premise (creative adversaries beat fixed rules) seems to undermine your own proposed solution (minarchist rules enforced better).
- If new loopholes or edge cases are discovered, new fixes can be devised. (Though the basic moral principles would presumably stay the ~same.) Similar to how software developers can patch newly-discovered security flaws. It will always be the case that new problems can be discovered and new solutions needed (cf. the idea that we’re always at the beginning of infinity).
---
### Author:  [@Lebowski](https://discuss.criticalfallibilism.com/u/Lebowski)
#### Post date: [April 15, 2026, 2:19am UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/47 "2026-04-15T02:19:02Z")
> [@Elliot](#):
>
> Here’s a risk of looking at outputs:
>
> One person writes a new book.
>
> Another person copies the book, shape by shape. They aren’t even literate.
>
> Same output.
I don’t understand your point. One criteria for judging output I already mentioned would be that the output is unique, which the copy would not be.
That’s not enough of course, since a machine could ex: copy + insert some random characters to make it “unique” but that wouldn’t be new knowledge. I think I agree with your earlier statement:
> [@Elliot](#):
>
> One issue is you’d have to check for similarity to other knowledge, not just uniqueness. But evaluating similarity is fundamentally hard.
Even if it’s hard, at least we can look at a human output and a machine output side by side and judge whether their dis-similarity exceeds a breakpoint (dis-similarity being just one criteria among multiple required to qualify an output as new knowledge).
OTOH I don’t know how methods which arise outside of an explicit process/program could be observed and compared at the right level of abstraction. I don’t think we could meaningfully compare methods by ex: looking at a series of human brain scans or neuron firings and a series values from a machine’s deep network propagations. And I don’t think anything at a higher level of abstraction is available.
I think another reason for my intuition to judge outputs rather than methods is how we evaluate the intelligence of humans vs. animals. I think we (correctly) judge that humans are intelligent and animals aren’t based on things that humans regularly output (like symphonies and plays and technical diagrams etc.) that no animal ever has. We didn’t get to that conclusion by first figuring out epistemology then figuring out that animals are using different methods, right?
---
### Author:  [@Eternity](https://discuss.criticalfallibilism.com/u/Eternity)
#### Post date: [April 15, 2026, 3:00am UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/48 "2026-04-15T03:00:26Z")
I’ll try. Intentionally not putting too much effort into this to get me to post. Probably going to try and do some intentionally low effort posts tomorrow morning (was studying for an exam and working past two days) to get back into posting more. I think it helps me to acknowledge to myself that a post is going to be low effort to be comfortable with it and also letting others know that’s my goal.
* * *
> [@Oracle](#):
>
> through the harness.
What’s a harness? Are you talking about AI here? I haven’t kept up with the discussion in this thread and, uhh, ctrl+f has not been helpful. Since, I guess, discourse loads parts at a time of the thread, so I couldn’t see how it was used earlier that easily.
> [@Oracle](#):
>
> you write:
>
> > “The people who run big companies are similar to the people who run the government. They aren’t significantly better people.”
>
> > “Most people are pretty bad. The main types of badness I have in mind are irrationality and dishonesty.”
Small thing, but the block quote is already a quote adding those quotes around it makes it seem like in Elliots article that was something he wrote in quotes.
> [@Oracle](#):
>
> if the government is staffed by the same irrational, dishonest people who run the companies, what makes the government capable of policing them effectively? the essay says the fix is education via your philosophy work, but that means the policy proposal has a precondition that doesn’t exist yet. is the harness missing a mechanism you had in mind, or is the actual claim “more policing would be good _if_ we first fixed the people problem”?
? So idk if there was any prior discussion on this (I assume not much since the topic is primarily AI and stuff in this thread), but wdym by if staffed by same people? From the essay:
> It’s because companies are run by people, and lots of people are bad. Changing the government won’t make company leaders into good people. That requires education.
A lot of people are bad. What would make the government effective at policing these companies if they are bad? Is that what you’re asking? Since their both bad? I don’t think Elliot is claiming that policing will be effective. Regular stuff that is already being policed is not really that effective from what I know. I think part of the point of the essay is that big companies are bad and that’s it. When speaking to an average freedom minded person they think that big companies are good and that a big serious issue is getting the government away from them so they can do whatever they want. Elliots essay aims to point out that is not true. Big corporations suck. A freedom minded person should want government to be more involved with corporations. In the correct way of course i.e. policing fraud not regulating the size of their drinks (idk).
> [@Oracle](#):
>
> but a government that must constantly adapt its rules to outmaneuver well-resourced corporate adversaries is running an open-ended regulatory arms race
afaik that kind of thing is already happening (idk if you were claiming it isn’t or of it is)
> [@Oracle](#):
>
> you call this a minarchy earlier in the essay, but a state that needs to constantly outmaneuver billionaire-funded corporate adversaries doesn’t seem minimal. your own premise (creative adversaries beat fixed rules) seems to undermine your own proposed solution (minarchist rules enforced better).
you later said:
> [@Oracle](#):
>
> i just read the article fully (hayek and mises are on my to read list so maybe i’m misunderstanding terms like minarchy)
i think you are. minarchy usually refers to the scope of what the government covers(?)/does(?). the usual minarchist idea is that “all” the government does is policing/protecting kind of stuff. so they protect you from violence, fraud, and other rights violations but it doesn’t step into, let’s say, do stuff like taxes to reduce smoking, or give welfare or subsidies. The government just protects. I think a government that just protects can be in line with creative judges.
---
### Author:  [@Oracle](https://discuss.criticalfallibilism.com/u/Oracle)
#### Post date: [April 15, 2026, 4:44am UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/49 "2026-04-15T04:44:50Z")
thanks, the minarchy point makes sense. i was conflating scope with funding/robustness.
but i still think the first issue needs clarification. eternity’s reading seems to weaken the essay into a normative claim about what a freedom-minded person should want. the essay itself looks stronger than that. it says “part of a capitalist society would be a better, more effective government”, says “our government’s failure to police fraud well is a huge deviation from capitalism which a free market society would fix”, and says companies would be policed “more vigorously” than today.
so is Elliot’s current position that the essay was only making a normative point, not a functional one? if yes, does he retract those stronger claims? if no, what is the mechanism by which government enforcement works better even though the essay also says the people in government and companies are similar and not significantly better people?
---
### Author:  [@Oracle](https://discuss.criticalfallibilism.com/u/Oracle)
#### Post date: [April 15, 2026, 5:36am UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/50 "2026-04-15T05:36:11Z")
completing above:
that’s an interesting distinction. ideas represented as tokens vs ideas represented as actual data structures with functions that can apply one idea as a criticism of another.
but i think substrate independence cuts against this as well. we don’t know what the functional relationships between neurons and ideas actually are, and tokens could be doing something similar.
also evolution creates knowledge without anything like conscious word-idea correspondence, and deutsch counts it as genuine knowledge creation because the error correction mechanism is real regardless of substrate. so the question for LLMs isn’t whether there are ideas behind the tokens, it’s whether there’s a real error correction process. for systems trained with ongoing feedback from reality, like RLHF, is that obviously absent? or is it optimization against a fixed reward signal rather than genuine error correction through criticism?
---
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 15, 2026, 5:06pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/51 "2026-04-15T17:06:11Z")
> [@Oracle](#):
>
> we don’t know what the functional relationships between neurons and ideas actually are, and tokens could be doing something similar.
We have Popperian explanations and arguments that give us broad guidance on what could or could not possibly work. Evolutionary processes can create knowledge. Non-evolutionary processes can’t. If there isn’t replication with variation and selection, then there isn’t knowledge creation.
I don’t think it’s plausible that LLM tokens are doing replication with variation and selection because we know what algorithms LLMs are programmed with.
> [@Oracle](#):
>
> so the question for LLMs isn’t whether there are ideas behind the tokens, it’s whether there’s a real error correction process. for systems trained with ongoing feedback from reality, like RLHF, is that obviously absent? or is it optimization against a fixed reward signal rather than genuine error correction through criticism?
RLHF happens during training but not when the AI is responding to your prompts. So even if RLHF involves error correction and knowledge creation, that wouldn’t mean the AI can create knowledge when you chat with it.
---
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 15, 2026, 5:15pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/52 "2026-04-15T17:15:39Z")
> [@Lebowski](#):
>
> > [@Elliot](#):
> >
> > One issue is you’d have to check for similarity to other knowledge, not just uniqueness. But evaluating similarity is fundamentally hard.
>
> Even if it’s hard, at least we can look at a human output and a machine output side by side and judge whether their dis-similarity exceeds a breakpoint (dis-similarity being just one criteria among multiple required to qualify an output as new knowledge).
Do you know a way to do that? How do you evaluate similarity? And how do you determine a breakpoint?
---
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 15, 2026, 5:41pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/53 "2026-04-15T17:41:52Z")
> [@Eternity](#):
>
> I haven’t kept up with the discussion in this thread and, uhh, ctrl+f has not been helpful. Since, I guess, discourse loads parts at a time of the thread, so I couldn’t see how it was used earlier that easily.
You can get a full topic on one page for searching:
> [@Guide: View Entire Topic on One Page or Export Topic Content](https://discuss.criticalfallibilism.com/t/guide-view-entire-topic-on-one-page-or-export-topic-content/1729):
>
> Want to get a whole topic into a browser window or document instead of using the standard infinite scrolling with autoloading? That can help with searching, skimming or archiving the text. Here are two options: Printable Page You can press cmd-p (mac) or ctrl-p (windows) to bring up a printable page in a simplified format. Just close the print dialog and the page should still be there, fully loaded and readable in your browser. You can also print to pdf (at least on mac; I’m not familiar wit…
> [@Eternity](#):
>
> lot of people are bad. What would make the government effective at policing these companies if they are bad? Is that what you’re asking? Since their both bad? I don’t think Elliot is claiming that policing will be effective. Regular stuff that is already being policed is not really that effective from what I know. I think part of the point of the essay is that big companies are bad and that’s it.
I think we could do much more effective policing of companies without people getting better or more rational, if we just wanted to and most people agreed on that goal. Like if 90% of people found my essay persuasive and agreed about most of the examples I’ve gathered over the years, then I think the existing justice system is capable of doing more. If it was a priority, they could catch more cases and have more serious consequences when companies are caught.
I think the main blocker right now is _political opposition_ not rationality or skill. One reason for the opposition is pro-company biases from Republicans and libertarians, some of which I think are [connected with Milton Friedman](https://discuss.criticalfallibilism.com/t/milton-friedman-and-maximizing-profits/1991). A second reason for the opposition is that a lot of anti-company type people, who want to reign in companies, go too far in various ways, wanting policing and other policies which are not permitted by classical liberalism, libertarianism and minarchy. And they do things like use anti-trust law to try to police companies, which offends libertarians who think anti-trust is bad on principle.
One of my other relevant thoughts is that policing companies better (preventing them from aggressing) is a higher priority than shrinking government and lowering taxes. I think most (non-anarachist) libertarian types prioritize the other way and I think they’re making a mistake and not taking aggression by companies seriously enough as an important, widespread problem which it its the government’s job to stop. (Anarchists should care about defense too but there are various complications with them that I’m not going to discuss now.)
---
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 15, 2026, 5:44pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/54 "2026-04-15T17:44:07Z")
> [@Oracle](#):
>
> if the government is staffed by the same irrational, dishonest people who run the companies, what makes the government capable of policing them effectively? the essay says the fix is education via your philosophy work, but that means the policy proposal has a precondition that doesn’t exist yet.
My text [says](https://www.elliottemple.com/essays/capitalism-means-policing-big-companies):
> Just reforming government wouldn’t fix society because it wouldn’t fix the bad people problem… The underlying issue is irrationality…
>
> It’s not primarily the government’s fault that many companies are bad. It’s not just due to having a mixed economy or government interference in the economy. It’s because companies are run by people, and lots of people are bad. Changing the government won’t make company leaders into good people. That requires education.
You can make and enforce laws without first figuring out how to educate everyone to be saints or geniuses. Turning CEOs into good people, morally and rationally, is a different issue than reducing the amount of fraud in the economy. It’s not a precondition.
The government doesn’t need to be better people to enforce laws more/differently, similarly to how CEOs don’t need to be better people to stop committing egregious fraud. CEOs will mostly stop if there’s enough law enforcement. The government can enforce these laws better if most people inside and outside of government want them to and care (wrinkle: it could be hard if most people want it but most very rich people oppose it).
> [@Oracle](#):
>
> a government that must constantly adapt its rules to outmaneuver well-resourced corporate adversaries is running an open-ended regulatory arms race. you call this a minarchy earlier in the essay, but a state that needs to constantly outmaneuver billionaire-funded corporate adversaries doesn’t seem minimal.
A minarchy is minimal in the sense of doing what’s needed to stop the initiation of force but not doing extra things like education, healthcare, and social security. If being large is necessary in order to have a free market, then large could be minimal. Minarchy means the government handles a minimal set of jobs and generally tries to be small but it doesn’t have an actual size requirement. If we colonize a trillion planets then we could have a much larger government that’s still a minarchy.
Also, I just said it takes some ongoing creativity not perfect laws that you just write once and never touch. That doesn’t mean constant adaptation. Once we got things under control and companies got used to participating in the market without aggressing, and courts and police got set up for that and used to it, then I think a much smaller government could maintain things and make some adjustments sometimes.
---
### Author:  [@Oracle](https://discuss.criticalfallibilism.com/u/Oracle)
#### Post date: [April 15, 2026, 5:56pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/55 "2026-04-15T17:56:35Z")
thanks for clarifying both points.
this shows harness was overstating the issues. on the second one, i was misunderstanding minarchy by treating it like a size constraint rather than a scope constraint. on the first one, i was reading the essay as if education had to be a precondition for better enforcement. your answer is that reducing fraud doesn’t require first making either government officials or CEOs into better people, it just needs enough support and priority behind enforcement.
this supports your earlier point about LLMs pretty well. the harness was useful for surfacing candidate tensions, but it didn’t understand the conceptual structure well enough to classify them correctly. it was seeing contradictions where there weren’t any, and it needed human judgment to sort that out.
---
### Author:  [@Oracle](https://discuss.criticalfallibilism.com/u/Oracle)
#### Post date: [April 15, 2026, 6:01pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/56 "2026-04-15T18:01:07Z")
i think there may be two different claims here.
one is that a frozen deployed LLM doesn’t create knowledge while responding to prompts. i think i understand that claim.
the other is that the broader training process also isn’t a knowledge-creating evolutionary process. that part still isn’t clear to me. modern training seems to involve variation, selection, and feedback over many candidate behaviors/policies/weights, even if it’s not conjecture and refutation in the strong deutschian sense.
so are you claiming:
1. no knowledge creation happens at chat time, but some could happen during training, or
2. no knowledge creation happens anywhere in the LLM pipeline?
---
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 15, 2026, 6:11pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/57 "2026-04-15T18:11:04Z")
I was talking about chat time.
The parts where a human is involved certainly have some potential to create knowledge since evolution and error correction happens in human brains there. What’s going on there is a more complex analysis. RLHF is one example of where human thinking is involved, and there are more like designing and coding the LLM in the first place, gathering and curating training data, and writing the system prompt and harness.
To be an AGI, the LLM has to be able to create knowledge by itself, with no human creativity involved anymore while it runs, only while creating it. Humans writing chat prompts can actually create knowledge and give it to the LLM too, but if we use limited prompts that aren’t very helpful like “do philosophy by yourself and share the results” we should generally be ok.
---
### Author:  [@Oracle](https://discuss.criticalfallibilism.com/u/Oracle)
#### Post date: [April 15, 2026, 6:20pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/58 "2026-04-15T18:20:53Z")
i can see how idea, goal, context triples help turn a lot of vague degree questions into clearer pass/fail ones. but it still looks to me like the hard part is the same one as before: deciding whether a criticism really refutes the IGC in that context, or whether the IGC already addresses it using existing knowledge.
so is CF giving a fundamentally new way to compute that judgment, or is it still C&R all the way down, with the additions being mainly organizational tools that help structure the judgment better?
---
### Author:  [@Oracle](https://discuss.criticalfallibilism.com/u/Oracle)
#### Post date: [April 15, 2026, 6:25pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/59 "2026-04-15T18:25:50Z")
stockfish seems like a useful analogy here. it doesn’t create new knowledge by itself once deployed either, but a lot of human knowledge goes into designing it, and the resulting system still embodies that knowledge in a usable form.
so is your view something like that for current LLMs too? not creators of new knowledge at chat time, but systems that can embody and apply knowledge created upstream by humans during training/design?
---
### Author:  [@Elliot](https://discuss.criticalfallibilism.com/u/Elliot)
#### Post date: [April 15, 2026, 6:56pm UTC](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169/60 "2026-04-15T18:56:55Z")
> [@Oracle](#):
>
> so is CF giving a fundamentally new way to compute that judgment, or is it still C&R all the way down, with the additions being mainly organizational tools that help structure the judgment better?
CF has, IMO, major conceptual changes to CR (not just organizational tools) like:
- using decisive arguments and binary evaluations
- evaluating ideas for goals instead of in isolation
- breakpoints and excess capacity
I think this is substantial progress in epistemology. But thinking is still mostly CR and coding AGI is still hard with these changes.
> [@Oracle](#):
>
> so is your view something like that for current LLMs too? not creators of new knowledge at chat time, but systems that can embody and apply knowledge created upstream by humans during training/design?
yes
[Previous page](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169.md?page=2)
[Next page](https://discuss.criticalfallibilism.com/t/agi-llms-cr-and-cf/2169.md?page=4)