i can see why you’d call those conceptual changes rather than just organizational tools. changing the evaluation target from isolated ideas to idea-goal-context, and changing the output from analog to binary, are both real changes in how the problem is framed.
but i still don’t see the answer to the bottleneck question. which of those changes gives a fundamentally new way to compute whether a criticism decisively refutes an IGC, as opposed to making the inputs and outputs cleaner while leaving the actual judgment step mostly where CR already had it?
your last sentence seems important: if thinking is still mostly CR and coding AGI is still hard with these changes, then that suggests the main bottleneck is still largely where it was before, and the changes are doing more to reframe the evaluation than to solve the core judgment problem.
Yeah I don’t think CF solves that. I’m not claiming to have a solution to AGI!
Since we can’t code AGI, there can be dozens of blocking problems, not just one bottleneck. A functioning chain has a weakest link, but a broken chain may have many broken links. I think I solved several relevant problems but there are still many left.
I expect there to be considerably more progress in epistemology before AGI is developed. I don’t think epistemology is advanced enough to just apply it now and create AGI. I consider epistemology research highly valuable for many reasons including AGI relevance, and also useful epistemology progress has historically been scarce which raises the importance of CF more.
lol yeah fair enough. if you had solved AGI already, this thread would probably be going very differently.
i meant the narrower point: if CF does not solve the hard judgment step, then is its contribution mainly in how it frames and structures criticism, while the actual deciding of who is right when criticisms conflict is still basically ordinary CR?
like with Paths Forward, decisive vs indecisive, breakpoints, IGCs, etc, are those best understood as useful ways of organizing and clarifying the problem, rather than as a breakthrough on the actual judging step itself?
Sort of. I see what you’re saying and I partly agree, but judging pass/fail instead of e.g. a real number from 0 to 1 is a change to the judging step itself. Judging an IGC instead of an IC (taking into account context isn’t new) is also part of the judging step. But there is a part of the judging step that’s basically still evolution/C&R as before (but there’s also recursion there as you consider arguments, counter-arguments, counter-counter-arguments, etc, so the changed parts can come up again as you recurse).
You can look for lower hanging fruit, easier places to make progress. Those are often good because you get benefits faster and that progress often ends up being relevant to harder things and making them easier.
You can consider what you’re good at or what you enjoy and work on that.
You can look at incentives like money and fame and prioritize based on those.
If you have to solve 10 problems before X will work, that doesn’t mean each solution is useless alone until you have all 10 solutions. They help with your understanding which can help creating other solutions. But also, in general interesting solutions help with many problems. You can start with something that will be useful for something else even without solving the other 9.
You can look at what other people are working on. You may want to collaborate with others. Or you may want to work on something with low competition so you have a better chance to make a big contribution yourself. Or you may want to accelerate the solution of the one you think may be solved last by other people since it’s getting the least attention.
i see. i think the part i was still trying to isolate is which part of that is a real improvement to the judgment itself, and which part is still downstream of the older CR problem of noticing and evaluating errors in the first place.
like it sounds as if your view is not that CF leaves the judging step untouched, but that it partially reshapes it while still inheriting the hardest part from CR. is that fair?
yeah. though CF is pretty new and should be tried out by more people before reaching a lot of conclusions about it. and when that happens, it may turn out that it was already 90% of the way to some additional discoveries that have become low hanging fruit and were maybe already implied by it. CF hasn’t yet stabilized where you could say we’ve exhausted the consequences yet. It’s hard to know what more could come just from working with the existing main ideas more without any new big ideas.
do you think the hard part of judging criticism is hard in something like a godel/turing sense, where there just is no general procedure for it, or more just that our epistemology is still not developed enough yet?
I think I and most people can determine similarity reasonably well, though inexplicitly / intuitively rather than with an explicit process. It’s reasonably common for example to see a movie and recognize the story as ‘just a rip-off’ of some other story you’ve seen before, rather than a new story. That’s true even if lots of details (names, setting, etc.) are changed in the ‘rip-off’. In that case the judgment that the story in a particular movie is a ‘rip-off’ vs. something that (like all movies) bears some similarity to other movies is a breakpoint judgment.
The patent system has an explicit legal process for judging dis-similarity (in patents it’s called ‘novelty’). It is a breakpoint process where an invention is either declared novel and gets a patent, or it isn’t and doesn’t. I don’t know many details other than that. I’d guess like most legal processes it is a combination of the application of formal standards in law, case law, and inexplicit judgment about the specific instance. My understanding is the barrier to machines being inventors on patents is procedural (the law explicitly requires inventors to be human) rather than any actual or contemplated problem with determining if an invention generated by a machine is novel.
DISCLAIMER: Both of these examples have flaws and I don’t think they are drop-in ready for the specific goal of evaluating whether a machine has generated new knowledge of any arbitrary kind. But I do consider them both good leads and an indication that such an evaluation is a practical possibility.
I think judgment involves replicating, varying and selecting millions of ideas. I don’t see a deep principle in our way for formalizing this, just a ton of complexity.
That’s for high level judgment like people use in their lives. How does the selection work at a low level in simple cases subconsciously? What does the brain actually do? I don’t know the details. That’s partly epistemology and partly neuroscience. I’d guess there’s more than one way it could work, more than one viable algorithm, so what we actually use is a scientific matter.
One thing I imagine is we have is an efficient way of taking a bunch of criticisms we already know, quickly estimating which are relevant, and then checking if a lot of variant ideas are refuted by any of the criticisms. In this case, the way you choose between the criticism and the idea it contradicts is simple: always favor the criticism. It’s a pre-existing idea you like. You’re just checking whether it applies, which is a relatively simple type of judgment compared to the general case of evaluating two contradictory ideas. So for example if you already think traveling to other continents is bad, it’s relatively easy to check which ideas involve traveling to another continent or not.
I think we must be generating millions of ideas at a low level and filtering most of them out before they get conscious attention.
You can set aside or reconsider criticisms if you want to but you do that separately, not in the middle of judging other ideas with it. You use some ideas to evaluate other ideas. You’re always using some ideas as premises that you don’t question right now. But any of the ideas can be questioned and evaluated, just not all at the same time.
This isn’t a full solution. First it’s just a general idea, not all the details. Second, there should be some other simpler mechanisms that help you reach the point of knowing a bunch of pretty good criticisms.
OK so judge similarity with conjectures and refutations. I don’t see how this is much help compared to directly judging knowledge creation with conjectures and refutations.
Tangentially, my understanding is that the process is pretty bad, and can be really egregiously awful for software algorithms and other tech stuff that the patent workers don’t understand.
It gives us something to put on a list of criteria to look at specifically (similarity to previous knowledge exceeding a breakpoint) vs. just trying to figure out if it’s new knowledge with no explicit criteria. Functionality is another I’d put on the list (does it solve the problem it was created to solve). There are probably others. And I still agree it’s hard.
I think we still disagree about whether it’s best to do C&R on the output vs. C&R on the methods used to generate the output.
i wonder if this connects to the earlier judgment issue in a deeper way.
in human thinking, a lot of important filtering seems to happen before things are fully explicit. intuitions, half-formed objections, quick relevance estimates, etc.
do you think an AGI would need some analogue of that pre explicit layer for judgment, or do you think explicit criticism and explicit representations are enough in principle?