AGI, LLMs, CR and CF

I doubt genetic evolution is very efficient. But it’s hard to know which optimizations have what downsides.

Aggressive pruning every generation, and a small gene pool, seems more likely to get stuck at local optima, which is already a problem for animals.

Having 1, 2 or 3+ sexes is important too.

yeah, i think the lack of a direct optimizer may matter. the question isn’t just whether evolution could be more efficient, but efficient at what.

this reminded me of a point from deutsch in deutsch files iv: he says ribosomes are the educational institutions of cells, and that their job is to pass on cellular knowledge as faithfully as evolution can make it. i might be misinterpreting, but that connection seemed relevant.

i take the lesson as: evolutionary processes don’t just favor more experimentation. some parts of the process can be selected to reduce variation and preserve a pattern accurately. that can be good if the pattern is already valuable, but it’s different from open-ended creativity.

so maybe there are several separate issues:

  1. how variants are generated
  2. how aggressively variants are pruned
  3. what gets preserved with high fidelity
  4. whether the process can make explanatory jumps instead of only local improvements

that makes me less inclined to treat more direct fitness measurement as automatically better. it might make some engineering searches more efficient relative to the metric, but it could also make the process narrower depending on what the measure rewards.

checkout Primer youtube channel. he makes animated simulations/explainers about evolution, game theory, emergence, cooperation, etc. a lot of them are toy models, but still interesting.

liked this one on evolution of teamwork.

I think just knowledge creation. It’s not a very satisfying answer (doesn’t give us a fitness heuristic or anything like that).

PS. am familiar with primer. I like his stuff mostly. I think I’ve had an issue with one or two things in the past but that’s like 1-3 years ago so don’t recall specifics.

1 Like

knowledge creation does seem like the right target.

the tricky part is that a fitness heuristic for knowledge creation might already need a lot of the thing we’re trying to explain. it has to tell real explanatory progress apart from local proxy wins.

so maybe the lack of a direct optimizer matters. direct scoring can make search faster against a metric, but if the metric is narrow, it can also prune away weird/open-ended paths too early.

that connects to the AGI/creativity issue for me: variation+selection by itself doesn’t explain much unless we say what is doing the selecting, pruning, and relevance-judging without turning it into a narrow proxy.

I’m curious, what do you think knowledge is?

Re ‘direct scoring’ and ‘metric’ – have you read Multi-Factor Decision Making Math ?

I basically agree with David’s definition of knowledge as information that keeps itself in existence, but I don’t think that helps us much with the thing we are interested in here which is explanatory knowledge and the processes that create it.

I’ve read it. As I understand it, the core point is that you can’t solve problems just by using weighted scoring. I agree with that. But I don’t think MCDM gives the missing fitness heuristic for knowledge creation. IGC seems like a useful reframing. It is better than evaluating ideas in isolation, but the hard part still seems to be judgment: choosing the goal and context, choosing the breakpoint, and deciding whether a criticism really refutes the idea in that context. So as I see it, the judgment problem is still there, just one level down.

Interesting reaction to AGI discourse. BoI is mentioned throughout.

Hmm, is GRPO evolutionary? The main idea is:

  1. provide past data
  2. get LLM to predict multiple answers
  3. select answer closest to measured data
  4. train towards selected answer, away from other answers

Not sure how it handles similar answers with different reasoning paths; maybe that’s why the reasoning degrades if you iterate too many times.

I am not convinced this actually is exceeding a human limit, because it isn’t going to create new explanations in the reasoning, but it might get better at which reasoning paths it uses.

I asked ChatGPT 5.5 High to study Elliot’s discussion trees article

and make a discussion tree about replacing incandescent lightbulbs with LED. This is what it came up with and I think it’s impressive. Am I too easily awed by this?

Discussion tree

Legend

  • [D] Decisive criticism: if correct, the parent claim fails.
  • [R] Reply to a criticism.
  • [E] Explanation or supporting information.
  • [✓] Tentatively resolved.
  • [?] Unresolved leaf.
  1. Replace every working household incandescent bulb with an LED immediately.

├── 1. [D] “Every bulb” is too broad.

│ ├── 1.1 Some bulbs have unusual jobs.
│ │
│ ├── 1.2 A particular LED may not work properly with the fixture or controls.
│ │
│ └── 1.3 [R] Restrict the proposal to ordinary household lighting.
│ The replacement LED must be suitable for the specific fixture.
│ [✓]

├── 2. [D] The replacement is wrong if it does not perform the required job.

│ ├── 2.1 The bulb must provide enough brightness.
│ │
│ ├── 2.2 The bulb must provide acceptable color.
│ │
│ ├── 2.3 The bulb must provide an acceptable beam shape.
│ │
│ ├── 2.4 The bulb must dim properly when dimming is required.
│ │
│ ├── 2.5 [R] Compare lumens, color temperature, shape, and compatibility.
│ │ Do not assume every LED is dimmable merely because some are.
│ │
│ └── 2.6 [?] Is there an LED that performs acceptably in this exact fixture?

├── 3. [D] Immediate replacement is wrong if it costs more over the
│ relevant period of ownership.

│ ├── 3.1 [E] Buying the LED imposes an immediate cost.
│ │
│ ├── 3.2 [E] Electricity savings occur gradually.
│ │
│ ├── 3.3 [E] Replacement savings also occur gradually.
│ │
│ ├── 3.4 [R] LEDs can use up to 90% less electricity and last up to
│ │ 25 times longer than traditional incandescent bulbs.
│ │ Frequently used bulbs therefore have a strong chance of repaying
│ │ their purchase cost. [Not by itself decisive.]
│ │
│ ├── 3.5 [D] A rarely used bulb may not repay the purchase cost before
│ │ the owner moves, remodels, or replaces the fixture.
│ │
│ └── 3.6 [?] Does this bulb repay its cost before the expected end
│ of its use?

├── 4. [D] Discarding a functioning incandescent bulb may create needless
│ manufacturing and disposal waste.

│ ├── 4.1 [E] Producing the replacement has environmental costs.
│ │
│ ├── 4.2 [E] Disposing of the old bulb also has environmental costs.
│ │
│ ├── 4.3 [R] A Department of Energy life-cycle analysis found that
│ │ electricity used during operation dominated the environmental
│ │ effects studied. The incandescent lamp had substantially greater
│ │ impact per equal amount of light than the LED.
│ │
│ ├── 4.4 [D] That general result may not settle the case of a bulb
│ │ that is almost never switched on.
│ │
│ └── 4.5 [?] How much future use makes immediate replacement
│ environmentally preferable to waiting for failure?

├── 5. [D] Incandescent heat may be useful during cold weather.

│ ├── 5.1 [E] Incandescent bulbs release roughly 90% of their energy
│ │ as heat.
│ │
│ ├── 5.2 [R] That heat is unwanted when the building is being cooled.
│ │
│ ├── 5.3 [R] A lighting fixture is usually a poorly controlled way
│ │ to heat a room.
│ │
│ └── 5.4 [?] In a particular cold building, does useful heat materially
│ alter the economic calculation?

├── 6. [D] The LED may fail much earlier than its advertised lifespan.

│ ├── 6.1 [E] The driver or other electronics can fail before the
│ │ light-emitting components wear out.
│ │
│ ├── 6.2 [R] Correct installation and product quality matter.
│ │ The Department of Energy says good-quality white LEDs are
│ │ expected to provide roughly 30,000–50,000 useful hours, compared
│ │ with around 1,000 hours for a typical incandescent.
│ │
│ └── 6.3 [?] Is the proposed LED reliable in this particular fixture?

└── 7. [R] Revised proposal:
Replace a working incandescent immediately when:

├── 7.1 The LED performs the required lighting job.
├── 7.2 The LED is compatible with the fixture and controls.
├── 7.3 Its purchase cost is recovered within the expected use period.
└── 7.4 No special requirement gives the incandescent an advantage.

Otherwise, continue using the incandescent until it fails or until
a satisfactory replacement becomes available.
[✓ Tentatively accepted]