Just had a thought re LLMs and science:
Karpathy has this optimization loop for doing automated ML research: GitHub - karpathy/autoresearch: AI agents running research on single-GPU nanochat training automatically · GitHub
I was thinking about this and research more generally, and it occurred to me that you cannot optimize your way to paradigm shifts reliably (maybe not at all, but sometimes I think there’s enough viable option-space that you can luck into it easy enough).
That’s not a new idea (the underlying idea about learning and knowledge-growth is from CF).
But in an AI context it means there are big limitations to what automated research can achieve. Practically, it is much more like a search algorithm through existing ideas that are in its training data or that it finds via search.
Possible solutions (geared towards evolution): maybe turning the temperature way up to get way higher variance on token output, or training specialized models to produce highly variable but coherent output. One idea I’ve had that I haven’t been able to look into is deliberately injecting noise into context (have been meaning to test that since december) but now I’d probably jump straight to getting residuals / latent vectors and mixing it all together.