Your point about intellectual tolerance and humility brings up an interesting AI design question. Would an AGI have an in-built advantage here because it doesn’t naturally have human social and emotional baggage? Or is it that any system capable of judgment must inherently start with some framework of initial heuristics and assumptions?
I’m thinking about how Deutsch interprets Gödel that it really just means you can’t mechanically derive all truth from a fixed, closed set of axioms. You need creativity to guess new explanations and step outside the initial system.
Translating that to agents: does that mean it’s formally impossible to build a ‘complete’, perfectly neutral judging machine that just mechanically outputs good judgments? And therefore an agent must be intrinsically fallible and reliant on its initial guesses, which means absolutely all the pressure to not stay wrong falls on having a robust error-correction environment?