the platonic representation hypothesis and what it says about the world

#thought
the platonic representation hypothesis and success of deep learning and transformers as a reflection of the structure of the world

• hypothesis: there are platonic representations because of evolution; therefore, say, the layer number in a vision neural network should correspond monotonically with when said layer was "added by evolution"

• e.g. for language models, this text is generated by humans, who are presumably representing this language as concepts in some manner

  • this representation has been built up in a sedimentary manner due to evolution
  • and it is one that is shared by all the tasks that we perform (e.g. image recognition)

• ideas

  • does how many foundation model layers are required for a particular competency correlate with evolution