I think most people (quite reasonably) think 'We built ChatGPT, so we must basically understand how it works'
This is not true at all. Humans did not build ChatGPT. In a way it would be closer to say we 'grew' it. We have basically no idea how it does what it does.

We're like a plane company CEO; we don't know how to build jets, we know how to hire engineers. Being good at paying engineers doesn't cause you to understand how jets actually work, at all. Especially since these 'engineers' never speak a word or document anything

Their results are bizarre and inhuman. Neel Nanda trained a tiny transformer to do addition, then spent weeks figuring out what it was doing - one of the only times in history someone has understood how a transformer works.
This is the algorithm it created. To *add two numbers*!