Beautiful illustrations I find, 'Playing' is just the free and motivated version of 'exploration'.

One thought on your nicely illustrated "key observation [is] that neural networks tend to place features along directions": my guess is that the neural net was TOLD to behave that way by choosing e.g. Cosine Loss?

Very nice visualizations, thanks for that!

One thing I still struggle with in my head is how these vision embeddings can then be used to give LLMs eyes.

Because you somehow need a giant training set which describes images in natural language, no? Is that actually how it works, or is there some smart trick so you don't need to pay labellers a bunch of money to look at pictures and describe them.

[dead]