But maybe worth reframing away from “discovering” - hierarchical planning, sub-goals, action chunking etc. are well understood - this is more of a… transcript of your personal learning journey?
Which is still cool to see. But I read through expecting there to be some kind of novel results. We probably need to categorize these kinds of posts a bit better.
Basic question: couldn't something like probabilistic roadmaps [1] trivially solve this? Or am I missing something about the limitations of PRM's?
And if so, is that the best test for JEPA?
However, based on this article, i have the feeling that on the contrary, JEPA is filling a super niche problem, and is pretty much only trying to improve robotic motions in space.
Which of the two ideas is it ? Can JEPA technics be applied to a much more wide range of problems ? (like LLM and transformers)
OF course your decoder model could end up doing a lot of the heavy lifting if your decoder loss was based on text reconstruction methods (which almost all LLMs use for eval purposes)
For certain tasks, you do need pixel/token level reconstruction fidelity to do a lot of things. Like image generation or text generation.
For other tasks, having a coherent representation of the world that can be computed quickly and intuitively is more salient. I suspect the human mind works more like JEPA than like an autoregressive LLM. Human children train on far less data to achieve extreme fidelity on a lot of tasks generative reconstruction loss based AI/ML techniques struggle with.
Awesome read!
If you’re going to use AI to generate text, keep it short and concise and meaty. I don’t hate reading AI, but I can’t read large walls of text for presumably some useful nugget of information in it.
https://www.pangram.com/history/2dd2b019-221a-4bee-bb6a-6b7e...
Anyway:
> The physics of pushing a block doesn't change if you shift, rotate or mirror the whole table. So when the graph has no states near a goal, it can take demonstration segments from elsewhere, transform them to fit the gap, and plan through those. With these transformed segments, success came back to 85%
This is an interesting technique to recover incomplete states or lost memories after demonstration. I'm guessing that the original LeWM does not have technique of transformative memories if I am correct?
There is a faint somewhat distracting background animation though I think…
> (3) That's where I decided to build a harder test, where each task takes its start from one demonstration and its goal from another
> (4) I also built an interactive demo where I could place the block anywhere and pick any goal. [but] The goals in the harder test are frames from expert demonstrations
Why lean so hard on the demonstration examples? Surely the actual task here is to move between any two arbitrary poses.
So if the demonstration set is statistically strange in any way, you keep banging into that. And even if you get great results, you still haven’t shown that it generalizes to the generic system. And the one thing you know 100% about a physical robot system is that you will never encounter exactly the same situation twice.
Is there actually a reason to pick eval examples from the demonstrations instead of independent samples from the underlying task?
I’m pretty sure that the T-shoving problem doesn’t have goal reachability issues. And if it did, test 4 could run into them since the start pose is arbitrary.
That seems like a really massive failure of the model to generalize? This doesn't seem like a new technique, it seems like you've discovered a pretty big issue with the fundamental setup