What you give up with the authors approach is "read". The screen becomes write-only. There are only a few functions that require reads: getting a pixels current color and doing operations like flood fills or paints.
Because I was going for the retro feel of BASIC, I wanted to support these, so I opted for a frame buffer. It isn't as technically cool. On the RP2350, it does require low resolution and low bit rates because they often have RAM limited to KB.
If you step-up to the ESP32-S3 variants with 4MB or 8MB of RAM the frame buffer gets much easier.
[0]: https://mitpress.mit.edu/9780262539760/racing-the-beam/
My code can JIT + render to the atlas faster at p100 than CoreText can render without the JIT at p50, and with better quality (measured against an exact cover reference).
Image 6 (with figures 8 through 12) is fun; you can eliminate the Shift if you're willing to have multiple different JITed functions for each character, each one generated knowing what its shift count is, leading to yet more optimization opportunities.