So we have two identically formatted piece of data, where only the text colour offers a clue on what's real and what isn't. Absolutely insane.
Did they try one method, and what was the feedback?
It's the ground truth, and it's not like it's more complex than parsing a pdf, at this point, the technology for OCR might even be better than PDF parsing, which is full of accidental instead of natural complexity.
Another thing to consider is OCR works well for English but not so well for other languages.
OCR of a digital image generated from those PDF draw commands only operates on the visible portions that are rendered.
Another case to be considered .. FOI requests for documents from "pain in the arse" government departments or law firms can result in "digital documents" that are deliberately generated to be difficult to process (go figure) .. eg: filled with circular and spaghetti draw commands that "print correctly" but are a nightmare horror show to parse.
( "technically compliant" is a legally compliant "fuck you" )
I’d be cautious about making “non-black means invisible” the general rule. A possible middle ground between raw text extraction and full OCR would be to render the page, map each extracted glyph’s bounding box back to the rendered pixels, and discard glyphs that have almost no contrast with their local background.
That would preserve the exact embedded text for visible characters while using rasterization only to determine visibility.
Has anyone tried this kind of hybrid approach?
I should have made it clear in the blog post, but the first thing I do when processing a PDF is I identify the document_type. Then I run special rules for that document_type.
So the "non-black means invisible" rule only applies when processing this document_type.
Instead I get a nice blog post about a knotty little problem caused by someone's abuse of semantic markup for style purposes. Shows that HTML doesn't have a monopoly on that issue. PDFs are already bad enough as a format for extracting data from, we don't need dumb HTML-inspired tricks on top of that.
Nice job finding the issue and explaining it clearly, OP. Thanks for an interesting writeup.