He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
[0] https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ [2] https://spacy.io/
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
I have no idea what "Aristotelian essentialism" is supposed to mean, or if you are saying that the unification was that.
> And apparently the Japanese complained bitterly over the ones that were merged.
and
> but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode [...] mostly due to vocal objections from the Japanese.
Seems to contradict each other.
[1] https://en.wikipedia.org/wiki/Ideographic_Research_Group
Page 55:
""" there are characters with no glyphs. glyphs that can correspond to a number of different characters according to context. Glyphs that correspond to multiple characters at the same time (with weightings assigned to each), and even more possibilities.
The problem of glyphs and characters is so complex that it has gone beyond the realm of computer specialists and has come to be of interest even to philosophers. For example, the Japanese philosopher Shigeki Moro, who has worked with ideographic characters in Buddhist documents, goes so far in his article Surface or Essence: Beyond Character Model Set [274] as to say that Unicode's approach is Aristotelian essentialist and to recommend supplanting it by an approach inspired by Jacques Derrida's theory of writing [114, 115]. The reader interested in the philosophical aspects of the issue is invited to consult [165,156], in addition to the works cited above. """
I think "essentialist" is probably a good description of the philosophy of how Unicode defines characters as opposed to fonts and glyphs, so I adopted it.
There has to be a reason why it happened(as to why Unicode suddenly started insisting it has to fit inside a 2^16 total chars or whatever).
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
Why didn't they simly replace the original bad one?
> nine hundred pages. Imagine tracking down a single character without a page reference
Not that hard to imagine, OCR existed back then?
<Multi_key> <c> <c> <c> <p> : "\xe2\x98\xad" #symbol representing proletarian solidarity between agricultural and industrial workers
As a slightly related tangent, The compose key mnemonic interface for rarely used characters is pretty great, beats trying to remember alt codes.