python (65.2k questions)

javascript (44.3k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (13k questions)

Questions - unicode-normalization

How do I process multi-character Unicode emojis in Python 3 with the unicodedata module?

While I was working with emojis and attempting to acquire their codepoint and names with the unicodedata module, I kept having issues with multi-character emojis. The module refuses to let me use stri...
test-img

Xevion

python

unicode

emoji

unicode-normalization

Votes: 0

Answers: 1

Latest Answer

Some human-perceived single-character emoji (called graphemes) are made up of multiple code points. Here's a way to handle them. I added a complicated example: import unicodedata as ud emojis = [&q...
test-img

Mark Tolonen

Where can I get examples of unicode that normalizes differently?

I'm adding yet another unicode normalization question because I've spent quite a bit of time looking and can't find what I need. I have a situation where I need to normalize unicode to check if string...
test-img

shortorian

unicode

unicode-normalization

Votes: 0

Answers: 1

Latest Answer

https://unicode.org/reports/tr15/#Norm_Forms has a good number of examples, and a significant amount of explanations around them.
test-img

tripleee

Convert string to canonical form with R

Here's an example of 2 strings that are not technically the same. I know for a fact that these 2 strings come from the same original value and have just been processed differently. str1 <- "Ta...
test-img

asachet

r

unicode

character-encoding

unicode-normalization

Votes: 0

Answers: 1

Latest Answer

I was indeed just missing a keyword: "unicode normalisation forms". This is easily done with the stri_trans_* functions from stringi, typically stringi::stri_trans_nfc. str1 <- "Tapa...
test-img

asachet

Why does utf-16 only support 2^20 code points?

Well, I'm starting to study unicode now, and I had several doubts, at this moment I'm learning what a plane is, I saw that a plane is a set of 2^16 code points, and that utf-16 encoding supports 17 pl...
test-img

Yuri Bittencourt

utf-8

utf-16

utf

unicode-normalization

utf-16le

Votes: 0

Answers: 3

Latest Answer

Have a look at how surrogate pairs encode a character U >= 0x10000: U' = yyyyyyyyyyxxxxxxxxxx // U - 0x10000 W1 = 110110yyyyyyyyyy // 0xD800 + yyyyyyyyyy W2 = 110111xxxxxxxxxx // 0xDC00 ...
test-img

Luatic

Posts

Questions

Blogs

Jobs