Skip to content

util: Treat zero-width joiner and non-joiner as combining characters - #4274

Open
ShryukGrandhi wants to merge 1 commit into
micro-editor:masterfrom
ShryukGrandhi:fix/issue-2155-replace-some-invisible-characters-with-s
Open

ShryukGrandhi wants to merge 1 commit into
micro-editor:masterfrom
ShryukGrandhi:fix/issue-2155-replace-some-invisible-characters-with-s

Conversation

@ShryukGrandhi

Copy link
Copy Markdown

Summary

Zero-width joiner (U+200D) and zero-width non-joiner (U+200C) are used a lot in Persian and other scripts. micro decoded each of them as a separate character, and since they have a display width of 0, tcell rendered every one of them as its own blank cell. A word like sl<ZWNJ x8>am was shown as sl am. This also didn't match micro's own width calculations (StringWidth, SliceVisualEnd, softwrap), which already counted these runes as width 0, so the cursor could be placed in the wrong column on lines that contain them.

Changes

  • isMark in internal/util/unicode.go now also matches unicode.Join_Control (U+200C, U+200D). This means DecodeCharacter/DecodeCharacterInString attach them to the preceding character the same way they already attach combining marks, and CharacterCount stops counting them. They get sent to the terminal together with the base character and no longer take up a cell of their own.
  • Bidi controls such as LRM/RLM (U+200E/U+200F) and the embedding/override characters are deliberately unchanged. Sending them to the terminal can reorder how the displayed text looks, so I left them out of this change.

Testing

  • Added TestDecodeCharacterJoinControl in internal/util/util_test.go. It fails without the fix (ZWNJ/ZWJ were decoded as separate characters) and passes with it.
  • go test ./... passes, go vet ./internal/util/ and gofmt -l internal/ are clean, and go build ./cmd/micro succeeds.

Fixes #2155

Signed-off-by: ShryukGrandhi <85529990+ShryukGrandhi@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace some invisible characters with space

1 participant