Leith.Text

Standards-based Unicode text semantics for .NET: grapheme segmentation and navigation, Unicode Character Database lookups, streaming state, and a deliberately separate terminal-cell-width policy.

Leith.Text is designed as a general-purpose library, not as a private implementation detail of the Leith applications. Leith.Editor, Leith.Term, and Leith.Commander are consumers of its public contracts; they do not define those contracts. A proposed API should make sense to a .NET user who has never seen another Leith repository.

Grapheme segmentation is universal Unicode text semantics. Terminal column width is a console policy — separate from GUI/font shaping width and kept isolated under Leith.Text.Terminal.

Why

string.Length and naive enumeration are not enough for anything user-facing: a single character a user can select or delete is often several UTF-16 code units, or several code points joined by a zero-width joiner (family emoji, flags, skin-tone modifiers). If you are building a text editor, terminal emulator, TUI, chat application, log viewer, protocol decoder, or any other Unicode-aware .NET component, this library provides reusable text-level semantics:

• Validated, not just implemented. Grapheme segmentation is tested against the official Unicode UAX #29 conformance corpus (GraphemeBreakTest.txt), not hand-picked examples.
• Zero dependencies beyond System.Memory on netstandard2.0. Nothing Leith-specific leaks into the public API.
• Multi-targeted: netstandard2.0, net8.0, net9.0, net10.0 — usable from .NET Framework 4.6.1+ through current .NET.
• Standards-pinned data. Unicode properties are generated from a pinned UCD version and the generated tables are verified in CI.
• Universal semantics and consumer policy stay distinct. Grapheme segmentation (UAX #29) is universal; terminal column width (TerminalCellWidth) is a separate, opinionated console policy.

Scope

The project owns reusable Unicode text semantics: operations and properties that can be specified, tested, and consumed independently of a particular editor, terminal, renderer, or application model.

A feature is a good fit when it:

• represents a reusable property, boundary, transformation, or state machine over Unicode text;
• has a standards-based or otherwise explicit semantic contract;
• can be tested independently of rendering, document storage, terminal grids, or application state;
• is useful to consumers outside the Leith workspace; and
• does not introduce Leith-specific concepts into the public API.

The project intentionally does not own font shaping or rendering, GUI layout, terminal screen-grid placement, editor document models, syntax highlighting, language parsing, or application-specific text policy.

See Docs/scope-and-roadmap.md for the project boundary and roadmap, and Docs/research/text-semantics.md for the research gate applied before expanding into additional Unicode semantics.

Layout

Leith.Text
├── Unicode/          UCD 17.0.0 properties and tables
├── Graphemes/        UTF-16 grapheme boundaries and navigation
└── Terminal/         terminal cell width measurement and truncation

One NuGet package (Leith.Text), three namespaces.

Used internally by

Extraction from Leith.Term is complete; Leith.Editor and Leith.Term consume this package as the UAX #29 source of truth, the same way any other .NET consumer would.

API sketch

using Leith.Text.Graphemes;
using Leith.Text.Terminal;
using Leith.Text.Unicode;

// Universal grapheme operations (UTF-16 offsets)
foreach (var cluster in Grapheme.Enumerate(line))
{
    ...
}

var count = Grapheme.Count("👨‍👩‍👧‍👦abc"); // 4

var previous = Grapheme.PreviousBoundary(line, caret);
var containing = Grapheme.GetContainingRange(line, caret);

// Unicode data
var widthClass = UnicodeProperties.GetEastAsianWidth(0x3042);

// Terminal policy (not GUI width)
var columns = TerminalCellWidth.Measure("🇵🇱");
var cjkColumns = TerminalCellWidth.Measure("·", TerminalWidthPolicy.WideAmbiguous);
var clipped = TerminalCellWidth.Truncate(filename, 30);

Document offsets stay UTF-16; GraphemeNavigator snaps and steps across legal grapheme boundaries.

Build

dotnet build Leith.Text.slnx -c Release
dotnet test --solution Leith.Text.slnx -c Release

Unicode table regeneration

Pinned UCD inputs: unicode/17.0.0/. Generator: tools/Leith.Text.UnicodeGenerator/.

# First-time or UCD bump
pwsh scripts/fetch-ucd.ps1 -Version 17.0.0

# Regenerate committed *.g.cs
pwsh scripts/regenerate-unicode-tables.ps1

CI reruns the generator and fails when src/Text/Internal/Generated/*.g.cs drift from the pinned UCD (git diff --exit-code).

Tests use xunit.v3 4.x with Microsoft.Testing.Platform (global.json). The test suite itself targets net10.0; the library targets netstandard2.0;net8.0;net9.0;net10.0.

Status

UCD 17.0.0 tables, allocation-free UTF-16 enumeration, GraphemeStreamState for incremental join state, GraphemeNavigator editor helpers, and terminal width under TerminalCellWidth. Consumed by Leith.Term (segmentation + width) and Leith.Editor (caret navigation). Multi-targeted netstandard2.0/net8.0/net9.0/net10.0; version 0.5.0.

The next phase is contract stabilization and research, not feature accumulation: establish the long-term Unicode/versioning policy, harden the existing API, and evaluate additional text semantics against the project scope before committing them to the roadmap.