Unicode write policy#

gate.policies.unicode is a disabled, standard-library-only policy extension for complete text fields from model-originated write and edit attempts. Importing or constructing it does not register a hook. A native adapter must still prove the pre-call candidate, target identity, denial support, and any replacement capability through the admission contract.

Enroll both policy classes in one kernel:

from gate.admission import AdmissionKernel
from gate.policies import UnicodeSafetyPolicy, UnicodeTechnicalPolicy

kernel = AdmissionKernel([UnicodeSafetyPolicy(), UnicodeTechnicalPolicy()])

The split preserves two kinds of authority:

  • UnicodeSafetyPolicy denies dangerous controls, bidi state, zero-width and tag hazards, shorthand and musical format controls, invisible Hangul fillers (including U+115F, U+1160, U+3164, and U+FFA0), surrogates, and Unicode noncharacters. These denials are not overridable.

  • UnicodeTechnicalPolicy rejects CJK-family scripts and non-ASCII technical punctuation. Its denials are overridable through the ordinary bound admission mechanism.

Conservative repair#

For .md, .mdx, .rst, and .txt targets, the technical policy offers one complete-field successor when every finding has a context-independent mapping:

  • non-ASCII members of Unicode 17’s Dash property outside the stricter CJK-family profile and with context-independent mappings become ASCII -;

  • Unicode 17 Quotation_Mark members outside the CJK-family profile and with context-independent mappings become ASCII ' and ";

  • U+2026 becomes three ASCII periods.

The kernel evaluates that complete successor once more before reporting a repair. The repairer is a fixed point: applying it to an accepted successor changes nothing.

U+207B SUPERSCRIPT MINUS and U+208B SUBSCRIPT MINUS remain members of the detected Unicode 17 Dash set, but Gate does not map either one to a baseline -. That mapping would discard their placement semantics, so prose containing either scalar receives a unicode.punctuation denial with explicit-replacement or bound-override recovery.

CJK-family text is never transliterated. Code-like and unknown targets are never rewritten: Gate returns coordinates and asks the caller to fix the source or present a bound override. The policy does not apply NFC or NFKC normalization, infer language, read the target, execute a tool, retry a tool, call a model, or use a network.

Coordinates and retained Unicode#

scan_unicode() reports the code point plus zero-based scalar and UTF-8 byte offsets and one-based line and column numbers. Diagnostics contain ASCII coordinates, not the source scalar. The scanner uses static Unicode 17.0 tables so Python’s Unicode database version cannot silently change the enrolled rule set. The punctuation oracle is the Unicode 17 PropList.txt; the CJK-family profile includes Kanbun, Khitan Small Script, Tangut supplements, and CJK Unified Ideographs Extension J from Blocks.txt.

The technical profile is intentionally not ASCII-only. Accented Latin, Greek, Cyrillic, mathematical symbols, and ordinary emoji remain unchanged unless they contain an explicit safety hazard or selected punctuation. Variation selectors and Unicode tag characters are invisible state and therefore receive a safety denial.

Python strings containing lone surrogates are serialized into admission digests with escaped surrogate code units. That keeps ordinary valid-Unicode digest bytes unchanged while allowing the non-overridable safety policy to return a content-free denial receipt instead of raising before evaluation.

Deterministic Unicode safety and technical-write admission policies.

The module accepts one complete candidate string. It never normalizes, transliterates, reads a target file, invokes a model, or talks to a network. Safe punctuation repair returns a complete successor candidate; every other finding returns bounded ASCII diagnostics.

class gate.policies.unicode.UnicodeIssue(kind: UnicodeIssueKind, scalar_index: int, utf8_offset: int, line: int, column: int, codepoint: str)#

One unwanted scalar with content-free coordinates.

diagnostic() → str#

Render a bounded ASCII diagnostic without the source character.

class gate.policies.unicode.UnicodeIssueKind(*values)#

Stable classes emitted by the Unicode scanner.

class gate.policies.unicode.UnicodeSafetyPolicy#

Non-overridable policy for controls, bidi state, and invisible hazards.

evaluate(attempt: Attempt) → Proposal#

Deny dangerous scalars and allow a complete clean text candidate.

class gate.policies.unicode.UnicodeTechnicalPolicy#

Overridable technical-style policy for punctuation and CJK-family scripts.

evaluate(attempt: Attempt) → Proposal#

Repair prose punctuation or deny technical text that needs judgment.

gate.policies.unicode.repair_unicode(value: str) → str | None#

Return a fixed-point ASCII-punctuation successor, or None if unsafe.

gate.policies.unicode.scan_unicode(value: str) → tuple[UnicodeIssue, ...]#

Return stable unwanted-scalar findings without normalizing value.

Activation frontier#

This package still has no native admission adapter. A later card must measure each harness’s ability to provide a complete pre-write field and to enact denial or full-field replacement. Until then, the policy is an explicit Python API and test fixture, not fleet enforcement.