> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myspellchecker.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Segmenters

> Two-level text segmentation for Myanmar: rule-based syllable splitting and dictionary-backed word tokenization with myword, CRF, or transformer engines.

Myanmar text segmentation operates at two levels, **syllable** and **word**, because Myanmar script has no spaces between words. mySpellChecker separates these concerns: a fast regex-based segmenter handles syllables, while a configurable word engine (myword, CRF, or transformer) handles word boundaries.

## Architecture

<img src="https://mintcdn.com/myspellchecker/J6ruVvwgW13OkpGM/images/segmentation-levels.png?fit=max&auto=format&n=J6ruVvwgW13OkpGM&q=85&s=cfb861bd74b92a118b83cf1828f1d706" alt="Segmentation architecture: Raw Myanmar Text forks into Syllable Segmentation (RegexSegmenter, rule-based) and Word Segmentation (DefaultSegmenter with myword/CRF/transformer engine)" width="1923" height="840" data-path="images/segmentation-levels.png" />

| Level        | Default Engine   | How It Works                           | Downloads                            |
| ------------ | ---------------- | -------------------------------------- | ------------------------------------ |
| **Syllable** | `RegexSegmenter` | Regex-based Sylbreak algorithm         | None                                 |
| **Word**     | `myword`         | Viterbi with unigram/bigram dictionary | `segmentation.mmap` from HuggingFace |

***

## Quick Start

```python theme={null}
from myspellchecker.segmenters import DefaultSegmenter

segmenter = DefaultSegmenter()

# Syllable segmentation (RegexSegmenter internally, no download)
syllables = segmenter.segment_syllables("မြန်မာနိုင်ငံ")
# ['မြန်', 'မာ', 'နိုင်', 'ငံ']

# Word segmentation (myword engine, downloads dictionary on first call)
words = segmenter.segment_words("မြန်မာနိုင်ငံသည်")
# ['မြန်မာ', 'နိုင်ငံ', 'သည်']

# Sentence segmentation
sentences = segmenter.segment_sentences("ပထမစာ။ ဒုတိယစာ။")
# ['ပထမစာ။', 'ဒုတိယစာ။']
```

***

## Syllable Segmentation

### RegexSegmenter

All syllable segmentation in mySpellChecker uses `RegexSegmenter`, a pure-Python, rule-based segmenter with zero external dependencies and no network downloads.

```python theme={null}
from myspellchecker.segmenters import RegexSegmenter

segmenter = RegexSegmenter(
    allow_extended_myanmar=False,  # Include Extended Myanmar Unicode blocks (default: False)
)

syllables = segmenter.segment_syllables("မြန်မာစကား")
# ['မြန်', 'မာ', 'စ', 'ကား']
```

**Characteristics:**

* Pure Python with optional Cython acceleration
* No downloads, no model, no dictionary needed
* Fork-safe for multiprocessing
* Handles stacked consonants (Virama ္), Kinzi sequences, and non-Myanmar text

<Note>
  `RegexSegmenter` only supports syllable and sentence segmentation. It raises `NotImplementedError` for `segment_words()`. Use `DefaultSegmenter` for word segmentation.
</Note>

### Sylbreak Algorithm

The segmenter uses an adapted [Sylbreak](https://github.com/ye-kyaw-thu/sylbreak) algorithm:

```python theme={null}
# Pattern components:
# 1. Myanmar consonant not preceded by stacking Virama
p_my_cons = r"(?<!(?<!\u103a)\u1039)[\u1000-\u1021](?!\u103a)"

# 2. Independent vowels, digits, symbols
p_other_starters = r"[\u1022-\u102a\u103f\u104c-\u104f\u1040-\u1049\u104a\u104b]"

# 3. Non-Myanmar characters (grouped)
# Default (allow_extended_myanmar=False): [^\u1000-\u104F]+
# Extended (allow_extended_myanmar=True): [^\u1000-\u109F]+
p_non_myanmar = r"[^\u1000-\u104F]+"
```

### Cython Acceleration

RegexSegmenter automatically uses a Cython-compiled implementation when available:

```python theme={null}
from myspellchecker.segmenters.regex import _HAS_CYTHON_SEGMENTER

if _HAS_CYTHON_SEGMENTER:
    print("Using fast Cython implementation")
else:
    print("Using pure Python implementation")
```

***

## Word Segmentation

Word segmentation is handled by `DefaultSegmenter`, which delegates to one of three word engines. Both `myword` and `crf` download resources from HuggingFace on first use.

### Word Engines

| Engine             | Accuracy | Speed                   | Model Source                                    | Dependencies                |
| ------------------ | -------- | ----------------------- | ----------------------------------------------- | --------------------------- |
| `myword` (default) | \~90%    | Fast                    | `segmentation.mmap` from HuggingFace            | None (pure Python + Cython) |
| `crf`              | \~92%    | Medium                  | `wordseg_c2_crf.crfsuite` from HuggingFace      | `pycrfsuite`                |
| `transformer`      | \~96%    | Slow (CPU) / Fast (GPU) | `chuuhtetnaing/myanmar-text-segmentation-model` | `transformers`, `torch`     |

### Engine Selection

```python theme={null}
from myspellchecker.segmenters import DefaultSegmenter

# myword (default) — Viterbi with unigram/bigram dictionary
segmenter = DefaultSegmenter(word_engine="myword")

# CRF — Conditional Random Field sequence tagger
segmenter = DefaultSegmenter(word_engine="crf")

# Transformer — XLM-RoBERTa fine-tuned for word boundaries
segmenter = DefaultSegmenter(
    word_engine="transformer",
    seg_model="chuuhtetnaing/myanmar-text-segmentation-model",  # default
    seg_device=-1,   # -1=CPU, 0+=GPU
)
```

### HuggingFace Resource Downloads

The `myword` and `crf` engines download their resources from the [`thettwe/myspellchecker-resources`](https://huggingface.co/datasets/thettwe/myspellchecker-resources) HuggingFace dataset repository on first use:

| Engine   | Resource                     | File                             | Size           |
| -------- | ---------------------------- | -------------------------------- | -------------- |
| `myword` | Word segmentation dictionary | `segmentation/segmentation.mmap` | Memory-mapped  |
| `crf`    | CRF model                    | `models/wordseg_c2_crf.crfsuite` | CRF model file |

Resources are cached at `~/.cache/myspellchecker/resources/` and only downloaded once.

```bash theme={null}
# Override cache directory
export MYSPELL_CACHE_DIR="/path/to/cache"

# Prevent network downloads (fail if resource not cached)
export MYSPELL_OFFLINE=true
```

<Note>
  Word segmenters use **lazy initialization**, so no download occurs when you create a `DefaultSegmenter` or `SpellChecker`. The download happens on the first call to `segment_words()`.
</Note>

### myword Engine

The default word segmentation engine, based on [myWord by Ye Kyaw Thu](https://github.com/ye-kyaw-thu/myWord). Uses a Viterbi algorithm with unigram and bigram probabilities from a memory-mapped dictionary.

```python theme={null}
segmenter = DefaultSegmenter(word_engine="myword")

words = segmenter.segment_words("မြန်မာနိုင်ငံသည်ကောင်းသည်")
# ['မြန်မာ', 'နိုင်ငံ', 'သည်', 'ကောင်း', 'သည်']

# Load additional custom words into the myword dictionary
segmenter.load_custom_dictionary(["ကျွန်တော်တို့", "မိသားစု"])
```

### CRF Engine

CRF-based sequence tagger trained on [myPOS corpus by Ye Kyaw Thu](https://github.com/ye-kyaw-thu/myPOS). Requires `pycrfsuite`.

```bash theme={null}
pip install pycrfsuite
```

```python theme={null}
segmenter = DefaultSegmenter(word_engine="crf")
words = segmenter.segment_words("မြန်မာနိုင်ငံသည်")
```

### Transformer Engine

XLM-RoBERTa model fine-tuned for Myanmar word boundary detection by [Chuu Htet Naing](https://huggingface.co/chuuhtetnaing/myanmar-text-segmentation-model). Uses B/I (Begin/Inside) token classification.

```bash theme={null}
pip install myspellchecker[transformers]
```

```python theme={null}
segmenter = DefaultSegmenter(
    word_engine="transformer",
    seg_device=0,  # GPU for speed
)
words = segmenter.segment_words("မြန်မာနိုင်ငံသည်")
```

| Attribute    | Value                                                                                                                 |
| ------------ | --------------------------------------------------------------------------------------------------------------------- |
| **Model**    | [chuuhtetnaing/myanmar-text-segmentation-model](https://huggingface.co/chuuhtetnaing/myanmar-text-segmentation-model) |
| **Base**     | XLM-RoBERTa                                                                                                           |
| **Accuracy** | 96.17%                                                                                                                |
| **F1 Score** | 78.66%                                                                                                                |

***

## Segmenter Interface

All segmenters implement the `Segmenter` abstract base class:

```python theme={null}
from myspellchecker.segmenters.base import Segmenter

class Segmenter(ABC):
    @abstractmethod
    def segment_syllables(self, text: str) -> list[str]:
        """Segment text into syllables."""

    @abstractmethod
    def segment_words(self, text: str) -> list[str]:
        """Segment text into words."""

    @abstractmethod
    def segment_sentences(self, text: str) -> list[str]:
        """Segment text into sentences."""

    def segment_and_tag(self, text: str) -> tuple[list[str], list[str]]:
        """Segment and POS-tag simultaneously. Optional — raises NotImplementedError by default."""
        raise NotImplementedError
```

***

## DefaultSegmenter

The production segmenter that combines RegexSegmenter (syllables) with a configurable word engine:

```python theme={null}
from myspellchecker.segmenters import DefaultSegmenter

segmenter = DefaultSegmenter(
    word_engine="myword",           # "myword" | "crf" | "transformer"
    allow_extended_myanmar=False,   # Extended Myanmar Unicode blocks
    seg_model=None,                 # Custom model path (transformer only)
    seg_device=-1,                  # -1=CPU, 0+=GPU (transformer only)
)
```

***

## Usage with SpellChecker

### Via Configuration

```python theme={null}
from myspellchecker import SpellChecker
from myspellchecker.core.config import SpellCheckerConfig
from myspellchecker.providers import SQLiteProvider

config = SpellCheckerConfig(
    word_engine="myword",  # or "crf" or "transformer"
)
provider = SQLiteProvider(database_path="path/to/dictionary.db")
checker = SpellChecker(config=config, provider=provider)
```

### Custom Segmenter

```python theme={null}
from myspellchecker import SpellChecker
from myspellchecker.segmenters import DefaultSegmenter

segmenter = DefaultSegmenter(word_engine="crf")
checker = SpellChecker(segmenter=segmenter)
```

### Via Builder

```python theme={null}
from myspellchecker.core.builder import SpellCheckerBuilder
from myspellchecker.segmenters import DefaultSegmenter

segmenter = DefaultSegmenter(word_engine="crf")
checker = SpellCheckerBuilder().with_segmenter(segmenter).build()
```

***

## Performance Comparison

| Segmenter                      | Level    | Speed                   | Memory         | Dependencies                       |
| ------------------------------ | -------- | ----------------------- | -------------- | ---------------------------------- |
| RegexSegmenter                 | Syllable | Very Fast               | Very Low       | None                               |
| DefaultSegmenter (myword)      | Word     | Fast                    | Low            | Downloads mmap dictionary          |
| DefaultSegmenter (crf)         | Word     | Medium                  | Low            | `pycrfsuite` + downloads CRF model |
| DefaultSegmenter (transformer) | Word     | Slow (CPU) / Fast (GPU) | High (\~500MB) | `transformers`, `torch`            |

***

## Sentence Boundaries

All segmenters split on Myanmar sentence separator (။):

```python theme={null}
text = "ပထမစာကြောင်း။ ဒုတိယစာကြောင်း။ တတိယစာကြောင်း။"
sentences = segmenter.segment_sentences(text)
# ['ပထမစာကြောင်း။', 'ဒုတိယစာကြောင်း။', 'တတိယစာကြောင်း။']
```

`DefaultSegmenter` also detects sentence-final particles (SFPs) as implicit sentence boundaries in longer texts.

***

## See Also

* [Segmentation Algorithm](/algorithms/segmentation) - Segmentation internals
* [Syllable Segmentation](/algorithms/syllable-segmentation) - Sylbreak algorithm details
* [Text Normalization](/features/normalization) - Pre-processing before segmentation
* [Joint Segmentation and Tagging](/features/pos-tagging#joint-segmentation-and-tagging) - Unified Viterbi segmentation + POS
* [Cython Guide](/guides/cython) - Performance optimization