> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myspellchecker.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Building Dictionaries

> Complete guide to building spell checking dictionaries using the CLI and Python API, with POS tagging, curated lexicons, and incremental updates.

Before you can spell-check Myanmar text, you need a dictionary database built from your corpus. This page walks through both the quick CLI path and the full Python API for fine-grained control.

## CLI Reference

### Basic Build

```bash theme={null}
# Build from text corpus
myspellchecker build --input corpus.txt --output dict.db

# Build sample database for testing
myspellchecker build --sample

# Build with POS tagging
myspellchecker build --input corpus.txt --output dict.db --pos-tagger transformer
```

### Build Options

| Option                 | Default                     | Description                                                                                           |
| ---------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------- |
| `--input FILE`         | Required                    | Input corpus file (TXT, CSV, TSV, JSON, JSONL, Parquet)                                               |
| `--output FILE`        | `mySpellChecker-default.db` | Output SQLite database path                                                                           |
| `--sample`             | —                           | Build a small sample database (no input needed)                                                       |
| `--incremental`        | —                           | Update existing database instead of rebuilding                                                        |
| `--min-frequency N`    | `50`                        | Minimum word frequency to include                                                                     |
| `--pos-tagger TYPE`    | None                        | POS tagger: `rule_based`, `viterbi`, or `transformer`. No POS tagging when omitted.                   |
| `--pos-model NAME`     | —                           | HuggingFace model for transformer tagger                                                              |
| `--pos-device ID`      | `-1`                        | Device for transformer POS tagger (-1=CPU, 0+=GPU)                                                    |
| `--num-workers N`      | CPU count                   | Parallel worker processes                                                                             |
| `--batch-size N`       | `10000`                     | Records per processing batch                                                                          |
| `--curated-input FILE` | —                           | CSV file with trusted vocabulary words                                                                |
| `--word-engine TYPE`   | `myword`                    | Word segmentation engine: `myword`, `crf`, or `transformer`                                           |
| `--validate`           | —                           | Pre-flight validation of input (no build)                                                             |
| `--no-enrich`          | —                           | Skip enrichment step (confusable pairs, compounds, collocations, register)                            |
| `--seg-model NAME`     | —                           | HuggingFace model name/path for transformer word segmentation (only with `--word-engine=transformer`) |
| `--seg-device ID`      | `-1`                        | Device for transformer segmentation inference (-1=CPU, 0+=GPU)                                        |
| `--curated-lexicon-hf` | —                           | Download and use official curated lexicon from HuggingFace (`thettwe/myspellchecker-resources`)       |
| `--no-dedup`           | —                           | Disable line deduplication during ingestion                                                           |
| `--no-desegment`       | —                           | Keep segmentation markers in text                                                                     |
| `--verbose` / `-v`     | —                           | Enable verbose logging with detailed timing breakdowns                                                |

<Note>
  Run `myspellchecker build --help` for additional flags including `--work-dir`, `--keep-intermediate`, `--col`, `--json-key`, and `--worker-timeout`.
</Note>

***

## Python API

### Basic Usage

```python theme={null}
from myspellchecker.data_pipeline import Pipeline

pipeline = Pipeline()
pipeline.build_database(
    input_files=["corpus.txt"],
    database_path="dict.db",
)
```

### With Configuration

```python theme={null}
from myspellchecker.data_pipeline import Pipeline, PipelineConfig

config = PipelineConfig(
    batch_size=10000,        # Records per batch
    num_shards=20,           # Shards for ingestion
    num_workers=4,           # Parallel workers (None = auto-detect)
    min_frequency=50,        # Minimum word frequency to include
    word_engine="myword",    # Word segmentation engine: "myword", "crf", or "transformer"
    keep_intermediate=False, # Keep intermediate Arrow files
    text_col="text",         # Column name for CSV/TSV
    json_key="text",         # Key name for JSON
)

pipeline = Pipeline(config=config)
pipeline.build_database(
    input_files=["corpus.txt"],
    database_path="dict.db",
)
```

### Building from Multiple Files

```python theme={null}
pipeline.build_database(
    input_files=[
        "general_corpus.txt",
        "domain_specific.txt",
        "organization_names.txt",
    ],
    database_path="combined.db",
)
```

***

## POS Tagging

Add POS tags to dictionary entries during build for grammar checking support:

```bash theme={null}
# CLI: rule-based (fast, no dependencies)
myspellchecker build --input corpus.txt --output dict.db --pos-tagger rule_based

# CLI: transformer (highest accuracy, requires GPU)
myspellchecker build --input corpus.txt --output dict.db \
  --pos-tagger transformer \
  --pos-model chuuhtetnaing/myanmar-pos-model \
  --pos-device 0
```

```python theme={null}
# Python API
from myspellchecker.core.config import POSTaggerConfig

config = PipelineConfig(
    pos_tagger=POSTaggerConfig(
        tagger_type="transformer",
        model_name="chuuhtetnaing/myanmar-pos-model",
        device=0,
    ),
)
pipeline = Pipeline(config=config)
pipeline.build_database(["corpus.txt"], "dict.db")
```

### POS Inference on Existing Database

Apply rule-based POS inference to an existing database without rebuilding:

```python theme={null}
from myspellchecker.data_pipeline import DatabasePackager

packager = DatabasePackager.from_existing("dictionary.db")
stats = packager.apply_inferred_pos(
    min_frequency=0,
    min_confidence=0.0,
)
packager.close()
print(f"Inferred POS for {stats['inferred']} words")
```

***

## Curated Lexicons

Curated words are trusted vocabulary inserted directly into the database before corpus processing. They are always recognized as valid regardless of corpus frequency.

### Create a Lexicon CSV

```csv theme={null}
word
ဆေးရုံ
ဆရာဝန်
လူနာ
ကုမ္ပဏီ
```

### Build with Curated Words

```bash theme={null}
myspellchecker build --input corpus.txt --output dict.db \
  --curated-input curated_lexicon.csv
```

### How Curated Words are Processed

| Scenario                     | frequency    | is\_curated |
| ---------------------------- | ------------ | ----------- |
| Curated only (not in corpus) | 0            | 1           |
| Curated + corpus overlap     | corpus\_freq | 1           |
| Corpus only                  | corpus\_freq | 0           |

Curated words are inserted first (`is_curated=1`, `frequency=0`), then corpus processing updates frequency while preserving the `is_curated` flag via `MAX()`.

***

## Incremental Updates

Add new data to an existing dictionary without rebuilding from scratch:

```bash theme={null}
myspellchecker build --input new_data.txt --output existing.db --incremental
```

```python theme={null}
pipeline.build_database(
    input_files=["new_data.txt"],
    database_path="existing.db",
    incremental=True,
)
```

The pipeline tracks processed files in a `processed_files` table to avoid reprocessing.

***

## Enrichment

After frequency computation, the pipeline runs an enrichment step that mines additional linguistic data from the corpus. Disable with `--no-enrich` on the CLI or `enrich=False` in `PipelineConfig`.

### What Gets Mined

| Enrichment          | Table                 | Purpose                                                                                                      |
| ------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------ |
| Confusable pairs    | `confusable_pairs`    | Phonetically/orthographically similar word pairs (aspiration swaps, medial swaps, tone marks, nasal endings) |
| Compound confusions | `compound_confusions` | Words that may be incorrectly split during segmentation (e.g., "မြန်မာ" split as "မြန်" + "မာ")              |
| Collocations        | `collocations`        | Statistically significant word pairs with PMI/NPMI scores                                                    |
| Register tags       | `register_tags`       | Formal/informal register classification based on marker co-occurrence                                        |

### Configuration

```python theme={null}
from myspellchecker.data_pipeline.config import PipelineConfig

config = PipelineConfig(
    # Master toggle
    enrich=True,

    # Individual toggles
    enrich_confusables=True,
    enrich_compounds=True,
    enrich_collocations=True,
    enrich_register=True,
)
```

### Enrichment Thresholds

Fine-tune the mining process via `EnrichmentConfig` (passed internally from `PipelineConfig`):

| Parameter                   | Default  | Description                                            |
| --------------------------- | -------- | ------------------------------------------------------ |
| `confusable_min_freq`       | `50`     | Minimum word frequency to generate confusable variants |
| `confusable_max_freq_ratio` | `1000.0` | Maximum frequency ratio between pair members           |
| `compound_min_freq`         | `100`    | Minimum compound frequency to include                  |
| `compound_min_split_count`  | `10`     | Minimum bigram count for split form                    |
| `compound_min_pmi`          | `2.0`    | Minimum PMI for compound pairs                         |
| `collocation_min_count`     | `20`     | Minimum bigram occurrences                             |
| `collocation_min_pmi`       | `3.0`    | Minimum PMI for collocations                           |
| `register_min_total`        | `50`     | Minimum co-occurrence count with register markers      |
| `register_threshold`        | `0.3`    | Score cutoff for formal/informal classification        |

### Confusable Pair Mining

Generates phonetic/orthographic variants for every word above a frequency threshold, then checks which variants are also valid dictionary words. Context overlap (cosine similarity of bigram context vectors) and frequency ratio are computed for each pair.

**Variant types mined:**

* Aspiration swaps (က↔ခ, ပ↔ဖ, etc.)
* Medial swaps (ျ↔ြ) and medial insertion/deletion
* Nasal ending confusion (န်↔မ်↔ံ)
* Stop-coda confusion
* Tone mark changes (visarga add/remove)
* Vowel length changes

### Compound Confusion Detection

Finds bigrams `(w1, w2)` where the concatenation `w1+w2` is a high-frequency dictionary word. Computes PMI to measure how strongly the parts associate:

```
PMI = log2( P(compound) / (P(w1) × P(w2)) )
```

### Collocation Mining

Extracts statistically significant word pairs using Pointwise Mutual Information. Normalized PMI (NPMI) provides a scale-independent score in \[-1, 1].

### Register Tagging

Classifies words as formal, informal, or neutral based on co-occurrence with register markers. Words appearing predominantly with formal sentence-final particles are tagged `formal`, and vice versa.

***

## Output Database Schema

```sql theme={null}
-- Core tables
syllables(id, syllable, frequency)
words(id, word, syllable_count, frequency, pos_tag, is_curated,
      inferred_pos, inferred_confidence, inferred_source)
bigrams(id, word1_id, word2_id, probability, count)
trigrams(id, word1_id, word2_id, word3_id, probability, count)
fourgrams(id, word1_id, word2_id, word3_id, word4_id, probability, count)
fivegrams(id, word1_id, word2_id, word3_id, word4_id, word5_id, probability, count)

-- POS probability tables (for Viterbi tagger)
pos_unigrams(pos, probability)
pos_bigrams(pos1, pos2, probability)
pos_trigrams(pos1, pos2, pos3, probability)

-- Enrichment tables (from --no-enrich to skip)
confusable_pairs(id, word1, word2, confusion_type, context_overlap, freq_ratio, suppress, source)
compound_confusions(id, compound, part1, part2, compound_freq, split_freq, pmi)
collocations(id, word1, word2, pmi, npmi, count)
register_tags(word, register, confidence, formal_count, informal_count)

-- Metadata and file tracking
metadata(key, value)
processed_files(path, mtime, size)
```

### Query Examples

```python theme={null}
# Lookup word frequency
cursor.execute("SELECT frequency FROM words WHERE word = ?", ("မြန်မာ",))

# Get bigram probability
cursor.execute("""
    SELECT b.probability
    FROM bigrams b
    JOIN words w1 ON b.word1_id = w1.id
    JOIN words w2 ON b.word2_id = w2.id
    WHERE w1.word = ? AND w2.word = ?
""", ("ထမင်း", "စား"))
```

### Verification

```python theme={null}
from myspellchecker.data_pipeline import DatabasePackager

# Verify an existing database (from_existing() opens the connection)
packager = DatabasePackager.from_existing("output.db")
packager.verify_database()
packager.print_stats()
packager.close()
```

***

## Performance

For large corpora, see [Optimization](/data-pipeline/optimization) for DuckDB acceleration (3-15x faster frequency counting) and Cython parallelization.

```bash theme={null}
# Tune for large corpora
myspellchecker build --input huge_corpus.txt \
  --num-workers 8 \
  --batch-size 500000
```

***

## See Also

* [Corpus Format](/data-pipeline/corpus-format) - Input file specifications
* [Optimization](/data-pipeline/optimization) - DuckDB, Cython, performance tuning
* [Custom Dictionaries](/guides/custom-dictionaries) - Curated lexicons, domain builds
* [Database Schema](/data-pipeline/database-schema) - Full schema reference
* [CLI Reference](/cli/index) - All CLI commands