> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myspellchecker.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cython Development Guide

> This guide covers how to work with the Cython (.pyx) modules in mySpellChecker.

Learn how to build, test, debug, and contribute Cython extensions, including OpenMP parallel processing and the Python fallback pattern.

## Overview

mySpellChecker uses Cython for performance-critical operations:

| Module                  | Purpose                     | Performance Gain |
| ----------------------- | --------------------------- | ---------------- |
| `batch_processor.pyx`   | Parallel batch processing   | 5-10x            |
| `frequency_counter.pyx` | Fast frequency calculations | 3-5x             |
| `normalize_c.pyx`       | Text normalization          | 2-3x             |
| `edit_distance_c.pyx`   | Levenshtein distance        | 4-6x             |
| `viterbi.pyx`           | Viterbi algorithm for POS   | 3-4x             |
| `word_segment.pyx`      | Word segmentation           | 2-3x             |
| `mmap_reader.pyx`       | Memory-mapped file reading  | 2-4x             |
| `syllable_rules_c.pyx`  | Syllable validation rules   | 2-3x             |
| `ingester_c.pyx`        | Corpus ingestion            | 2-3x             |
| `repair_c.pyx`          | Segmentation repair         | 2-3x             |
| `tsv_reader_c.pyx`      | TSV file reading            | 2-3x             |

## Prerequisites

### Required Tools

```bash theme={null}
# macOS
brew install python cython libomp

# Ubuntu/Debian
sudo apt-get install python3-dev cython3 libomp-dev

# Windows (with Visual Studio)
pip install cython
```

### Verify Installation

```bash theme={null}
# Check Cython version
cython --version

# Check C++ compiler
c++ --version  # macOS/Linux
cl             # Windows
```

## Project Structure

<Tree>
  <Tree.Folder name="src/myspellchecker" defaultOpen>
    <Tree.Folder name="text" defaultOpen>
      <Tree.File name="normalize.py" />

      <Tree.File name="normalize_c.pyx" />

      <Tree.File name="normalize_c.pxd" />
    </Tree.Folder>

    <Tree.Folder name="algorithms" defaultOpen>
      <Tree.File name="viterbi.py" />

      <Tree.File name="viterbi.pyx" />

      <Tree.Folder name="distance" defaultOpen>
        <Tree.File name="edit_distance_c.pyx" />
      </Tree.Folder>
    </Tree.Folder>

    <Tree.Folder name="data_pipeline" defaultOpen>
      <Tree.File name="batch_processor.py" />

      <Tree.File name="batch_processor.pyx" />
    </Tree.Folder>
  </Tree.Folder>
</Tree>

### File Types

| Extension      | Purpose                               | Git Tracked? |
| -------------- | ------------------------------------- | ------------ |
| `.pyx`         | Cython source code                    | Yes          |
| `.pxd`         | C-level declarations (like C headers) | Yes          |
| `.py`          | Python wrapper/fallback               | Yes          |
| `.cpp`         | Generated C++ code                    | No           |
| `.so` / `.pyd` | Compiled binary                       | No           |

## Building Cython Extensions

### Development Build

```bash theme={null}
# Rebuild after modifying .pyx files
python setup.py build_ext --inplace

# Clean and rebuild
python setup.py clean --all
python setup.py build_ext --inplace
```

### Build with Debug Symbols

```bash theme={null}
# For debugging with gdb/lldb
CFLAGS="-g -O0" python setup.py build_ext --inplace
```

### Build Options

The `setup.py` automatically detects:

* OpenMP availability (macOS requires `brew install libomp`)
* C++ compiler capabilities
* Platform-specific flags

## Writing Cython Code

### Basic Pattern

```python theme={null}
# normalize_c.pyx
from cpython cimport PyUnicode_AsUTF8String
from libc.string cimport memcpy, strlen

cdef class Normalizer:
    """Cython normalizer for Myanmar text."""

    cdef str _text

    def __init__(self, str text):
        self._text = text

    cpdef str normalize(self):
        """Public method callable from Python."""
        return self._normalize_impl()

    cdef str _normalize_impl(self):
        """Private C-level method (faster, not callable from Python)."""
        # Implementation here
        pass
```

### Creating .pxd Files

```python theme={null}
# normalize_c.pxd
cdef class Normalizer:
    cdef str _text
    cpdef str normalize(self)
    cdef str _normalize_impl(self)
```

### Cross-Module Imports

```python theme={null}
# corpus_segmenter.pyx
from myspellchecker.text.normalize_c cimport Normalizer

cdef class CorpusSegmenter:
    cdef Normalizer normalizer

    def __init__(self):
        self.normalizer = Normalizer("")
```

## Import Pattern

The core `normalize.py` module imports directly from the Cython extension
without pure Python fallbacks:

```python theme={null}
# normalize.py - direct Cython imports (no fallback)
from myspellchecker.text.normalize_c import (
    get_myanmar_ratio as c_get_myanmar_ratio,
)
from myspellchecker.text.normalize_c import (
    remove_zero_width_chars as c_remove_zero_width,
)
from myspellchecker.text.normalize_c import (
    reorder_myanmar_diacritics as c_reorder_diacritics,
)

def normalize(text: str, form: str = "NFC", ...) -> str:
    """Main normalization function (public API)."""
    text = c_remove_zero_width(text)
    text = c_reorder_diacritics(text)
    # ... more normalization steps
    return text
```

> **Note**: Unlike some other modules that use try/except ImportError fallbacks,
> `normalize.py` requires the Cython extension. For systems without a C++ compiler,
> install from a pre-built wheel.

## OpenMP Integration

For parallel processing (used in `batch_processor.pyx`):

```python theme={null}
# corpus_segmenter.pyx
from cython.parallel cimport prange, parallel
from openmp cimport omp_get_num_threads, omp_set_num_threads

cdef class CorpusSegmenter:
    cdef int num_threads

    def __init__(self, int num_threads=0):
        if num_threads <= 0:
            self.num_threads = omp_get_num_threads()
        else:
            self.num_threads = num_threads

    cpdef list process_batch(self, list items):
        cdef int n = len(items)
        cdef int i
        results = [None] * n

        with nogil, parallel(num_threads=self.num_threads):
            for i in prange(n, schedule='dynamic'):
                with gil:
                    results[i] = self._process_item(items[i])

        return results
```

### macOS OpenMP Setup

```bash theme={null}
# Install OpenMP
brew install libomp

# Set environment variables (add to ~/.zshrc)
export LDFLAGS="-L/opt/homebrew/opt/libomp/lib"
export CPPFLAGS="-I/opt/homebrew/opt/libomp/include"
```

## Testing Cython Code

### Unit Tests

```python theme={null}
# tests/test_normalize_c.py
import pytest
from myspellchecker.text.normalize import normalize
from myspellchecker.text.normalize_c import remove_zero_width_chars

class TestNormalize:
    def test_zero_width_removal(self):
        result = remove_zero_width_chars("test\u200btext")
        assert result == "testtext"

    def test_basic_normalization(self):
        result = normalize("မြန်မာ")
        assert isinstance(result, str)

    def test_cython_required_for_normalize(self):
        """normalize_c is required -- import failure means build is broken."""
        from myspellchecker.text.normalize_c import remove_zero_width_chars
        assert callable(remove_zero_width_chars)

    def test_cython_optional_for_viterbi(self):
        """viterbi has an optional Cython backend with Python fallback."""
        from myspellchecker.algorithms.viterbi import _HAS_CYTHON_VITERBI
        # Test passes either way -- just reports which backend is active
        print(f"Cython viterbi: {_HAS_CYTHON_VITERBI}")

    def test_performance(self):
        """Test normalization performance."""
        import time
        text = "test" * 1000

        start = time.perf_counter()
        for _ in range(1000):
            remove_zero_width_chars(text)
        elapsed = time.perf_counter() - start

        assert elapsed < 1.0  # Adjust threshold as needed
```

### Benchmark Tests

```bash theme={null}
# Run benchmarks
pytest tests/ -k benchmark --benchmark-only
```

## Debugging

### Print Debugging

```python theme={null}
# In .pyx file
from libc.stdio cimport printf

cdef void debug_print(const char* msg) nogil:
    printf("DEBUG: %s\n", msg)
```

### GDB/LLDB

```bash theme={null}
# Build with debug symbols
CFLAGS="-g -O0" python setup.py build_ext --inplace

# Debug with lldb (macOS)
lldb python
(lldb) run -c "from myspellchecker.text.normalize_c import remove_zero_width_chars"

# Debug with gdb (Linux)
gdb python
(gdb) run -c "from myspellchecker.text.normalize_c import remove_zero_width_chars"
```

### Memory Profiling

```bash theme={null}
# Check for memory leaks
valgrind --leak-check=full python -c "
from myspellchecker.text.normalize_c import remove_zero_width_chars
for _ in range(10000):
    remove_zero_width_chars('test')
"
```

## Common Pitfalls

### 1. Forgetting to Rebuild

After modifying `.pyx` files, always rebuild:

```bash theme={null}
python setup.py build_ext --inplace
```

### 2. GIL Management

```python theme={null}
# Wrong - will deadlock
cdef void process() nogil:
    # Python operations require GIL
    result = some_python_function()  # Error!

# Correct
cdef void process() nogil:
    with gil:
        result = some_python_function()
```

### 3. Memory Management

```python theme={null}
# Wrong - memory leak
cdef char* create_string():
    cdef char* s = <char*>malloc(100)
    return s  # Caller must free!

# Better - use Python strings
cdef str create_string():
    return "result"  # Python handles memory
```

### 4. Type Declarations

```python theme={null}
# Slow - Python object
def slow_function(x):
    return x * 2

# Fast - typed
cpdef int fast_function(int x):
    return x * 2
```

## Performance Tips

1. **Use `cdef` for internal functions** - Not callable from Python, but faster
2. **Use typed memoryviews** - Faster than NumPy arrays in loops
3. **Minimize GIL acquisition** - Use `nogil` where possible
4. **Use `cpdef` for hybrid** - Callable from Python and fast from Cython
5. **Profile before optimizing** - Use `cython -a` to see Python interactions

### Annotation Output

```bash theme={null}
# Generate HTML with Python interaction highlighting
cython -a normalize_c.pyx

# Yellow lines indicate Python interactions (slow)
# White lines are pure C (fast)
```

## Contributing

When contributing Cython code:

1. Include both `.pyx` and `.py` wrapper
2. Add `.pxd` file if cross-module imports needed
3. Write tests that work with both backends (where applicable)
4. Document performance characteristics
5. Test on multiple platforms if possible

### Cython Integration Patterns

Different modules use different integration patterns depending on whether a pure Python fallback is needed:

**Pattern 1: Required Cython (no Python fallback)**

`normalize.py` imports directly from `normalize_c.pyx` without a `try/except ImportError` guard. The Cython extension is required -- if it is not compiled, the import fails at runtime. This pattern is used when the Cython code is the only implementation.

```python theme={null}
# normalize.py - direct import, no fallback
from myspellchecker.text.normalize_c import remove_zero_width_chars
from myspellchecker.text.normalize_c import reorder_myanmar_diacritics
```

**Pattern 2: Optional Cython with Python fallback**

`viterbi.py` tries to import the Cython extension and falls back to a pure Python implementation if it is not available. The `_HAS_CYTHON_VITERBI` flag controls which backend runs at runtime.

```python theme={null}
# viterbi.py - optional Cython with fallback
try:
    from myspellchecker.algorithms import viterbi_c
    _HAS_CYTHON_VITERBI = True
except ImportError:
    _HAS_CYTHON_VITERBI = False

class ViterbiTagger:
    def tag_sequence(self, words):
        if _HAS_CYTHON_VITERBI and self._cython_tagger:
            return self._cython_tagger.tag_sequence(words)
        else:
            # Pure Python fallback
            ...
```

When writing tests, be aware of this distinction:

* For **required Cython** modules (like `normalize`), tests can import the Cython functions directly. If the extension is not compiled, the test will fail at import time, which is expected.
* For **optional Cython** modules (like `viterbi`), tests should verify both backends work. Check `_HAS_CYTHON_VITERBI` to report which backend is active, but the tests should pass either way.