Performance Tuning

​

1. Caching

• Setting: provider_config.cache_size (default 1024) via ProviderConfig.
• Advice: If you have RAM, increase this to 10,000 or more. This drastically improves speed for common words.

​

2. Validation Levels

• Action: Set use_context_checker=False in config.
• Result: Removes the N-gram lookup step, providing a speed boost.

​

3. Async Execution

• Why: The core check logic is CPU-bound. Running it synchronously blocks the event loop. check_async offloads it to a thread.

​

4. Memory vs. Disk

• MemoryProvider: If startup time (~5-10s) is acceptable and you have 500MB+ RAM, this is the fastest option.
• SQLiteProvider: Instant startup, low RAM. Best for CLIs or limited environments.

​

5. Batch Processing

• While currently a wrapper, future versions may parallelize this operation.

​

6. Data Pipeline Performance (Building Dictionaries)

• Sharding: The pipeline automatically shards input for parallel processing. Ensure your machine has multiple cores available.
• Disk I/O: Use a fast SSD. Intermediate files (shards) are written to disk to keep RAM usage low.
• Word Engine: The default myword engine is a custom rule-based segmenter. For maximum speed during build, ensure the Cython extensions are compiled properly, which speeds up the segmentation loop by ~10x.

​

7. POS Tagging & Segmentation Performance

​

Tagger Selection

​

Joint Segmentation

• Cost: Slower than sequential mode because the state space is larger (Words × Tags).
• Beam Width: Controlled by joint.beam_width (default 15).
  • Lower (e.g., 5-10): Faster, slightly less accurate.
  • Higher (e.g., 20+): Slower, diminishing returns on accuracy.

​

8. Connection Pool Configuration

​

Pool Sizing

from myspellchecker.core.config.validation_configs import ConnectionPoolConfig

pool_config = ConnectionPoolConfig(
    min_size=2,      # Minimum connections to maintain
    max_size=10,     # Maximum connections allowed
    timeout=5.0,     # Checkout timeout in seconds
)

​

SQLite Timeout

pool_config = ConnectionPoolConfig(
    sqlite_timeout=30.0,  # Wait up to 30s for database lock
)

• Default: 30 seconds
• High contention: Increase to 60-120 seconds
• Low contention: Decrease to 5-10 seconds for faster failure

​

Connection Age

pool_config = ConnectionPoolConfig(
    max_connection_age=3600.0,  # Recreate connections after 1 hour
)

​

9. N-gram Context Checker Performance

​

Smoothing Strategy

from myspellchecker.algorithms.ngram_context_checker import NgramContextChecker
from myspellchecker.core.config import NgramContextConfig

# Stupid Backoff (default) - fast and effective
checker = NgramContextChecker(
    provider=provider,
    config=NgramContextConfig(smoothing_strategy="stupid_backoff"),
)

# No smoothing - fastest (for pre-smoothed data)
checker = NgramContextChecker(
    provider=provider,
    config=NgramContextConfig(smoothing_strategy="none"),
)

​

10. Edit Distance Performance

• Ensure Cython extensions are compiled (python setup.py build_ext --inplace)
• Cython version is ~10x faster than pure Python fallback
• Use damerau_levenshtein_distance for integer distances (fastest)
• Use weighted_damerau_levenshtein_distance for float distances (more accurate)