> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myspellchecker.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingestion Stage

> The ingestion stage reads and parses input corpus files into a standardized format for processing.

The first step in the dictionary building pipeline converts raw corpus files (TXT, CSV, JSON, Parquet) into Arrow shards for efficient parallel processing downstream.

## Overview

<img src="https://mintcdn.com/myspellchecker/LrG59DCXVrHk60Tz/images/ingestion-stage-overview.png?fit=max&auto=format&n=LrG59DCXVrHk60Tz&q=85&s=bba78922caea91a41bbbaa6519c5c081" alt="Ingestion stage overview showing input files flowing through CorpusIngester to Arrow shards" width="400" height="405" data-path="images/ingestion-stage-overview.png" />

## Supported Formats

| Format     | Extension  | Use Case                           |
| ---------- | ---------- | ---------------------------------- |
| Plain Text | `.txt`     | Raw corpus text                    |
| CSV        | `.csv`     | Structured word lists              |
| TSV        | `.tsv`     | Tab-separated word lists           |
| JSON       | `.json`    | Complex data with metadata         |
| JSON Lines | `.jsonl`   | Streaming JSON                     |
| Parquet    | `.parquet` | Columnar format for large datasets |

## Using the CorpusIngester

### Basic Usage

The `CorpusIngester` reads corpus files and outputs Arrow shards for efficient processing:

```python theme={null}
from myspellchecker.data_pipeline import CorpusIngester

ingester = CorpusIngester()

# Generate sample corpus for testing
sample_lines = ingester.generate_sample_corpus()

# Process files into Arrow shards (used internally by Pipeline)
# The ingester produces Arrow shard files in the work directory
```

### With Pipeline (Recommended)

For most use cases, use the `Pipeline` class which manages ingestion automatically:

```python theme={null}
from myspellchecker.data_pipeline import Pipeline, PipelineConfig

config = PipelineConfig(
    text_col="text",         # Column name for CSV/TSV
    json_key="text",         # Key name for JSON
)

pipeline = Pipeline(config=config)
pipeline.build_database(
    input_files=["corpus.txt"],
    database_path="dict.db",
)
```

## Configuration Options

Configuration is handled through `PipelineConfig` and `IngesterConfig`:

```python theme={null}
from myspellchecker.data_pipeline import PipelineConfig, IngesterConfig

# Pipeline-level configuration
pipeline_config = PipelineConfig(
    text_col="text",         # Column name for CSV/TSV
    json_key="text",         # Key name for JSON
    num_shards=20,           # Number of Arrow shards to create
)

# Ingester-specific configuration
ingester_config = IngesterConfig(
    batch_size=10000,        # Records per batch for Arrow writing
    encoding="utf-8",        # File encoding
    skip_empty_lines=True,   # Skip empty lines during ingestion
    normalize_unicode=True,  # Apply Unicode normalization
)
```

| Option              | Default   | Description                 |
| ------------------- | --------- | --------------------------- |
| `batch_size`        | `10000`   | Records per batch           |
| `encoding`          | `"utf-8"` | File encoding               |
| `skip_empty_lines`  | `True`    | Skip empty lines            |
| `normalize_unicode` | `True`    | Apply Unicode normalization |

## Format-Specific Options

### Plain Text

```python theme={null}
# Text files are read line by line
pipeline = Pipeline(config=PipelineConfig())
pipeline.build_database(
    input_files=["corpus.txt"],
    database_path="dict.db",
)
```

### CSV/TSV

```python theme={null}
config = PipelineConfig(
    text_col="text",  # Column name containing text
)
```

### JSON

```python theme={null}
config = PipelineConfig(
    json_key="text",  # Key name containing text
)
```

### Parquet

Parquet files are read using PyArrow. The ingester automatically detects the text column:

1. Looks for a column named `text`
2. Falls back to the first string/large\_string column

```python theme={null}
# Create a Parquet corpus file
import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({"text": ["မြန်မာစာ", "ကောင်းပါတယ်"]})
pq.write_table(table, "corpus.parquet")

# Then use with pipeline
pipeline.build_database(
    input_files=["corpus.parquet"],
    database_path="dict.db",
)
```

## Streaming Large Files

The pipeline automatically handles large files by streaming and sharding:

```python theme={null}
from myspellchecker.data_pipeline import Pipeline, PipelineConfig

# Large files are automatically handled with parallel sharding
config = PipelineConfig(
    num_shards=50,       # More shards for better parallelization
    batch_size=50000,    # Larger batches for throughput
)

pipeline = Pipeline(config=config)
pipeline.build_database(
    input_files=["large_corpus.txt"],
    database_path="dict.db",
)
```

## Error Handling

The ingester validates input files and raises `IngestionError` for missing files:

```python theme={null}
from myspellchecker.data_pipeline import Pipeline, IngestionError

pipeline = Pipeline()

try:
    pipeline.build_database(
        input_files=["missing.txt"],
        database_path="dict.db",
    )
except IngestionError as e:
    print(f"Ingestion failed: {e}")
    print(f"Missing files: {e.missing_files}")
```

## Multiple Input Files

### Multiple Paths

```python theme={null}
pipeline = Pipeline()
pipeline.build_database(
    input_files=[
        "corpus1.txt",
        "corpus2.txt",
        "data/corpus3.csv",
    ],
    database_path="dict.db",
)
```

### Glob Patterns (via CLI)

```bash theme={null}
myspellchecker build --input "corpus/*.txt" --output dict.db
```

## Performance Tips

1. **Use multiple workers** for parallel ingestion
2. **Increase num\_shards** for large corpora
3. **Use SSD** for faster I/O

```python theme={null}
# Optimized for large files
config = PipelineConfig(
    num_shards=50,      # More shards for parallelization
    num_workers=8,      # More workers
    batch_size=50000,   # Larger batches
)
```

## Integration with Pipeline

```python theme={null}
from myspellchecker.data_pipeline import Pipeline, PipelineConfig

config = PipelineConfig(
    text_col="text",         # For CSV files
    json_key="text",         # For JSON files
)

pipeline = Pipeline(config=config)
pipeline.build_database(
    input_files=["corpus.txt"],
    database_path="dict.db",
)
```

## See Also

* [Corpus Format](/data-pipeline/corpus-format) - Input specifications
* [Processing Stage](/data-pipeline/processing) - Next pipeline stage
* [Pipeline Index](/data-pipeline/index) - Pipeline overview