AI Data Pipeline Engineer

Aditi Consulting; Redmond; On-site; AUD 186,000 - 198,000; Full time

Benefits offered by this job

Medical, dental, and vision benefits

401(k) plan with employer match

Paid time off and bereavement leave

Job summary

Aditi Consulting seeks a hands-on Data Pipeline Engineer to build and operate Python-based workflows for transforming large document and text datasets into high-quality data for AI model training. The role includes OCR processing, data cleaning, scalable pipeline development, and Azure storage management.

Responsibilities include data ingestion, OCR, text extraction, validation, and preparing training-ready datasets, while optimizing throughput, memory, and cloud costs.

Qualifications

• Demonstrated experience in data engineering or production data pipelines.
• Strong Python proficiency with modular code, testing, and logging.
• Experience with memory-efficient processing of large datasets.
• Practical Azure storage and data lake experience.

Responsibilities

• Develop, maintain, and optimize Python pipelines for ingestion, OCR, extraction, cleaning, and deduplication.
• Run and monitor large-scale batch jobs, investigate failures, implement retries.
• Build automated data quality checks for missing/corrupted files and metadata issues.
• Prepare training-ready datasets via filtering, tokenization, and format conversion.
• Manage bulk transfers between local environments and Azure storage ensuring integrity.
• Organize data with consistent naming, metadata, and versioning.

Skills

Python programming

Data pipeline development

Troubleshooting

Communication

Education

Tools

Azure Blob Storage

Azure Data Lake Storage

Pandas/NumPy

Parquet/JSONL/CSV

Git

Linux

Job description

Aditi Consulting seeks a hands-on Data Pipeline Engineer to build and operate Python-based workflows for transforming large document and text datasets into high-quality data for AI model training. The role includes OCR processing, data cleaning, scalable pipeline development, and Azure storage management.

Responsibilities include data ingestion, OCR, text extraction, validation, and preparing training-ready datasets, while optimizing throughput, memory, and cloud costs.