Senior AI Data Engineer: Build Scalable ML Data Pipelines

Aquent; Redmond; Presencial; AUD 120.000 - 150.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flexible work options

Descripción de la vacante

Aquent is partnering with an industry-leading global technology pioneer to shape the future of AI. In this role, you will design and operate robust Python-based workflows that turn complex document datasets into high-quality training inputs for AI models.

You will collaborate with applied scientists to translate data requirements into scalable processing pipelines, including OCR, data cleaning, and dataset preparation, with a focus on performance, reliability, and cost optimization.

Formación

• Five years of hands-on data engineering and data pipeline development with production programming responsibilities.
• Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors.
• Experience processing large structured and unstructured datasets using memory-efficient, parallel, or distributed techniques.
• Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting.
• Familiarity with Python data-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet.
• Working knowledge of SQL, Git, Linux environments, and command-line tools.
• Educational background in computer science, data science, mathematics, or a related technical field.

Responsabilidades

• Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication.
• Run and monitor large-scale batch processing jobs; implement retry and recovery mechanisms.
• Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies.
• Prepare training-ready datasets through filtering, text segmentation, tokenization, and conversion into structured formats.
• Manage bulk uploads, downloads, and synchronization between local environments and Azure storage, ensuring data integrity and completeness.
• Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability.
• Optimize processing throughput, memory usage, storage costs using parallel or distributed processing approaches.
• Track dataset statistics, processing coverage, quality metrics, and exceptions to produce clear operational reports.

Conocimientos

Python programming

Data engineering

Troubleshooting

Educación

Computer science / data science / mathematics degree

Herramientas

Azure Blob Storage

Azure Data Lake Storage

Pandas

NumPy

JSON/JSONL

CSV

Parquet

Git

Linux

Azure Databricks

Azure Machine Learning

Azure Data Factory

Hugging Face Datasets

Tokenization

Descripción del empleo

Aquent is partnering with an industry-leading global technology pioneer to shape the future of AI. In this role, you will design and operate robust Python-based workflows that turn complex document datasets into high-quality training inputs for AI models.

You will collaborate with applied scientists to translate data requirements into scalable processing pipelines, including OCR, data cleaning, and dataset preparation, with a focus on performance, reliability, and cost optimization.