AI Data Engineer

Aquent; Redmond; Presencial; AUD 120.000 - 150.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flexible work options

Descripción de la vacante

Aquent is partnering with an industry-leading global technology pioneer to shape the future of AI. In this role, you will design and operate robust Python-based workflows that turn complex document datasets into high-quality training inputs for AI models.

You will collaborate with applied scientists to translate data requirements into scalable processing pipelines, including OCR, data cleaning, and dataset preparation, with a focus on performance, reliability, and cost optimization.

Formación

• Five years of hands-on data engineering and data pipeline development with production programming responsibilities.
• Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors.
• Experience processing large structured and unstructured datasets using memory-efficient, parallel, or distributed techniques.
• Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting.
• Familiarity with Python data-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet.
• Working knowledge of SQL, Git, Linux environments, and command-line tools.
• Educational background in computer science, data science, mathematics, or a related technical field.

Responsabilidades

• Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication.
• Run and monitor large-scale batch processing jobs; implement retry and recovery mechanisms.
• Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies.
• Prepare training-ready datasets through filtering, text segmentation, tokenization, and conversion into structured formats.
• Manage bulk uploads, downloads, and synchronization between local environments and Azure storage, ensuring data integrity and completeness.
• Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability.
• Optimize processing throughput, memory usage, storage costs using parallel or distributed processing approaches.
• Track dataset statistics, processing coverage, quality metrics, and exceptions to produce clear operational reports.

Conocimientos

Python programming

Data engineering

Troubleshooting

Educación

Computer science / data science / mathematics degree

Herramientas

Azure Blob Storage

Azure Data Lake Storage

Pandas

NumPy

JSON/JSONL

CSV

Parquet

Git

Linux

Azure Databricks

Azure Machine Learning

Azure Data Factory

Hugging Face Datasets

Tokenization

Descripción del empleo

Partnering with an industry-leading global technology pioneer through Aquent, you will have the unique opportunity to shape the future of artificial intelligence. In this dynamic role, you will directly impact the development of next-generation AI models by building and optimizing the critical data pipelines that power them. If you are passionate about transforming massive, unstructured datasets into high-quality training data and want to see your work influence cutting-edge technology used by millions worldwide, this is the perfect opportunity to elevate your career.

The Opportunity

As a key contributor to an advanced applied science team, you will design and operate robust Python-based workflows that turn complex document and text datasets into high-quality inputs for AI model training. This hands-on position bridges the gap between raw data and sophisticated AI, combining OCR processing, advanced data cleaning, and scalable pipeline development. You will collaborate closely with applied scientists to translate complex data requirements into reproducible, high-performance processing workflows, making a tangible impact on the capabilities of modern AI models.

Key Responsibilities

• Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication.
• Run and monitor large-scale batch processing jobs, proactively investigating failures and implementing reliable retry and recovery mechanisms.
• Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies.
• Prepare training-ready datasets through filtering, text segmentation, tokenization, and conversion into structured formats.
• Manage bulk uploads, downloads, and synchronization between local environments and Azure storage, ensuring data integrity and completeness.
• Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability.
• Optimize processing throughput, memory usage, storage consumption, and cloud costs using parallel or distributed processing approaches.
• Track dataset statistics, processing coverage, quality metrics, and exceptions to produce clear operational reports.

Required Qualifications

• At least five years of dedicated, hands-on data engineering, data pipeline development, or data science experience with substantial production programming responsibilities (a software development background alone without direct data management experience is not a fit).
• Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors.
• Demonstrated experience processing large structured and unstructured datasets using memory-efficient, parallel, or distributed techniques.
• Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting.
• Familiarity with Python data-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet.
• Working knowledge of SQL, Git, Linux environments, and command-line tools.
• Strong troubleshooting skills, meticulous attention to data quality, and the ability to communicate technical issues clearly.
• Educational background in computer science, data science, mathematics, or a related technical field.

Preferred Qualifications

• Prior experience working within large-scale enterprise environments.
• Experience with OCR pipelines, scanned documents, multilingual text, or document-layout processing.
• Experience with Azure Databricks, Azure Machine Learning, Azure Data Factory, or workflow orchestration tools.
• Familiarity with Hugging Face Datasets, tokenization, near-duplicate detection, and training/evaluation data separation.
• Experience with dataset lineage, reproducibility, containerized environments, and CI/CD.

Client Description:

Join a global technology leader driving innovation and empowering people and organizations worldwide to achieve more. Here, you'll collaborate with passionate, diverse teams to create impactful solutions that shape the future. The culture prioritizes growth, inclusion, and purpose—where your ideas are valued, and your work drives real change. Wherever your passion lies and wherever your career is headed—you’ll play a role in projects that touch billions of lives. With flexible work options and a supportive environment, you'll have the tools to thrive both personally and professionally. Be part of something that truly matters.

Aquent is an equal opportunity employer. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, and other legally protected characteristics.