Ai Data Engineering Lead

Optimum Solutions Pte Ltd; Singapore; On-site; SGD 180,000 - 280,000; Full time

Job summary

Optimum Solutions Pte Ltd is seeking an experienced AI Data Engineering Lead to design and orchestrate data platforms for AI, ML and GenAI initiatives in Singapore. You will lead scalable pipelines across structured and unstructured data and guide a team of data engineers.

You will implement data quality, governance, security controls, and CI/CD practices while partnering with architects, security, and business stakeholders to deliver reliable, cost-efficient solutions.

Qualifications

• 8+ years in data engineering with leadership experience.
• Hands-on AI/ML data prep, GenAI architectures, RAG.
• Degree in CS/IT or related field.
• Proficiency in Spark, PySpark, and data pipelines.

Responsibilities

• Lead scalable data platforms and pipelines for AI/ML and GenAI.
• Mentor data engineers and review code/architecture.
• Ensure data governance, security, and quality.
• Collaborate with architects and stakeholders.

Skills

Python

SQL

Mentoring

Data modeling

Leadership

Education

Bachelor's or Master's in Computer Science / Data Engineering

Tools

Apache Spark

PySpark

Airflow

Databricks

Snowflake

Kafka

CI/CD

Job description

Our client is seeking an experienced AI Data Engineering Lead to design, build, and lead data engineering solutions that support Artificial Intelligence (AI), Machine Learning (ML), and Generative AI (GenAI) initiatives.

Responsibilities

• Lead the design and development of scalable data platforms and pipelines to support AI, ML, analytics, and GenAI applications.
• Build and maintain data pipelines for structured, semi-structured, and unstructured data from enterprise systems.
• Develop batch and real-time data processing solutions to support AI model training, inference, and evaluation.
• Design data ingestion and processing workflows for GenAI applications, including Retrieval-Augmented Generation (RAG), document processing, embedding generation, and vector search.
• Integrate enterprise data platforms with AI services, search engines, vector databases, and other relevant technologies.
• Establish data engineering standards, reusable frameworks, automated testing, CI/CD, and DataOps practices to improve development efficiency and solution quality.
• Implement data quality checks, data lineage, metadata management, monitoring, and error-handling mechanisms to ensure data accuracy and reliability.
• Ensure data engineering solutions comply with data governance, information security, privacy, and regulatory requirements.
• Work with security, risk, compliance, and data governance teams to implement appropriate access controls and safeguards for sensitive data.
• Lead and mentor data engineers, conduct code and architecture reviews, and provide technical guidance throughout the development lifecycle.
• Collaborate with enterprise architects, AI engineers, platform teams, and business stakeholders to translate requirements into effective technical solutions.
• Continuously improve data pipeline performance, platform scalability, operational resilience, and infrastructure cost efficiency.

Requirements

• At least 8 years of experience in data engineering, data platforms, or related technology roles, with demonstrated technical leadership experience.
• Practical understanding of AI/ML data preparation, Generative AI architectures, RAG pipelines, document processing, embeddings, and vector search.
• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
• Strong programming skills in Python and SQL, with hands-on experience in Apache Spark, PySpark, or equivalent distributed processing technologies.
• Proven experience designing and implementing enterprise data pipelines, data lakes, lakehouses, data warehouses, and data integration solutions.
• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud, and workflow orchestration tools such as Apache Airflow or equivalent.
• Experience with CI/CD, automated testing, monitoring, and production support for data platforms.
• Strong understanding of data quality, data governance, security controls, data lineage, and access management.
• Experience in banking, financial services, or another regulated industry is a plus.
• Familiarity with technologies such as Databricks, Snowflake, Kafka, Azure AI Search, MLOps, or LLMOps is an advantage.