Ai Data Engineering Lead

optimum solutions (singapore) pte ltd; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

Optimum Solutions (Singapore) Pte Ltd is seeking an experienced AI Data Engineering Lead to design, build, and lead data engineering solutions that power AI, ML and GenAI initiatives. You will guide the development of scalable data platforms, and data pipelines across enterprise systems, enabling model training, inference and evaluation.

The role requires expertise in Python/SQL, Spark, Airflow, cloud platforms, and DataOps, with responsibility for governance, security, and collaboration with

Qualifications

• 8+ years in data engineering or related roles with leadership experience.
• Practical understanding of AI/ML data prep, GenAI architectures, RAG pipelines, embeddings and vector search.
• Degree in computer science, data engineering, IT, or related discipline.
• Strong Python and SQL skills with Spark/PySpark or equivalent.
• Experience designing enterprise data pipelines, lakes, lakehouses, and data warehouses.
• Experience with cloud platforms and workflow orchestration tools like Airflow.
• Experience with CI/CD, monitoring, and production support for data platforms.
• Knowledge of data quality, governance, security, lineage, and access management.
• Experience in banking/financial services or regulated industries is a plus.
• Familiarity with Databricks, Snowflake, Kafka, Azure AI Search, MLOps or LLMOps is advantageous.

Responsibilities

• Lead the design and development of scalable data platforms and pipelines to support AI, ML, analytics, and GenAI applications.
• Build and maintain data pipelines for structured, semi-structured, and unstructured data from enterprise systems.
• Develop batch and real-time data processing solutions to support AI model training, inference, and evaluation.
• Design data ingestion and processing workflows for GenAI applications, including RAG, document processing, embedding generation, and vector search.
• Integrate enterprise data platforms with AI services, search engines, vector databases, and other relevant technologies.
• Establish data engineering standards, reusable frameworks, automated testing, CI/CD, and DataOps practices to improve development efficiency and solution quality.
• Implement data quality checks, data lineage, metadata management, monitoring, and error-handling mechanisms to ensure data accuracy and reliability.
• Ensure data engineering solutions comply with data governance, information security, privacy, and regulatory requirements.
• Work with security, risk, compliance, and data governance teams to implement appropriate access controls and safeguards for sensitive data.
• Lead and mentor data engineers, conduct code and architecture reviews, and provide technical guidance throughout the development lifecycle.
• Collaborate with enterprise architects, AI engineers, platform teams, and business stakeholders to translate requirements into effective technical solutions.
• Continuously improve data pipeline performance, platform scalability, operational resilience, and infrastructure cost efficiency.

Skills

Python

SQL

Spark

Airflow

Data governance

CI/CD

Cloud platforms

MLOps/GenAI

Education

Bachelor's or Master's in CS/IT

Tools

Databricks

Snowflake

Kafka

Azure/AWS/GCP

Job description

Our client is seeking an experienced AI Data Engineering Lead to design, build, and lead data engineering solutions that support Artificial Intelligence (AI), Machine Learning (ML), and Generative AI (GenAI) initiatives.

Responsibilities

• Lead the design and development of scalable data platforms and pipelines to support AI, ML, analytics, and GenAI applications.
• Build and maintain data pipelines for structured, semi-structured, and unstructured data from enterprise systems.
• Develop batch and real-time data processing solutions to support AI model training, inference, and evaluation.
• Design data ingestion and processing workflows for GenAI applications, including Retrieval-Augmented Generation (RAG), document processing, embedding generation, and vector search.
• Integrate enterprise data platforms with AI services, search engines, vector databases, and other relevant technologies.
• Establish data engineering standards, reusable frameworks, automated testing, CI/CD, and DataOps practices to improve development efficiency and solution quality.
• Implement data quality checks, data lineage, metadata management, monitoring, and error-handling mechanisms to ensure data accuracy and reliability.
• Ensure data engineering solutions comply with data governance, information security, privacy, and regulatory requirements.
• Work with security, risk, compliance, and data governance teams to implement appropriate access controls and safeguards for sensitive data.
• Lead and mentor data engineers, conduct code and architecture reviews, and provide technical guidance throughout the development lifecycle.
• Collaborate with enterprise architects, AI engineers, platform teams, and business stakeholders to translate requirements into effective technical solutions.
• Continuously improve data pipeline performance, platform scalability, operational resilience, and infrastructure cost efficiency.

Requirements

• At least 8 years of experience in data engineering, data platforms, or related technology roles, with demonstrated technical leadership experience.
• Practical understanding of AI/ML data preparation, Generative AI architectures, RAG pipelines, document processing, embeddings, and vector search.
• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
• Strong programming skills in Python and SQL, with hands-on experience in Apache Spark, PySpark, or equivalent distributed processing technologies.
• Proven experience designing and implementing enterprise data pipelines, data lakes, lakehouses, data warehouses, and data integration solutions.
• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud, and workflow orchestration tools such as Apache Airflow or equivalent.
• Experience with CI/CD, automated testing, monitoring, and production support for data platforms.
• Strong understanding of data quality, data governance, security controls, data lineage, and access management.
• Experience in banking, financial services, or another regulated industry is a plus.
• Familiarity with technologies such as Databricks, Snowflake, Kafka, Azure AI Search, MLOps, or LLMOps is an advantage.