Senior Data Engineer: Scalable Pipelines & ML Ops

SINGAPORE AIRLINES LIMITED; Singapore; On-site; SGD 80,000 - 120,000; Full time

Job summary

SINGAPORE AIRLINES LIMITED is looking for a Lead Data Engineer to develop custom open‑source data ingestion and MLOps platforms. This senior role requires a strong software engineering background and the ability to tackle complex data engineering challenges.

Candidates should possess at least a BS in Computer Science and over 5 years of industry experience, with advanced skills in Python, AWS cloud components, and hands-on experience with Databricks and Apache Airflow. The position demands innovative thinking to improve data ingestion workflows.

Qualifications

• BS in Computer Science or related discipline is required.
• 5+ years of relevant industry experience in data engineering fields.
• Advanced programming skills in Python and conversant with data structures.

Responsibilities

• Design and develop data engineering services and their ecosystem.
• Develop MLOps production pipelines for data scientists.
• Design, build, and operate scalable data pipelines.

Skills

Advanced programming skills in Python

Experience in building data pipelines

Knowledge of AWS components

Hands-on experience with Databricks

Understanding of CI/CD and agile development

Education

BS in Computer Science or related discipline

Advanced degree in Computer Science (PhD, MS)

Tools

AWS (S3, RedShift, Elasticache, EMR)

Databricks

Apache Airflow

Kafka

Spark

Job description

Job Description

The lead data engineer is a senior software developer with strong softwareengineering skills who is responsible for building custom open-source-baseddata ingestion and MLOps platforms. He/she has deep appreciation of thecomplexity of the data engineering process, such as the challenges of dataingestion involving large or near-real-time datasets, the maintenance of highdata quality, and the importance of automation for increasing pipelinerobustness and reducing the need for human intervention.

Responsibilities

• Be an effective distributed-system implementer in the following coreactivities:

o Design and develop data engineering services and their ecosystem usingdistributed databases (relational, columnar, graph, in-memory); orchestration(Apache Airflow); and distributed stream/batch data processing (Kafka, Kinesis,Spark).

oDesign and develop MLOps production pipelines; provide technical support todata scientists/ML engineers by getting their ML/DL models deployed at scaleand meeting SLAs on both cloud and on-premises GPU and CPU instances.

o Design data models for mission-critical, high-volume, near-real-time/batchdata; build idempotent/atomic production data pipelines to make data ingestionmore fault tolerant.

o Design and develop intuitive, highly automated, self-service data platformfunctions for business users.

o Design, build, and operate scalable and reliable data pipelines on theDatabricks platform.

• Explore, evaluate and champion the introduction of next-generationtechnologies in the data-ingestion workflow. Participate in project planningand provide technical guidance on cloud architecture for data projects.

Requirements

• BS in Computer Science or other related discipline is required. Advanceddegrees in Computer Science (PhD, MS) are highly desirable.

•5+ years of relevant industry experience in some or most of the followingtechnical areas:

o Advanced programming skills in Python. Conversant with data structures andalgorithm design.

o Experience in building data pipelines (including data collection,warehousing, processing, analysis, monitoring, and governance) usingopen-source data ingestion platforms.

o Intermediate-level knowledge and experience with AWS cloud components andbest practices. Good understanding in deploying data stores such as S3,RedShift, Elasticache, PostgreSQL, and EMR.

o Hands on experience with Databricks workspace, cluster management, AI Agentcapabilities, and job orchestration

o Prior experience in modern software development is required (such as webfrontend UI, backend API microservices, understanding of CI/CD and Scrum/Kanbanagile development). Strong grasp on object-oriented or functional programming(using e.g. Python, Java, Scala, or C#).