Information Technology - Lead Data Engineer

Singapore Airlines

Singapore

On-site

SGD 180,000 - 280,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Singapore Airlines Limited is seeking a Lead Data Engineer to architect and build open-source-based data ingestion and MLOps platforms. The role emphasizes handling large and near-real-time datasets with a focus on data quality and automation.

You will design data models, implement robust pipelines, and enable self-service data platform features for business users across cloud and on-prem environments. The position involves collaboration with data scientists and engineers to deliver scalable

Qualifications

  • BS in Computer Science or related field is required.
  • Advanced degrees (MS/PhD) are highly desirable.
  • 5+ years of experience in data engineering, data ingestion, and pipelines using open-source platforms.

Responsibilities

  • Design and develop data engineering services and their ecosystem using distributed databases, orchestration (Airflow), and stream/batch processing (Kafka, Kinesis, Spark).
  • Develop MLOps pipelines and support ML/DL models deployed at scale across cloud and on-premises environments.
  • Build scalable, fault-tolerant data ingestion pipelines and intuitive self-service data platform functions for business users.
  • Lead evaluation of next-generation technologies in data ingestion workflows and contribute to cloud-architecture discussions.

Skills

Python programming
Data pipelines
AWS cloud
Databricks
Airflow
CI/CD & Agile
Scala/Java/C#
Distributed systems

Education

BS in Computer Science
MS/PhD desirable

Tools

Databricks
Airflow
Kafka
Kinesis
S3
Redshift
PostgreSQL
Spark
EC2
EMR

Job description

Job Description

The lead data engineer is a senior software developer with strong software engineering skills who is responsible for building custom open-source-based data ingestion and MLOps platforms. He/she has deep appreciation of the complexity of the data engineering process, such as the challenges of data ingestion involving large or near-real-time datasets, the maintenance of high data quality, and the importance of automation for increasing pipeline robustness and reducing the need for human intervention.

Responsibilities
  • Be an effective distributed-system implementer in the following core activities:
    • Design and develop data engineering services and their ecosystem using distributed databases (relational, columnar, graph, in-memory); orchestration (Apache Airflow); and distributed stream/batch data processing (Kafka, Kinesis, Spark).
    • Design and develop MLOps production pipelines; provide technical support to data scientists/ML engineers by getting their ML/DL models deployed at scale and meeting SLAs on both cloud and on-premises GPU and CPU instances.
    • Design data models for mission-critical, high-volume, near-real-time/batchdata; build idempotent/atomic production data pipelines to make data ingestion more fault tolerant.
    • Design and develop intuitive, highly automated, self-service data platform functions for business users.
    • Design, build, and operate scalable and reliable data pipelines on the Databricks platform.
  • Explore, evaluate and champion the introduction of next-generation technologies in the data-ingestion workflow. Participate in project planning and provide technical guidance on cloud architecture for data projects.
Requirements
  • BS in Computer Science or other related discipline is required. Advanced degrees in Computer Science (PhD, MS) are highly desirable.
  • 5+ years of relevant industry experience in some or most of the following technical areas:
    • Advanced programming skills in Python. Conversant with data structures and algorithm design.
    • Experience in building data pipelines (including data collection, warehousing, processing, analysis, monitoring, and governance) using open-source data ingestion platforms.
    • Intermediate-level knowledge and experience with AWS cloud components and best practices. Good understanding in deploying data stores such as S3, RedShift, Elasticache, PostgreSQL, and EMR.
    • Hands on experience with Databricks workspace, cluster management, AI Agent capabilities, and job orchestration
    • Prior experience in modern software development is required (such as web frontend UI, backend API microservices, understanding of CI/CD and Scrum/Kanbanagile development). Strong grasp on object-oriented or functional programming (using e.g. Python, Java, Scala, or C#).
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Information Technology - Lead Data Engineer
Information Technology - Lead Data Engineer

Singapore Airlines • Singapore

On-site
SGD 80,000 - 120,000
Data Engineer (Databricks & DevOps)
Data Engineer (Databricks & DevOps)

COGNIZANT TECHNOLOGY SOLUTIONS ASIA PACIFIC PTE. LTD. • Singapore

On-site
SGD 90,000 - 150,000
Databricks Workflows
CloudWatch Monitoring
Data Engineer
Data Engineer

EPS COMPUTER SYSTEMS PTE LTD • Singapore

On-site
SGD 90,000 - 150,000
Senior Data Engineer
Senior Data Engineer

optimum solutions (singapore) pte ltd • Singapore

On-site
SGD 120,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Epergne Solutions • Singapore

On-site
SGD 120,000 - 180,000
Data Engineer
Data Engineer

Epergne Solutions • Singapore

On-site
SGD 120,000 - 180,000
Data Engineer (Databricks)
Data Engineer (Databricks)

BASIL TECHNOLOGIES PTE. LTD. • Singapore

On-site
SGD 90,000 - 130,000
Data Engineer, AI Integration Specialist
Data Engineer, AI Integration Specialist

PERSOL • Singapore

On-site
SGD 90,000 - 130,000
Databricks Data Engineer
Databricks Data Engineer

Unison Group • Singapore

On-site
SGD 120,000 - 180,000
Data Engineer (Databricks)
Data Engineer (Databricks)

NCS Pte Ltd • Singapore

On-site
SGD 90,000 - 150,000