Senior Data Engineer - GenAI & Streaming Data Platforms

morgan mckinley pte. ltd.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Morgan McKinley is seeking an experienced Data Engineer to design, build, and maintain scalable batch and streaming pipelines on a hybrid cloud platform. You will transform data with PySpark/SQL and ensure data quality across varied sources, while guiding integration and governance practices.

You will lead RAG-enabled solutions for enterprise AI use cases, collaborate with engineers and stakeholders, and drive production readiness with strong observability and CI/CD processes.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 5-8 years of experience in data engineering, data platform engineering, or cloud-scale analytics delivery, with proven ownership of production pipelines.
• Hands-on expertise in Python, SQL, and Apache Spark (PySpark) for large-scale data processing.
• Experience building knowledge base/RAG solutions for agentic AI use cases.
• Strong understanding of enterprise data architecture, cloud security, CI/CD, release management, and production operations.
• Self-starter with strong problem-solving skills, attention to detail, and the ability to lead technical discussions across engineering, architecture, and business stakeholder groups.
• Strong documentation and communication skills.

Responsibilities

• Design, build, and operate scalable batch and streaming data pipelines across a hybrid cloud data platform.
• Perform data transformation and cleansing (PySpark/SQL) to meet business and technical requirements.
• Monitor, troubleshoot, and ensure the quality and reliability of data workflows.
• Lead integration of data from diverse sources (files, APIs, databases, streaming platforms).
• Lead the implementation of knowledge base and Retrieval-Augmented Generation (RAG) solutions to support GenAI use cases.
• Provide technical guidance to engineers and delivery partners on platform patterns and production support practices.
• Own production readiness for assigned components—observability, incident triage, root-cause analysis, and runbook improvements.
• Maintain metadata/pipeline documentation and contribute to CI/CD automation efforts.
• Apply data governance, security, and access control policies throughout design and implementation.

Skills

Python

SQL

Apache Spark (PySpark)

Databricks

Kafka

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks

Kafka

PySpark

Job description

Morgan McKinley is seeking an experienced Data Engineer to design, build, and maintain scalable batch and streaming pipelines on a hybrid cloud platform. You will transform data with PySpark/SQL and ensure data quality across varied sources, while guiding integration and governance practices.

You will lead RAG-enabled solutions for enterprise AI use cases, collaborate with engineers and stakeholders, and drive production readiness with strong observability and CI/CD processes.