Data Engineer - up to SGD15k

MORGAN MCKINLEY PTE. LTD.; Singapore; On-site; SGD 110,000 - 160,000; Full time

Job summary

Morgan McKinley Pte Ltd is seeking a Senior Data Engineer to architect, develop, and refine high-throughput data systems powering analytics, AI, and RAG solutions.

You will build scalable ingestion/processing across hybrid cloud, optimise batch/streaming workflows with Databricks and Kafka, and deliver secure data for reporting, ML, and GenAI use cases. Strong Python/SQL and Spark experience required.

Qualifications

• Bachelor's degree in CS, engineering, or related field.
• 5+ years in data engineering or cloud-scale analytics with production pipelines.
• Experience with Python and SQL for data transformation and validation.
• Hands-on with Spark (PySpark) and large-scale processing.
• Experience with knowledge bases and RAG for AI use cases.
• Strong problem-solving and independent work ethic.

Responsibilities

• Build scalable ingestion, transformation, and serving across hybrid cloud environments.
• Optimise batch and streaming workflows with Databricks and Kafka.
• Deliver secure data for reporting, ML, and GenAI use cases.
• Lead setup of knowledge bases and RAG stacks for AI apps.
• Ensure pipeline reliability, observability, and production readiness.
• Drive CI/CD, version control, and data governance.

Skills

Python

SQL

Apache Spark

Data pipelines

Databricks

Kafka

RAG

CI/CD

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks

Kafka

Delta Lake

Microsoft Fabric

Job description

We are seeking a Senior Data Engineer . In this position, you will architect, develop, and fine-tune high-throughput data systems that power scalable analytics, network telemetry processing, and advanced AI/RAG solutions. You'll work closely with data scientists, analysts, and system owners to translate complex business requirements into secure, automated, and resilient data pipelines.

Key Responsibilities

• Build scalable ingestion, transformation, and serving capabilities across a hybrid cloud environment, ensuring reliability, security, and compliance with enterprise standards.
• Optimise batch and streaming workflows using Databricks and Kafka, applying best practices for performance, resilience, and maintainability.
• Deliver high-quality, secure data to support reporting, machine learning, and GenAI use cases.
• Lead the setup and operationalisation of knowledge bases and retrieval-augmented generation stacks for agentic AI applications.
• Perform cleansing and transformation using PySpark or SQL based on business requirements.
• Monitor, troubleshoot, and ensure pipeline reliability, while owning production readiness, observability, incident triage, and release coordination.
• Provide guidance on reusable components, code quality, deployment readiness, and production support practices.
• Connect diverse data sources (files, APIs, databases, streams), maintain metadata, and ensure transparency through documentation.
• Work with Microsoft Fabric, Databricks, Delta Lake, and other platform components.
• Drive CI/CD workflows, version control, and enforce data governance, security, and compliance policies.

Requirements

• Bachelor's degree in Computer Science, Engineering, or related field.
• At least 5 years of experience in data engineering or cloud-scale analytics, with proven ownership of production pipelines.
• Strong ability to design, build, optimise, and operate production-grade batch/streaming pipelines, including orchestration, error handling, and performance tuning.
• Hands-on expertise in Python and SQL for data transformation and validation.
• Familiarity with Apache Spark (especially PySpark) and large-scale data processing.
• Experience implementing knowledge bases and RAG solutions for AI use cases.
• Self-starter with problem-solving skills, attention to detail, and ability to work independently.
• Skilled at leading technical discussions with engineers, architects, product owners, and stakeholders.
• Solid understanding of enterprise data architecture, cloud security, CI/CD, release management, and production operations for data/AI platforms.

By sending us your personal data and curriculum vitae (CV), you are deemed to consent to Morgan Mckinley Pte Ltd and its affiliates to collect, use and disclose your personal data for the purposes set out in the Privacy Policy available at https://www.morganmckinley.com/sg/privacy-policy. You acknowledge that you have read, understood, and agree with the Privacy Policy.