Senior AI Data Engineer — Scalable Data & GenAI Platform

Singapore Telecommunications Limited; Singapore; Hybrid; SGD 120,000 - 180,000; Full time

Job summary

Singapore Telecommunications Limited is seeking an experienced Data Platform Engineer to design, build, and operate scalable data ingestion pipelines in a modern hybrid cloud platform. You will transform data with PySpark and SQL, ensure reliability, and lead the integration of RAG and knowledge-base solutions across multiple domains.

You will own production readiness, implement observability, and collaborate with engineers, architects, and data owners to deliver secure, maintainable data

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5–8 years of experience in data engineering, data platform engineering, or cloud-scale analytics solution delivery, with ownership of production pipelines and platform components.
• Proven ability to independently design, build, optimise, and operate production‑grade batch or streaming data pipelines, including orchestration, observability, error handling, performance tuning, and operational support.
• Hands‑on experience with Python and SQL for data transformation and validation.
• Familiarity with Apache Spark (especially PySpark) and large-scale data processing concepts.
• Experience with implementing knowledge base and RAG solutions for agentic AI use cases.
• Self‑starter with strong problem‑solving skills and keen attention to detail.
• Able to work independently and lead technical discussions with engineers, architects, product owners, source‑system teams, and business stakeholders to translate requirements into secure and maintainable platform solutions.
• Strong documentation and communication skills.
• Strong understanding of enterprise data architecture, cloud security, access control, CI/CD, release management, and production operations for data and AI platform solutions.

Responsibilities

• Design, build, optimise, and operate production‑grade batch and streaming data pipelines using Databricks and Kafka.
• Perform data transformation and cleansing using PySpark or SQL based on requirements.
• Monitor and troubleshoot data workflows for data quality and reliability.
• Provide guidance on data platform patterns, reusable components, and production readiness practices.
• Lead integration of data from diverse source systems and define ingestion patterns and delivery timelines.
• Own production readiness, including observability, incident triage, root-cause analysis, and runbook improvements.
• Collaborate on integrating pipelines with tools like Microsoft Fabric, Databricks, and Delta Lake.
• Build and maintain knowledge bases and RAG solutions on hosting platforms.
• Implement knowledge base storage, lifecycle management, and embedding/vectorization.
• Contribute to automation with version control and CI/CD workflows.
• Apply governance, security, access control, and risk policies to ensure enterprise compliance.

Skills

Python

SQL

PySpark

Apache Spark

Databricks

Kafka

Data governance

CI/CD

Observability

Communication

Education

Bachelor's degree in Computer Science, Engineering, or a related field

Tools

Databricks

Kafka

Delta Lake

Microsoft Fabric

Job description

Singapore Telecommunications Limited is seeking an experienced Data Platform Engineer to design, build, and operate scalable data ingestion pipelines in a modern hybrid cloud platform. You will transform data with PySpark and SQL, ensure reliability, and lead the integration of RAG and knowledge-base solutions across multiple domains.

You will own production readiness, implement observability, and collaborate with engineers, architects, and data owners to deliver secure, maintainable data