Senior AI Data Engineer

Singapore Telecommunications Limited; Singapore; Hybrid; SGD 120,000 - 180,000; Full time

Job summary

Singapore Telecommunications Limited is seeking an experienced Data Platform Engineer to design, build, and operate scalable data ingestion pipelines in a modern hybrid cloud platform. You will transform data with PySpark and SQL, ensure reliability, and lead the integration of RAG and knowledge-base solutions across multiple domains.

You will own production readiness, implement observability, and collaborate with engineers, architects, and data owners to deliver secure, maintainable data

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5–8 years of experience in data engineering, data platform engineering, or cloud-scale analytics solution delivery, with ownership of production pipelines and platform components.
• Proven ability to independently design, build, optimise, and operate production‑grade batch or streaming data pipelines, including orchestration, observability, error handling, performance tuning, and operational support.
• Hands‑on experience with Python and SQL for data transformation and validation.
• Familiarity with Apache Spark (especially PySpark) and large-scale data processing concepts.
• Experience with implementing knowledge base and RAG solutions for agentic AI use cases.
• Self‑starter with strong problem‑solving skills and keen attention to detail.
• Able to work independently and lead technical discussions with engineers, architects, product owners, source‑system teams, and business stakeholders to translate requirements into secure and maintainable platform solutions.
• Strong documentation and communication skills.
• Strong understanding of enterprise data architecture, cloud security, access control, CI/CD, release management, and production operations for data and AI platform solutions.

Responsibilities

• Design, build, optimise, and operate production‑grade batch and streaming data pipelines using Databricks and Kafka.
• Perform data transformation and cleansing using PySpark or SQL based on requirements.
• Monitor and troubleshoot data workflows for data quality and reliability.
• Provide guidance on data platform patterns, reusable components, and production readiness practices.
• Lead integration of data from diverse source systems and define ingestion patterns and delivery timelines.
• Own production readiness, including observability, incident triage, root-cause analysis, and runbook improvements.
• Collaborate on integrating pipelines with tools like Microsoft Fabric, Databricks, and Delta Lake.
• Build and maintain knowledge bases and RAG solutions on hosting platforms.
• Implement knowledge base storage, lifecycle management, and embedding/vectorization.
• Contribute to automation with version control and CI/CD workflows.
• Apply governance, security, access control, and risk policies to ensure enterprise compliance.

Skills

Python

SQL

PySpark

Apache Spark

Databricks

Kafka

Data governance

CI/CD

Observability

Communication

Education

Bachelor's degree in Computer Science, Engineering, or a related field

Tools

Databricks

Kafka

Delta Lake

Microsoft Fabric

Job description

Be a Part of Something BIG!

• Responsible for designing, building, and operating scalable data ingestion, transformation, and serving capabilities across a modern hybrid cloud data platform, ensuring solutions are reliable, secure, reusable, and aligned to enterprise architecture and governance standards.
• Develop and optimise batch and streaming pipelines using cloud tools such as Databricks and Kafka, applying sound engineering practices to ensure performance, resilience, and maintainability.
• Contribute to the delivery of reliable, secure, and high-quality data for analytics, reporting, and machine learning use cases
• Lead the implementation and operationalisation of knowledge base and retrieval-augmented generation solution stacks to support scalable GenAI and agentic use cases across business domains

Make an Impact by:

• Design, build, optimise, and maintain batch and streaming data ingestion pipelines using platforms such as Databricks and Kafka, ensuring scalability, reliability, observability, and alignment with enterprise data architecture standards.
• Perform data transformation and cleansing using PySpark or SQL based on business and technical requirements
• Monitor and troubleshoot data workflows to ensure data quality and pipeline reliability
• Provide technical guidance to engineers and delivery partners on data platform patterns, reusable components, code quality, deployment readiness, and production support practices.
• Lead integration of data from diverse source systems including files, APIs, databases, and streaming platforms, working with source-system owners and consuming teams to define fit-for-purpose ingestion patterns and delivery timelines.
• Help maintain metadata and pipeline documentation for transparency and traceability
• Own production readiness for assigned data and AI platform components, including observability, incident triage, root-cause analysis, release coordination, and continuous improvement of operational runbooks.
• Participate in integrating pipelines with tools such as Microsoft Fabric, Databricks, Delta Lake, and other platform components
• Build and maintain knowledge base and RAG solution on variety of hosting platforms
• Implement and operate knowledge base storage, lifecycle management and embedding/vectorization
• Contribute to automation efforts using version control and CI/CD workflows
• Apply data governance, security, access control, and operational risk policies during solution design and implementation, ensuring pipelines and knowledge platforms meet enterprise compliance requirements.

Skills for Success:

• Bachelor's degree in Computer Science, Engineering, or a related field
• 5-8 years of experience in data engineering, data platform engineering, or cloud-scale analytics solution delivery, with demonstrated ownership of production pipelines and platform components.
• Proven ability to independently design, build, optimise, and operate production‑grade batch or streaming data pipelines, including orchestration, observability, error handling, performance tuning, and operational support.
• Hands‑on experience with Python and SQL for data transformation and validation
• Familiarity with Apache Spark (especially PySpark) and large-scale data processing concepts
• Experience with implementing knowledge base and RAG solutions for agentic AI use cases
• Self‑starter with strong problem‑solving skills and a keen attention to detail
• Able to work independently and lead technical discussions with engineers, architects, product owners, source‑system teams, and business stakeholders to translate requirements into secure and maintainable platform solutions.
• Strong documentation and communication skills
• Strong understanding of enterprise data architecture, cloud security, access control, CI/CD, release management, and production operations for data and AI platform solutions.