Senior AI Data Engineer — Build Scalable GenAI Pipelines

Singtel; Singapore; On-site; Confidential; Full time

Job summary

Singtel in Singapore invites you to join a data platform engineering role focused on building scalable data ingestion and processing capabilities across a modern cloud data platform. You will design, implement, and operate batch and streaming pipelines using Databricks, Kafka, PySpark, and SQL, while ensuring performance, reliability, and security.

Collaborate with engineers, architects, product owners, and data partners to enable analytics, ML use cases, and governance across enterprise data

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 5–8 years of experience in data engineering, data platform engineering, or cloud-scale analytics solution delivery.
• Proven ability to independently design, build, optimise, and operate production‑grade data pipelines.
• Hands‑on experience with Python and SQL for data transformation and validation.
• Familiarity with Apache Spark (PySpark) and large-scale data processing concepts.
• Experience with knowledge base and RAG solutions for agentic AI use cases.
• Strong documentation and communication skills.
• Strong understanding of enterprise data architecture, cloud security, access control, CI/CD, release management, and production operations for data and AI platform solutions.

Responsibilities

• Design, build, optimise, and maintain batch and streaming data ingestion pipelines using platforms such as Databricks and Kafka, ensuring scalability, reliability, observability, and alignment with enterprise data architecture standards.
• Perform data transformation and cleansing using PySpark or SQL based on business and technical requirements.
• Monitor and troubleshoot data workflows to ensure data quality and pipeline reliability.
• Provide technical guidance to engineers and delivery partners on data platform patterns, reusable components, code quality, deployment readiness, and production support practices.
• Lead integration of data from diverse source systems including files, APIs, databases, and streaming platforms, working with source-system owners and consuming teams to define fit-for-purpose ingestion patterns and delivery timelines.
• Help maintain metadata and pipeline documentation for transparency and traceability.
• Own production readiness for assigned data and AI platform components, including observability, incident triage, root-cause analysis, release coordination, and continuous improvement of operational runbooks.
• Participate in integrating pipelines with tools such as Microsoft Fabric, Databricks, Delta Lake, and other platform components.
• Build and maintain knowledge base and RAG solution on variety of hosting platforms.
• Implement and operate knowledge base storage, lifecycle management and embedding/vectorization.
• Contribute to automation efforts using version control and CI/CD workflows.
• Apply data governance, security, access control, and operational risk policies during solution design and implementation, ensuring pipelines and knowledge platforms meet enterprise compliance requirements.

Skills

Python

SQL

PySpark

Apache Spark

Databricks

Kafka

Data pipelines

CI/CD

Data governance

Security & access control

Cloud platforms

Observability

Education

Bachelor’s degree in Computer Science, Engineering, or a related field

Tools

Databricks

Kafka

Delta Lake

Microsoft Fabric

Job description

Singtel in Singapore invites you to join a data platform engineering role focused on building scalable data ingestion and processing capabilities across a modern cloud data platform. You will design, implement, and operate batch and streaming pipelines using Databricks, Kafka, PySpark, and SQL, while ensuring performance, reliability, and security.

Collaborate with engineers, architects, product owners, and data partners to enable analytics, ML use cases, and governance across enterprise data