AI Data Engineering

re-zoo-me; Singapore; Hybrid; SGD 120,000 - 160,000; Full time

Job summary

unknown is seeking an experienced Data Engineer to own end-to-end data solutions in a hybrid-cloud environment. The role focuses on building scalable data pipelines using Databricks, Apache Spark and Kafka, and laying foundations for analytics, ML and AI capabilities.

The ideal candidate has 5–8 years in data engineering with strong Python and SQL skills, experience with GenAI data solutions and familiarity with Databricks, Delta Lake and Microsoft Fabric.

Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, Information Technology or related technical discipline.
• 5–8 years of professional experience spanning data engineering, data platforms, cloud data solutions or large-scale analytics engineering.
• Experience delivering robust data pipelines at scale with orchestration, fault handling and production operations.
• Strong programming and data manipulation capabilities in Python and SQL.
• Practical experience with Apache Spark / PySpark and distributed data processing at scale.
• Experience developing knowledge management, Retrieval-Augmented Generation (RAG) or retrieval-based data solutions for GenAI.
• Exposure to Databricks, Kafka, Delta Lake and Microsoft Fabric.
• Solid understanding of data platform architecture, cloud environments, security controls and CI/CD.

Responsibilities

• Own the end-to-end engineering of enterprise data solutions, from ingestion to data consumption in a hybrid-cloud environment.
• Engineer high-volume batch and real-time data pipelines using Databricks, Spark and Kafka.
• Establish data foundations for analytics, ML and AI, ensuring data is accessible and trusted.
• Drive engineering and productionisation of GenAI data capabilities including RAG architectures.
• Build data processing and transformation logic using Python, PySpark and SQL.

Job description

Responsibilities

• Own the end-to-end engineering of enterprise data solutions, from ingestion and processing through to data consumption, across a hybrid-cloud environment. Build solutions that are scalable, resilient, secure and aligned with the organisation’s technology architecture and governance framework.
• Engineer high-volume batch and real-time data pipelines using platforms such as Databricks, Apache Spark and Kafka, with emphasis on performance, reliability, monitoring and long-term maintainability.
• Establish and enhance data foundations for analytics, ML and AI, ensuring data is accessible, trusted and fit for downstream use cases.
• Drive the engineering and productionisation of GenAI data capabilities, including knowledge bases and Retrieval-Augmented Generation (RAG) architectures supporting enterprise AI and agentic applications.
• Build data processing and transformation logic using Python, PySpark and SQL, including data cleansing, validation and enrichment based on defined business and technical requirements.
• Design appropriate ingestion approaches for data originating from APIs, databases, files, event streams and other enterprise systems, collaborating with upstream and downstream teams to establish effective integration patterns.
• Engineer the underlying capabilities required for knowledge retrieval and AI applications, including knowledge storage, document/data lifecycle management, embedding generation, vectorisation and related components.
• Take ownership of data pipeline health and operational performance, proactively identifying data quality issues, failures, bottlenecks and opportunities for optimisation.
• Establish engineering standards and provide technical direction to engineers and implementation partners, covering architecture patterns, reusable frameworks, coding practices, deployment standards and production support.
• Ensure data and AI components are production-ready, with appropriate monitoring, alerting, incident response, troubleshooting, root-cause analysis, release processes and operational documentation.
• Work across the broader data ecosystem, integrating solutions with platforms including Microsoft Fabric, Databricks and Delta Lake, as well as other relevant enterprise technologies.
• Improve engineering efficiency through automation and modern software delivery practices, including source control, CI/CD and repeatable deployment processes.
• Maintain clear technical documentation, metadata and lineage to support governance, transparency, troubleshooting and ongoing platform management.
• Incorporate security, access management, data governance and technology risk controls throughout the development lifecycle, ensuring solutions comply with enterprise policies and regulatory requirements.

Requirements

• Bachelor’s degree in Computer Science, Computer Engineering, Information Technology or a related technical discipline.
• 5–8 years of professional experience spanning data engineering, data platforms, cloud data solutions or large-scale analytics engineering, with experience taking solutions into and supporting production environments.
• Demonstrated ability to independently deliver robust data pipelines at scale, covering areas such as orchestration, fault handling, monitoring, performance optimisation and production operations.
• Strong programming and data manipulation capabilities in Python and SQL.
• Practical experience with Apache Spark / PySpark and distributed data processing at scale.
• Experience developing knowledge management, RAG or retrieval-based data solutions for GenAI, LLM or agentic AI applications.
• Exposure to modern data engineering ecosystems, particularly Databricks, Kafka, Delta Lake and/or Microsoft Fabric.
• Good understanding of data platform architecture, cloud environments, security controls, identity and access management, CI/CD and production release practices.
• Strong analytical and troubleshooting capabilities, with a structured approach to resolving complex technical problems.
• Comfortable taking ownership of technical deliverables and driving discussions with architects, engineers, product teams, business stakeholders and upstream/downstream system owners.
• Strong written and verbal communication skills, with the ability to document technical solutions clearly and translate complex requirements into practical engineering outcomes.
• A strong focus on engineering quality, scalability, reliability and operational excellence, with the ability to work effectively in a fast-moving technology environment.