Senior AI Data Engineer — Data Platform & Pipelines Lead

jobline resources pte. ltd.; Singapore; On-site; SGD 90,000 - 180,000; Full time

Job summary

jobline resources pte. ltd. in Singapore seeks an experienced Data Platform Engineer to design, build, and operate production-grade batch and streaming pipelines. You will transform data with Python/SQL, ensure reliability, and guide multiple teams on platform patterns and deployment readiness.

You will lead integration from diverse sources, maintain metadata, and boost observability and incident response. Experience with PySpark, Spark, Databricks, and knowledge-base/RAG solutions is essential.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 5–8 years of experience in data engineering, data platform engineering, or cloud-scale analytics solution delivery.
• Proven ability to independently design, build, optimise, and operate production‑grade batch or streaming data pipelines.
• Hands‑on experience with Python and SQL for data transformation and validation.
• Familiarity with Apache Spark (especially PySpark) and large‑scale data processing concepts.
• Experience with implementing knowledge base and RAG solutions for agentic AI use cases.
• Strong documentation and communication skills.
• Able to lead technical discussions with engineers, architects, and product owners to translate requirements into secure platform solutions.

Responsibilities

• Design, build, optimise, and maintain batch and streaming data pipelines.
• Transform and cleanse data using PySpark or SQL.
• Monitor and troubleshoot data workflows for data quality and reliability.
• Provide technical guidance on data platform patterns, reusable components, and deployment readiness.
• Lead integration of data from diverse source systems (files, APIs, databases, streaming).
• Maintain metadata and pipeline documentation for transparency and traceability.
• Own production readiness for data and AI platform components including observability and incident triage.
• Integrate pipelines with tools like Microsoft Fabric, Databricks, Delta Lake.
• Build and maintain knowledge base and RAG solutions across hosting platforms.
• Implement knowledge base storage, lifecycle management, and embedding/vectorization.
• Contribute to automation with version control and CI/CD workflows.
• Apply data governance, security, access control, and compliance.

Skills

Python

SQL

PySpark

Apache Spark

Databricks

Kafka

Data pipelines

CI/CD

Education

Bachelor’s degree in Computer Science, Engineering, or a related field

Tools

Databricks

Kafka

Delta Lake

CI/CD

Job description

jobline resources pte. ltd. in Singapore seeks an experienced Data Platform Engineer to design, build, and operate production-grade batch and streaming pipelines. You will transform data with Python/SQL, ensure reliability, and guide multiple teams on platform patterns and deployment readiness.

You will lead integration from diverse sources, maintain metadata, and boost observability and incident response. Experience with PySpark, Spark, Databricks, and knowledge-base/RAG solutions is essential.