Lead Databricks Engineer - 12 months contract

PERSOL Outsourcing; Singapore; On-site; SGD 210,000 - 270,000; Full time

Job summary

PERSOL Outsourcing is seeking a highly experienced Databricks Architect for a 1-year engagement in Central Singapore. You will lead technical delivery, set engineering standards, and mentor data engineers across multiple Databricks initiatives on AWS.

You will define Lakehouse architectures, build production pipelines with PySpark, Python, SQL and Delta Lake, and enable AI/ML solutions while ensuring security, governance and operational readiness.

Qualifications

• Bachelor's degree in CS/IT/SE or related field.
• 10+ years of experience in data engineering and data-platform design.
• 5+ years Databricks experience including production deployment and optimization.
• 2+ years as Databricks architect or equivalent architecture responsibilities.
• 2+ years delivering AI/ML or generative AI workloads on Databricks in production.
• 5+ years AWS experience supporting data platforms.
• Databricks certification required; Databricks Certified Data Engineer Professional preferred.
• Strong Apache Spark, Python/PySpark, SQL, Delta Lake, and lakehouse architecture knowledge.
• Experience with Databricks orchestration, Unity Catalog, RBAC, metadata, governance, performance tuning, production ops.
• Experience with AI workloads on Databricks: MLflow, Model Serving, vector search, RAG.

Responsibilities

• Lead the technical delivery of multiple Databricks initiatives, setting engineering standards, reviewing designs and code, and mentoring data engineers.
• Act as the Databricks architect, translating requirements into secure, scalable architectures on AWS.
• Define Lakehouse architecture, workspace strategy, data integration patterns, governance and security controls, compute strategy, and operations readiness.
• Design and build production data pipelines using Python, PySpark, SQL and Delta Lake across Bronze-Silver-Gold layers.
• Lead hands-on development of data models, reusable components and data-quality controls.
• Design and enable production AI/ML and generative AI solutions on Databricks, including feature and data pipelines, model integration, evaluation, deployment and monitoring.
• Integrate Databricks with AWS services and enterprise systems; work with IAM, networking, storage and deployment architecture.
• Optimize Spark workloads for performance and cost; resolve platform/pipeline issues.
• Establish Git-based CI/CD, automated testing, observability and release practices; maintain docs.
• Collaborate with stakeholders to estimate effort, manage dependencies and communicate design decisions.

Skills

Leadership & mentoring

Stakeholder communication

Problem solving

Project management

Education

Bachelor's degree in CS/IT/SE or related

Tools

Databricks

AWS

Python

PySpark

SQL

Delta Lake

Spark

Git-based CI/CD

Terraform

MLflow

Model Serving

Vector search

Job description

Duration: 1 year

Working Hours: 8.30am - 6pm, Monday - Fridays. Standby and shift work may be required

Location: Central

Key Responsibilities

• Lead the technical delivery of multiple Databricks initiatives, setting engineering standards, reviewing designs and code, and mentoring data engineers.

• Act as the Databricks architect, translating business and technical requirements into secure, scalable and maintainable architectures on AWS.

• Define Databricks Lakehouse architecture, workspace and environment strategy, data integration patterns, governance and security controls, compute strategy, and operational readiness.

• Design and build production data pipelines using Python, PySpark, SQL and Delta Lake, covering batch, streaming and near-real-time processing using Medallion architecture across Bronze, Silver and Gold layers.

• Lead hands‑on development of data models, reusable engineering components and data-quality controls, including reconciliation, schema handling, recovery and reprocessing.

• Design and enable production AI/ML and generative AI solutions on Databricks, including data preparation, feature and data pipelines, model integration, evaluation, deployment and monitoring, working with AI/ML specialists where appropriate.

• Integrate Databricks with AWS services and enterprise systems, and work with platform and security teams on IAM, networking, storage and deployment architecture.

• Optimise Spark workloads, compute and storage for performance, reliability and cost; resolve complex platform, pipeline and production issues.

• Establish Git-based CI/CD, automated testing, observability and release practices for Databricks workloads, and maintain architecture and operational documentation.

• Work with business stakeholders, architects and delivery teams to estimate effort, manage technical dependencies and communicate design decisions across initiatives.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Software Engineering or a related discipline.

• 10+ years of experience in data engineering, including enterprise data-platform design and delivery.

• 5+ years of substantial hands‑on Databricks experience, including production implementation, optimization and troubleshooting.

• 2+ years serving as a Databricks architect or performing equivalent architecture responsibilities, including solution design and technical decision-making.

• 2+ year Hands‑on experience delivering AI/ML or generative AI workloads on Databricks, with production implementation experience strongly preferred.

• 5+ years of hands‑on AWS experience supporting data platforms and enterprise solutions.

• Current Databricks certification is required, with Databricks Certified Data Engineer Professional strongly preferred. Additional Databricks certifications in machine learning or generative AI are advantageous.

• Deep practical expertise in Apache Spark, Python/PySpark, SQL, Delta Lake, data modelling and lakehouse architecture, supported by strong data-engineering fundamentals.

• Strong experience with Databricks orchestration, Unity Catalog, role‑based access controls, metadata and lineage, data governance, performance tuning and production operations.

• Practical experience supporting AI workloads on Databricks, including capabilities such as MLflow, Model Serv­ing, model evaluation, vector search and retrieval‑augmented generation, as relevant to prior projects.

• Solid AWS expertise across S3, IAM, networking, security, compute and data integration, with the ability to select and integrate appropriate AWS services.

• Experience with Git‑based CI/CD, automated testing and deployment. Hands‑on experience with Databricks Asset Bundles or equivalent Databricks deployment automation is strongly preferred; Terraform experience is advantageous.

• AWS certifications are preferred, particularly in architecture or data engineering.

• Able to lead multiple technical workstreams while remaining hands‑on. Strong problem‑solving, stakeholder communication and mentoring skills, with clear accountability for architecture quality, engineering standards and delivery outcomes.