Lead Databricks Engineer - 12 months contract

PERSOL TECH SERVICES PTE. LTD.; Singapore; Sur place; SGD 180 000 - 240 000; Plein temps

Résumé du poste

PERSOL TECH SERVICES PTE. LTD. is seeking a seasoned Databricks Architect to lead the technical delivery of multiple Databricks initiatives on AWS, translate requirements into secure, scalable architectures, and mentor data engineers.

The role covers designing Lakehouse architecture, building production data pipelines with Python, PySpark, SQL and Delta Lake, enabling AI/ML workloads, and ensuring governance, IAM, and cost optimization across the platform.

Qualifications

• Bachelor's degree or equivalent in CS/IT/SE or related field.
• 10+ years of data engineering experience with enterprise data-platform design.
• 5+ years hands-on Databricks experience including production implementations.
• 2+ years as a Databricks architect or equivalent.
• 2+ years delivering AI/ML or generative AI workloads on Databricks.
• 5+ years AWS experience supporting data platforms.
• Databricks certification required; additional ML/GenAI certs advantageous.
• Deep expertise in Spark, PySpark, SQL, Delta Lake and lakehouse architecture.
• Experience with Databricks orchestration, Unity Catalog, RBAC, metadata, lineage and governance.
• Hands-on with CI/CD, testing and deployment; Terraform experience advantageous.

Responsabilités

• Lead the technical delivery of multiple Databricks initiatives and mentor data engineers.
• Act as the Databricks architect translating requirements into secure, scalable architectures on AWS.
• Define Databricks Lakehouse architecture, workspace strategy and security controls.
• Design and build production data pipelines using Python, PySpark, SQL and Delta Lake.
• Lead development of data models, reusable components and data-quality controls.
• Design and enable production AI/ML and generative AI solutions on Databricks.
• Integrate Databricks with AWS services and enterprise systems, coordinating IAM and networking.
• Optimize Spark workloads for performance, reliability and cost.
• Establish Git-based CI/CD, automated testing, observability and release practices.
• Collaborate with stakeholders to estimate effort and communicate design decisions across initiatives.

Connaissances

Leadership

Mentoring

Architectural design

Communication

Problem-solving

Formation

Bachelor's degree in Computer Science or related field

Outils

Databricks

AWS

Python

PySpark

SQL

Delta Lake

Unity Catalog

MLflow

Terraform

S3

IAM

Description du poste

Duration: 1 year

Working Hours: 8.30am - 6pm, Monday - Fridays. Standby and shift work may be required

Location: Central

Key Responsibilities

• Lead the technical delivery of multiple Databricks initiatives, setting engineering standards, reviewing designs and code, and mentoring data engineers.
• Act as the Databricks architect, translating business and technical requirements into secure, scalable and maintainable architectures on AWS.
• Define Databricks Lakehouse architecture, workspace and environment strategy, data integration patterns, governance and security controls, compute strategy, and operational readiness.
• Design and build production data pipelines using Python, PySpark, SQL and Delta Lake, covering batch, streaming and near-real-time processing using Medallion architecture across Bronze, Silver and Gold layers.
• Lead hands-on development of data models, reusable engineering components and data-quality controls, including reconciliation, schema handling, recovery and reprocessing.
• Design and enable production AI/ML and generative AI solutions on Databricks, including data preparation, feature and data pipelines, model integration, evaluation, deployment and monitoring, working with AI/ML specialists where appropriate.
• Integrate Databricks with AWS services and enterprise systems, and work with platform and security teams on IAM, networking, storage and deployment architecture.
• Optimise Spark workloads, compute and storage for performance, reliability and cost; resolve complex platform, pipeline and production issues.
• Establish Git-based CI/CD, automated testing, observability and release practices for Databricks workloads, and maintain architecture and operational documentation.
• Work with business stakeholders, architects and delivery teams to estimate effort, manage technical dependencies and communicate design decisions across initiatives.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Software Engineering or a related discipline.
• 10+ years of experience in data engineering, including enterprise data-platform design and delivery.
• 5+ years of substantial hands-on Databricks experience, including production implementation, optimization and troubleshooting.
• 2+ years serving as a Databricks architect or performing equivalent architecture responsibilities, including solution design and technical decision-making.
• 2+ year Hands-on experience delivering AI/ML or generative AI workloads on Databricks, with production implementation experience strongly preferred.
• 5+ years of hands-on AWS experience supporting data platforms and enterprise solutions.
• Current Databricks certification is required, with Databricks Certified Data Engineer Professional strongly preferred. Additional Databricks certifications in machine learning or generative AI are advantageous.
• Deep practical expertise in Apache Spark, Python/PySpark, SQL, Delta Lake, data modelling and lakehouse architecture, supported by strong data-engineering fundamentals.
• Strong experience with Databricks orchestration, Unity Catalog, role-based access controls, metadata and lineage, data governance, performance tuning and production operations.
• Practical experience supporting AI workloads on Databricks, including capabilities such as MLflow, Model Serving, model evaluation, vector search and retrieval-augmented generation, as relevant to prior projects.
• Solid AWS expertise across S3, IAM, networking, security, compute and data integration, with the ability to select and integrate appropriate AWS services.
• Experience with Git-based CI/CD, automated testing and deployment. Hands-on experience with Databricks Asset Bundles or equivalent Databricks deployment automation is strongly preferred; Terraform experience is advantageous.
• AWS certifications are preferred, particularly in architecture or data engineering.
• Able to lead multiple technical workstreams while remaining hands-on. Strong problem-solving, stakeholder communication and mentoring skills, with clear accountability for architecture quality, engineering standards and delivery outcomes.