Senior Data Engineer (Native Databricks Preferred)

ATS Client; Cape Town; On-site; ZAR 900,000 - 1,500,000; Full time

Job summary

ATS Client is seeking a Senior Databricks Engineer/Architect to design and optimize scalable data platforms in native Databricks environments. You will lead SAP-to-Databricks migrations, implement governance, and collaborate with cloud teams.

Responsibilities include developing data models, ETL frameworks, and mentoring teams to adopt best practices in DataOps and security while staying current with Databricks advances in Azure, AWS, or GCP ecosystems.

Qualifications

• Databricks Platform Architect Certification is mandatory.

Responsibilities

• Design and architect scalable data platforms and pipelines within native Databricks environments.
• Lead and execute SAP to Databricks migration projects, ensuring seamless data integration and transformation.
• Collaborate with business stakeholders, data engineers, and analysts to define architecture and data strategy.
• Implement best practices for data governance, security, and performance optimization on the Databricks platform.
• Develop and maintain data models, ETL processes, and automation frameworks for high-performance analytics.
• Guide and mentor technical teams on Databricks architecture, implementation standards, and platform utilization.
• Work closely with cloud teams (Azure, AWS, or GCP) to ensure seamless integration with enterprise data ecosystems.
• Stay current with Databricks feature releases and advancements to continuously enhance platform capabilities.

Skills

Databricks Certification

Databricks environments

SAP to Databricks migration

PySpark

SQL

Delta Lake

Databricks Workflows

Data governance

Security best practices

Leadership/mentoring

Tools

Databricks

PySpark

SQL

Delta Lake

Databricks Workflows

AWS S3

Azure

GCP Storage

Job description

Job Description

We are seeking a highly skilled Senior Databricks Engineer / Architect with hands‑on experience in native Databricks environments and a proven track record in SAP to Databricks data migration projects. The ideal candidate will be responsible for designing, implementing, and optimizing scalable data architectures that support advanced analytics, data engineering, and business intelligence initiatives.

Key Responsibilities

• Design and architect scalable data platforms and pipelines within native Databricks environments.
• Lead and execute SAP to Databricks migration projects, ensuring seamless data integration and transformation.
• Collaborate with business stakeholders, data engineers, and analysts to define architecture and data strategy.
• Implement best practices for data governance, security, and performance optimization on the Databricks platform.
• Develop and maintain data models, ETL processes, and automation frameworks for high-performance analytics.
• Guide and mentor technical teams on Databricks architecture, implementation standards, and platform utilization.
• Work closely with cloud teams (Azure, AWS, or GCP) to ensure seamless integration with enterprise data ecosystems.
• Stay current with Databricks feature releases and advancements to continuously enhance platform capabilities.

Requirements

• Databricks Platform Architect Certification (mandatory).
• 3+ years of hands‑on experience working with native Databricks environments.
• Proven experience in migrating data from SAP systems to Databricks.
• Strong expertise in PySpark, SQL, Delta Lake, and Databricks Workflows.
• Solid understanding of data warehousing concepts, data modeling, and ELT/ETL design patterns.
• Experience integrating Databricks with cloud data platforms (AWS S3, GCP Storage, or Azure).
• Knowledge of data governance, lineage, and security best practices.
• Excellent problem‑solving, analytical, and communication skills.
• Experience in leading or architecting enterprise data solutions.