Lead Cloud Data Platform Engineer — Databricks on AWS

The Judge Group; Irvine (CA); On-site; USD 130,000 - 140,000; Full time

Job summary

The Judge Group seeks an experienced AWS Databricks Engineering Lead to design, build, and oversee enterprise-scale cloud-native data platforms supporting Asset Management. You will lead platform engineering, cloud infrastructure, DevOps automation, platform reliability, security, and operational excellence across the Enterprise Data Office.

Responsibilities include defining standards, architecting Databricks ecosystems, driving AWS initiatives, and implementing IaC and CI/CD pipelines.

Qualifications

• Strong hands-on experience with Databricks, dbt, Airflow, AWS, Terraform and Python.
• Experience leading cloud platform engineering and DevOps initiatives.
• Expertise in CI/CD, Infrastructure as Code, Docker, and Kubernetes.
• Experience supporting Asset Management or Investment Management organizations.
• Strong knowledge of platform reliability, security, governance, and operational best practices.

Responsibilities

• Lead design, build, and management of enterprise data platforms.
• Define platform engineering standards, best practices, and automation frameworks.
• Architect and administer Databricks platforms including Delta Lake and Unity Catalog.
• Drive AWS cloud engineering initiatives using S3, ECS, EKS, Lambda, Glue, and CloudWatch.
• Develop Infrastructure-as-Code solutions using Terraform.
• Build and manage CI/CD pipelines with GitHub Actions, GitLab CI/CD, Jenkins and Azure DevOps.
• Support containerized environments leveraging Docker and Kubernetes.
• Establish governance, security, monitoring, and operational controls across environments.
• Implement SRE practices with SLAs, SLOs, observability, and incident management.
• Drive platform performance, scalability, cost optimization, and continuous improvement.

Skills

Databricks

dbt

Airflow

AWS

Terraform

Python

CI/CD

IaC

Docker

Kubernetes

Asset Management

Security

Governance

Reliability

Tools

GitHub Actions

GitLab CI/CD

Jenkins

Azure DevOps

Terraform

Job description

The Judge Group seeks an experienced AWS Databricks Engineering Lead to design, build, and oversee enterprise-scale cloud-native data platforms supporting Asset Management. You will lead platform engineering, cloud infrastructure, DevOps automation, platform reliability, security, and operational excellence across the Enterprise Data Office.

Responsibilities include defining standards, architecting Databricks ecosystems, driving AWS initiatives, and implementing IaC and CI/CD pipelines.