AWS Databricks Engineering Lead (1146996)

The Judge Group; Irvine (CA); On-site; USD 130,000 - 140,000; Full time

Job summary

The Judge Group seeks an experienced AWS Databricks Engineering Lead to design, build, and oversee enterprise-scale cloud-native data platforms supporting Asset Management. You will lead platform engineering, cloud infrastructure, DevOps automation, platform reliability, security, and operational excellence across the Enterprise Data Office.

Responsibilities include defining standards, architecting Databricks ecosystems, driving AWS initiatives, and implementing IaC and CI/CD pipelines.

Qualifications

• Strong hands-on experience with Databricks, dbt, Airflow, AWS, Terraform and Python.
• Experience leading cloud platform engineering and DevOps initiatives.
• Expertise in CI/CD, Infrastructure as Code, Docker, and Kubernetes.
• Experience supporting Asset Management or Investment Management organizations.
• Strong knowledge of platform reliability, security, governance, and operational best practices.

Responsibilities

• Lead design, build, and management of enterprise data platforms.
• Define platform engineering standards, best practices, and automation frameworks.
• Architect and administer Databricks platforms including Delta Lake and Unity Catalog.
• Drive AWS cloud engineering initiatives using S3, ECS, EKS, Lambda, Glue, and CloudWatch.
• Develop Infrastructure-as-Code solutions using Terraform.
• Build and manage CI/CD pipelines with GitHub Actions, GitLab CI/CD, Jenkins and Azure DevOps.
• Support containerized environments leveraging Docker and Kubernetes.
• Establish governance, security, monitoring, and operational controls across environments.
• Implement SRE practices with SLAs, SLOs, observability, and incident management.
• Drive platform performance, scalability, cost optimization, and continuous improvement.

Skills

Databricks

dbt

Airflow

AWS

Terraform

Python

CI/CD

IaC

Docker

Kubernetes

Asset Management

Security

Governance

Reliability

Tools

GitHub Actions

GitLab CI/CD

Jenkins

Azure DevOps

Terraform

Job description

Location

Irvine, CA

Salary

$130,000.00 USD Annually - $140,000.00 USD Annually

Job Summary

We are seeking an experienced AWS Databricks Engineering Lead to design, build, and oversee enterprise-scale cloud-native data platforms supporting Asset Management business functions. This role will lead platform engineering, cloud infrastructure, DevOps automation, platform reliability, security, and operational excellence across the Enterprise Data Office (EDO).

Key Responsibilities

• Lead the design, implementation, and management of enterprise data platforms.
• Define platform engineering standards, best practices, and automation frameworks.
• Architect and administer Databricks platforms, including Delta Lake, Unity Catalog, DLT, MLflow, and Structured Streaming.
• Drive AWS cloud engineering initiatives utilizing S3, ECS, EKS, Lambda, Glue, and CloudWatch.
• Develop Infrastructure-as-Code solutions using Terraform.
• Build and manage CI/CD pipelines using GitHub Actions, GitLab CI/CD, Jenkins, and Azure DevOps.
• Support containerized environments leveraging Docker and Kubernetes.
• Establish governance, security, monitoring, and operational controls across environments.
• Implement Site Reliability Engineering (SRE) practices, including SLAs, SLOs, observability, and incident management.
• Drive platform performance, scalability, cost optimization, and continuous improvement.

Required Qualifications

• Strong hands-on experience with Databricks, dbt, Airflow, AWS, Terraform, and Python.
• Experience leading cloud platform engineering and DevOps initiatives.
• Expertise in CI/CD, Infrastructure as Code, Docker, and Kubernetes.
• Experience supporting Asset Management or Investment Management organizations.
• Strong knowledge of platform reliability, security, governance, and operational best practices.

By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge Group, Inc. and its affiliates (collectively "Judge") to such phone number regarding job opportunities, your job application, and for other related purposes. Message & data rates apply and message frequency may vary. Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared with third parties for marketing/promotional purposes. Reply STOP to opt out of receiving telephone calls and text messages from Judge and HELP for help.