Lead Data Engineer (1147926)

The Judge Group

Tustin (CA)

On-site

USD 180,000 - 190,000

Full time

10 hours ago

Be an early applicant

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

The Judge Group in Irvine, CA seeks an experienced Lead Data Engineer (EDO Platform Engineering Lead) to architect, build, and manage enterprise-scale cloud-native data platforms supporting Asset Management functions. You will lead platform engineering, infrastructure automation, DevOps enablement, platform reliability, security, and operational excellence across the Enterprise Data Office (EDO).

The ideal candidate will bring deep expertise in Databricks, dbt, Airflow, AWS Cloud, Terraform,

Qualifications

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
10+ years of experience in Data Engineering, Platform Engineering, Cloud Engineering, or related disciplines.
Strong experience designing and supporting enterprise data platforms.
Hands-on expertise with: Databricks, dbt, Apache Airflow, AWS Cloud Services, Terraform, Python, CI/CD pipelines, Docker & Kubernetes.
Experience implementing Infrastructure as Code (IaC) and DevOps practices.
Strong understanding of data governance, security, and compliance controls.
Experience supporting large-scale enterprise data environments.
Excellent leadership, communication, and stakeholder management skills.

Responsibilities

Lead the design, implementation, and management of enterprise-scale data platforms.
Define platform engineering strategy aligned with Enterprise Data Office (EDO) and enterprise architecture standards.
Establish reusable platform frameworks, accelerators, reference architectures, and engineering best practices.
Drive cloud-native modernization initiatives across data and analytics ecosystems.
Own platform lifecycle management, scalability, reliability, availability, and performance.
Lead Databricks platform architecture, administration, and governance.
Design and implement modern Lakehouse architectures utilizing Delta Lake.
Manage and optimize Databricks Workflows, Delta Live Tables (DLT), Unity Catalog, MLflow, Structured Streaming, Cluster Policies, Security & Access Controls.
Optimize platform performance, scalability, and cost efficiency.
Establish governance controls across development, testing, and production environments.
Lead cloud engineering initiatives utilizing AWS services including Amazon S3, ECS, EKS, AWS Lambda, AWS Glue, CloudWatch.
Design highly available and secure cloud-based data platform solutions.
Ensure platform resiliency, security, compliance, and operational excellence.
Develop and maintain reusable Terraform modules.
Automate environment provisioning and deployment processes.
Implement infrastructure standardization and governance.
Maintain infrastructure version control and change management.
Enforce compliance through Infrastructure-as-Code best practices.
Design and implement CI/CD pipelines for platform and data engineering teams.
Automate deployment, release management, and operational processes.
Enable GitOps and Infrastructure-as-Code methodologies.
Improve deployment speed, reliability, and operational efficiency.
Support self-service platform capabilities for engineering teams.
Establish and implement Site Reliability Engineering (SRE) practices.
Define and monitor platform SLAs, SLOs, and key operational metrics.
Lead platform monitoring, observability, and incident management strategies.
Drive root cause analysis, remediation, and preventive action programs.
Support API integrations and operational workflows.

Skills

Databricks

dbt

Apache Airflow

AWS Cloud

Terraform

Python

CI/CD pipelines

Docker & Kubernetes

Platform Engineering

DevOps

Infrastructure as Code

Asset Management Industry Experience

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field

Tools

Databricks

Delta Lake

Unity Catalog

MLflow

Kubernetes

Jenkins

GitHub Actions

GitLab CI/CD

Azure DevOps

Docker

Apache Airflow

dbt

Job description

Location: Tustin, CA

Salary: $180,000.00 USD Annually - $190,000.00 USD Annually

Lead Data Engineer (EDO Platform Engineering Lead)

Location: Irvine, CA

Salary: $180,000-$190,000

About The Role

We are seeking an experienced Lead Data Engineer / EDO Platform Engineering Lead to architect, build, and manage enterprise-scale cloud-native data platforms supporting Asset Management business functions. This role will lead platform engineering, infrastructure automation, DevOps enablement, platform reliability, security, and operational excellence across the Enterprise Data Office (EDO).

The ideal candidate will possess deep expertise in Databricks, dbt, Airflow, AWS Cloud, Terraform, Python, DevOps, Platform Engineering, and Data Operations, along with a strong background supporting Asset Management and Investment Management organizations.

This position will be responsible for establishing and operating a secure, scalable, highly available enterprise data platform supporting analytics, reporting, regulatory compliance, portfolio management, risk management, and investment operations.

Responsibilities

Platform Engineering & Architecture

Lead the design, implementation, and management of enterprise-scale data platforms.
Define platform engineering strategy aligned with Enterprise Data Office (EDO) and enterprise architecture standards.
Establish reusable platform frameworks, accelerators, reference architectures, and engineering best practices.
Drive cloud-native modernization initiatives across data and analytics ecosystems.
Own platform lifecycle management, scalability, reliability, availability, and performance.

Databricks Platform Leadership

Lead Databricks platform architecture, administration, and governance.
Design and implement modern Lakehouse architectures utilizing Delta Lake.
Manage and optimize:
- Databricks Workflows
- Delta Live Tables (DLT)
- Unity Catalog
- MLflow
- Structured Streaming
- Cluster Policies
- Security & Access Controls
Optimize platform performance, scalability, and cost efficiency.
Establish governance controls across development, testing, and production environments.

Cloud Engineering & Infrastructure

Lead cloud engineering initiatives utilizing AWS services including:
- Amazon S3
- ECS
- EKS
- AWS Lambda
- AWS Glue
- CloudWatch
Design highly available and secure cloud-based data platform solutions.
Ensure platform resiliency, security, compliance, and operational excellence.

Infrastructure as Code (IaC)

Develop and maintain reusable Terraform modules.
Automate environment provisioning and deployment processes.
Implement infrastructure standardization and governance.
Maintain infrastructure version control and change management.
Enforce compliance through Infrastructure-as-Code best practices.

DevOps & Automation

Design and implement CI/CD pipelines for platform and data engineering teams.
Automate deployment, release management, and operational processes.
Enable GitOps and Infrastructure-as-Code methodologies.
Improve deployment speed, reliability, and operational efficiency.
Support self-service platform capabilities for engineering teams.

Platform Reliability & Operations

Establish and implement Site Reliability Engineering (SRE) practices.
Define and monitor platform SLAs, SLOs, and key operational metrics.
Lead platform monitoring, observability, and incident management strategies.
Drive root cause analysis, remediation, and preventive action programs.
Support API integrations and operational workflows.

Tools & Technologies

Databricks
Delta Lake
dbt
Apache Airflow
AWS Cloud
Terraform
Python
GitHub Actions
GitLab CI/CD
Jenkins
Azure DevOps
Docker
Kubernetes

Required Qualifications

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
10+ years of experience in Data Engineering, Platform Engineering, Cloud Engineering, or related disciplines.
Strong experience designing and supporting enterprise data platforms.
Hands-on expertise with:
- Databricks
- dbt
- Apache Airflow
- AWS Cloud Services
- Terraform
- Python
- CI/CD pipelines
- Docker & Kubernetes
Experience implementing Infrastructure as Code (IaC) and DevOps practices.
Strong understanding of data governance, security, and compliance controls.
Experience supporting large-scale enterprise data environments.
Excellent leadership, communication, and stakeholder management skills.

Preferred Qualifications

Experience within Asset Management, Wealth Management, Investment Management, or Financial Services organizations.
Expertise in Lakehouse Architecture and Data Platform Modernization.
Experience implementing Site Reliability Engineering (SRE) practices.
AWS Certifications preferred.
Databricks Certifications preferred.
Experience supporting enterprise-scale analytics and regulatory reporting platforms.

Must-Have Skills