Databricks Platform Tech Lead — Data Infrastructure

Cardlytics; United States; Hybrid; USD 150,000 - 210,000; Full time

Benefits offered by this job

Flexible paid time off

Medical, dental, and vision insurance

401(k) with company match

Employee Stock Purchase Plan

Educational assistance

Job summary

Cardlytics is seeking a Principal Software Engineer, Tech Lead for the Databricks Platform to own and evolve our Databricks environment on AWS. You will lead a small team, set platform standards, and partner with Analysts, ML Engineers, and business stakeholders to enable scalable data access.

You will drive platform administration, design, governance, and DevOps practices, delivering reliable data pipelines and security across the organization.

Qualifications

• 7+ years of experience in data engineering or data platform roles.
• 2+ years in a technical lead or architecture capacity.
• Deep Databricks expertise across Delta Lake, Spark, Unity Catalog, Workflows, and cluster management on AWS.
• Experience delivering Lakehouse architectures and large-scale ELT/ETL pipelines.

Responsibilities

• Platform Administration: manage Databricks on AWS, workspace config, access controls, and cost governance.
• Architecture & Design: evolve Databricks architecture, reference architectures for batch, streaming, and ML workloads.
• Governance & Security: drive Unity Catalog adoption, data lineage, encryption, and audit logging.
• Performance & Reliability: monitor health, define SLAs, tune Spark jobs, manage incidents.
• User Enablement: create self-service guides and onboarding materials for Analysts, ML Engineers, and business users.
• Infrastructure as Code: own CI/CD for Databricks deployments using Terraform, GitHub Actions, and Asset Bundles.
• Leadership: mentor a small data/platform engineering team and manage vendor relationships.

Skills

Databricks

Spark (PySpark)

Delta Lake

Unity Catalog

Workflows

Terraform

AWS

Leadership

Platform governance

Tools

Databricks

Terraform

AWS

Job description

Cardlytics is seeking a Principal Software Engineer, Tech Lead for the Databricks Platform to own and evolve our Databricks environment on AWS. You will lead a small team, set platform standards, and partner with Analysts, ML Engineers, and business stakeholders to enable scalable data access.

You will drive platform administration, design, governance, and DevOps practices, delivering reliable data pipelines and security across the organization.