Lead Databricks Engineer: Scalable Lakehouse/Data Pipelines

Archer; Wilmington (DE); On-site; USD 180,000 - 240,000; Full time

Job summary

Archer, in collaboration with JPMorgan Chase, seeks a Lead Software Engineer-Databricks to advance data platform capabilities. You will shape high-throughput pipelines, design lakehouse architectures, and drive secure, scalable data solutions across multiple business units.

You will own Databricks cluster strategy, implement CI/CD for data projects, and promote responsible AI tooling while maintaining strong data governance and performance tuning capabilities.

Qualifications

• 5+ years of software/data engineering experience.
• Production delivery with Apache Spark on Databricks and/or AWS EMR.
• Advanced Databricks expertise across Delta Lake, Unity Catalog, Workflows.
• Strong Python/Java for data processing and tooling.
• Experience leading AI-assisted development tools and setting validation standards.

Responsibilities

• Lead architecture and delivery of high-throughput, low-latency data pipelines on Databricks.
• Evolve Lakehouse patterns with Delta Lake for scalable data platforms.
• Own Databricks cluster strategy, configs, autoscaling, and policies.
• Orchestrate pipelines with Databricks Workflows and AWS integrations.
• Design secure ingestion/transformation frameworks and data models.
• Enforce data quality, lineage, and governance via Unity Catalog/AWS Glue Catalog.
• Drive Spark/Databricks performance tuning for cost and throughput.
• Build reusable libraries/APIs in Python/Java with solid tests.
• Implement CI/CD for data projects with Git, Terraform, and automated releases.
• Promote AI-assisted engineering practices with standardized validation.

Skills

Databricks

Apache Spark

Python

Java

SQL

CI/CD

Terraform

Airflow

Security

Leadership

Education

Bachelor's degree in Computer Science

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Repos/notebooks

SQL Warehouses

Job description

Archer, in collaboration with JPMorgan Chase, seeks a Lead Software Engineer-Databricks to advance data platform capabilities. You will shape high-throughput pipelines, design lakehouse architectures, and drive secure, scalable data solutions across multiple business units.

You will own Databricks cluster strategy, implement CI/CD for data projects, and promote responsible AI tooling while maintaining strong data governance and performance tuning capabilities.