Databricks Lakehouse Architect on AWS - Lead Platform

Quantum Technologies. LLC; Atlanta (GA); On-site; USD 150,000 - 210,000; Full time

Job summary

Quantum Technologies. LLC is seeking a highly experienced Databricks Architect to design, develop, and optimize enterprise-scale Databricks solutions on AWS.

The role requires deep hands-on expertise with Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and Databricks Workflows. You will lead the end-to-end data platform design, build scalable pipelines, implement medallion architecture, and collaborate with data engineering and cloud teams to ensure performance, security, and

Qualifications

• Extensive hands-on Databricks development and architecture experience.
• Expertise in Databricks Lakehouse, Delta Lake, and PySpark.
• Experience architecting enterprise-scale Databricks solutions on AWS.
• Candidates with only general AWS Cloud Security, IAM, or governance experience without strong Databricks hands-on expertise will not be considered.

Responsibilities

• Design end-to-end enterprise Databricks Lakehouse architecture on AWS.
• Build and optimize scalable data pipelines using Databricks and Apache Spark.
• Develop complex ETL/ELT workflows using PySpark and Spark SQL.
• Implement and manage Delta Lake architecture, including Delta Tables and Delta Live Tables / Lakeflow Declarative Pipelines.
• Design and implement Unity Catalog for centralized data governance, access control, and metadata management.
• Optimize Spark jobs, cluster configurations, partitioning, caching, and query performance.
• Implement Databricks Workflows for job orchestration, scheduling, and pipeline automation.
• Develop scalable batch and streaming data processing solutions.
• Implement Medallion Architecture (Bronze, Silver, Gold).
• Integrate Databricks with AWS services such as S3, IAM, Glue, and cloud-native data services.
• Establish best practices for data quality, reliability, monitoring, and performance optimization.
• Lead Databricks platform architecture, technical design, and implementation.
• Collaborate with data engineering, cloud, and enterprise architecture teams.

Skills

Databricks on AWS

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

Databricks Workflows

Delta Live Tables

Lakehouse Architecture

ETL/ELT Pipelines

Python

SQL

S3 integration

IAM integration

Performance tuning

Tools

Spark

Delta Lake

Job description

Quantum Technologies. LLC is seeking a highly experienced Databricks Architect to design, develop, and optimize enterprise-scale Databricks solutions on AWS.

The role requires deep hands-on expertise with Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and Databricks Workflows. You will lead the end-to-end data platform design, build scalable pipelines, implement medallion architecture, and collaborate with data engineering and cloud teams to ensure performance, security, and