Senior Databricks Lakehouse Architect on AWS

Cloud Analytics Technologies, LLC; Atlanta (GA); On-site; USD 150,000 - 210,000; Full time

Job summary

Cloud Analytics Technologies, LLC is seeking a highly experienced Databricks Architect to design and implement enterprise Databricks Lakehouse solutions on AWS. The role demands hands-on expertise with Delta Lake, Unity Catalog, and Spark technologies, delivering scalable data platforms and governance across the organization.

The ideal candidate will lead architecture, optimize pipelines, and collaborate with data engineering, cloud, and enterprise teams to drive high-performance data solutions

Qualifications

• Must have extensive hands-on Databricks development and architecture experience.
• Strong Databricks Lakehouse, Delta Lake, and PySpark expertise.
• Experience architecting enterprise-scale Databricks solutions on AWS.
• Candidates with only general AWS security/governance experience without Databricks hands-on skills will not be considered.

Responsibilities

• Design end-to-end enterprise Databricks Lakehouse architecture on AWS.
• Build and optimize scalable data pipelines using Databricks and Apache Spark.
• Develop complex ETL/ELT workflows with PySpark and Spark SQL.
• Implement and manage Delta Lake architecture and optimization.
• Design Unity Catalog for centralized data governance and access control.
• Optimize Spark jobs, cluster config, partitioning, caching, and queries.
• Implement Databricks Workflows for orchestration and automation.
• Develop scalable batch and streaming data processing solutions.
• Implement Medallion Architecture (Bronze/Silver/Gold).
• Integrate Databricks with AWS services (S3, IAM, Glue).
• Establish data quality, monitoring, and performance best practices.
• Lead Databricks platform architecture and technical design.

Skills

Databricks architecture

Lakehouse architecture

Spark optimization

ETL/ELT design

Data governance

Data modeling

Data security

Python

SQL

Scala/Java

Tools

Databricks on AWS

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

Delta Live Tables

Lakeflow Declarative Pipelines

S3

IAM

Glue

dbt

Airflow

Git/SVN

JIRA/Confluence

Lambda/Step Functions

Redshift

Snowflake

Job description

Cloud Analytics Technologies, LLC is seeking a highly experienced Databricks Architect to design and implement enterprise Databricks Lakehouse solutions on AWS. The role demands hands-on expertise with Delta Lake, Unity Catalog, and Spark technologies, delivering scalable data platforms and governance across the organization.

The ideal candidate will lead architecture, optimize pipelines, and collaborate with data engineering, cloud, and enterprise teams to drive high-performance data solutions