Senior Databricks Architect on AWS: Lakehouse Mastery

Ethereum Technologies; Atlanta (GA); On-site; USD 150,000 - 230,000; Full time

Job summary

Ethereum Technologies is seeking a highly experienced Databricks Architect to design, develop, and optimize enterprise Databricks solutions on AWS. The role demands hands-on mastery of Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and Databricks Workflows to build secure, scalable data platforms.

The candidate will lead end-to-end architecture, develop scalable ETL/ELT pipelines with PySpark, and optimize Spark performance while integrating with S3, IAM, and Glue.

Qualifications

• Must have extensive hands-on Databricks development and architecture experience.
• Must have strong Databricks Lakehouse, Delta Lake, and PySpark expertise.
• Experience architecting and implementing enterprise-scale Databricks solutions on AWS.

Responsibilities

• Design and implement end-to-end enterprise Databricks Lakehouse architecture on AWS.
• Build and optimize scalable data pipelines using Databricks and Apache Spark.
• Develop complex ETL/ELT workflows using PySpark and Spark SQL.
• Implement and manage Delta Lake architecture, including Delta Tables and Delta Live Tables.
• Design Unity Catalog for centralized data governance and metadata management.
• Optimize Spark jobs, cluster configurations, partitioning, caching, and query performance.
• Implement Databricks Workflows for job orchestration and automation.
• Develop scalable batch and streaming data processing solutions.
• Implement Medallion Architecture (Bronze, Silver, Gold).
• Integrate Databricks with AWS services (S3, IAM, Glue).
• Establish best practices for data quality, reliability, monitoring, and performance.

Skills

Databricks on AWS

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

Databricks Workflows

Delta Live Tables

Lakeflow Declarative Pipelines

Medallion Architecture

ETL/ELT pipelines

Streaming data processing

Python

SQL

AWS integration (S3/IAM/Glue)

Job description

Ethereum Technologies is seeking a highly experienced Databricks Architect to design, develop, and optimize enterprise Databricks solutions on AWS. The role demands hands-on mastery of Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and Databricks Workflows to build secure, scalable data platforms.

The candidate will lead end-to-end architecture, develop scalable ETL/ELT pipelines with PySpark, and optimize Spark performance while integrating with S3, IAM, and Glue.