Senior Databricks Architect

Machine Learning Technologies, LLC; Atlanta (GA); On-site; USD 140,000 - 180,000; Full time

Job summary

Machine Learning Technologies, LLC in Atlanta seeks a highly experienced Databricks Architect to design, implement, and optimize enterprise Databricks solutions on AWS. You will lead Lakehouse architectures, Delta Lake, Unity Catalog, and Spark-based data platforms to deliver secure, scalable analytics.

You'll build end-to-end data pipelines, orchestration with Databricks Workflows, and ensure high performance for batch and streaming workloads, integrating S3/IAM/Glue, and advocating best

Qualifications

• Extensive hands-on Databricks development and architecture experience.
• Strong Databricks Lakehouse, Delta Lake, and PySpark expertise.
• Experience architecting enterprise-scale Databricks solutions on AWS.
• Candidates lacking Databricks hands-on expertise in AWS will not be considered.

Responsibilities

• Design end-to-end enterprise Databricks Lakehouse architecture on AWS.
• Build and optimize scalable data pipelines with Databricks and Spark.
• Develop ETL/ELT workflows using PySpark and Spark SQL.
• Implement Delta Lake and Unity Catalog for governance.
• Optimize Spark jobs and cluster configurations.
• Implement Databricks Workflows for orchestration.
• Develop batch and streaming data processing solutions.
• Lead Databricks platform architecture and design.

Skills

Databricks on AWS

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

Databricks Workflows

ETL/ELT pipelines

Batch processing

Streaming processing

Medallion Architecture

Python

SQL

Tools

AWS Glue

Amazon S3

IAM

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Work Authorization:US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description:

We are seeking a highly experiencedDatabricks Architectwith deep, hands-on expertise in designing, developing, implementing, and optimizing enterprise-scale Databricks solutions on AWS.

The ideal candidate must have strong expertise in the Databricks ecosystem, including Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and Databricks Workflows. This role requires a hardcore Databricks professional with extensive hands-on experience building scalable, secure, and high-performance data platforms.

Key Responsibilities:

Design and implement end-to-end enterprise Databricks Lakehouse architecture on AWS.

Build and optimize scalable data pipelines using Databricks and Apache Spark.

Develop complex ETL/ELT workflows using PySpark and Spark SQL.

Implement and manage Delta Lake architecture, including Delta Tables, Delta Live Tables / Lakeflow Declarative Pipelines, and Delta Lake optimization.

Design and implement Unity Catalog for centralized data governance, access control, and metadata management.

Optimize Spark jobs, cluster configurations, partitioning, caching, and query performance.

Implement Databricks Workflows for job orchestration, scheduling, and pipeline automation.

Develop scalable batch and streaming data processing solutions.

Implement Medallion Architecture (Bronze, Silver, and Gold layers).

Integrate Databricks with AWS services such as S3, IAM, Glue, and cloud-native data services.

Establish best practices for data quality, reliability, monitoring, and performance optimization.

Lead Databricks platform architecture, technical design, and implementation.

Collaborate with data engineering, cloud, and enterprise architecture teams.

Required Technical Skills:

Strong hands-on experience withDatabricks on AWS.

Advanced expertise inApache Spark, PySpark, and Spark SQL.

Deep understanding ofDelta Lake and Lakehouse Architecture.

Strong experience implementingUnity Catalog.

Expertise in Databricks Workflows and job orchestration.

Experience with Delta Live Tables / Lakeflow Declarative Pipelines.

Strong knowledge of Medallion Architecture.

Hands-on experience building complex ETL/ELT data pipelines.

Experience with batch and real-time/streaming data processing.

Strong understanding of Spark performance tuning and optimization.

Experience integrating Databricks with AWS S3, IAM, and Glue.

Proficiency in Python and SQL.

Mandatory Requirements:

Must have extensive hands-on Databricks development and architecture experience.

Must have strong Databricks Lakehouse, Delta Lake, and PySpark expertise.

Must have experience architecting and implementing enterprise-scale Databricks solutions on AWS.

Candidates with only general AWS Cloud Security, IAM, or governance experience without strong Databricks hands-on expertise will not be considered.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.