Senior Databricks Architect

Artifintech; Atlanta (GA); On-site; USD 150,000 - 230,000; Full time

Job summary

Artifintech is seeking a highly experienced Databricks Architect to design, build, and optimize enterprise-scale Databricks solutions on AWS. The role demands deep hands-on expertise across Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and Databricks Workflows.

You will lead end-to-end architecture, develop scalable data pipelines, and collaborate with data engineering and cloud teams to deliver high-performance, secure data platforms.

Qualifications

• Extensive hands-on Databricks development and architecture experience.
• Deep expertise in Databricks Lakehouse, Delta Lake, and PySpark.
• Experience designing enterprise-scale Databricks solutions on AWS.
• Strong knowledge of Spark performance tuning.
• Proven ability to lead architecture and design discussions.

Responsibilities

• Design and implement end-to-end enterprise Databricks Lakehouse architecture on AWS.
• Build and optimize scalable data pipelines using Databricks and Apache Spark.
• Develop complex ETL/ELT workflows using PySpark and Spark SQL.
• Implement and manage Delta Lake architecture, including Delta Tables and Delta Live Tables.
• Design and implement Unity Catalog for centralized data governance, access control, and metadata management.
• Optimize Spark jobs, cluster configurations, partitioning, caching, and query performance.
• Implement Databricks Workflows for job orchestration and automation.
• Develop scalable batch and streaming data processing solutions.
• Establish Medallion Architecture across Bronze, Silver, and Gold layers.
• Integrate Databricks with AWS services such as S3, IAM, Glue.
• Establish best practices for data quality and monitoring.
• Lead Databricks platform architecture and implementation.
• Collaborate with data engineering and enterprise architecture teams.

Skills

Databricks expertise

Python

SQL

ETL/ELT design

Data governance

Tools

Databricks on AWS

Apache Spark

PySpark

Delta Lake

Unity Catalog

Delta Live Tables

Lakeflow

Medallion Architecture

AWS S3/IAM/Glue

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Work Authorization:US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description:

We are seeking a highly experiencedDatabricks Architectwith deep, hands-on expertise in designing, developing, implementing, and optimizing enterprise-scale Databricks solutions on AWS.

The ideal candidate must have strong expertise in the Databricks ecosystem, including Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and Databricks Workflows. This role requires a hardcore Databricks professional with extensive hands-on experience building scalable, secure, and high-performance data platforms.

Key Responsibilities:

Design and implement end-to-end enterprise Databricks Lakehouse architecture on AWS.

Build and optimize scalable data pipelines using Databricks and Apache Spark.

Develop complex ETL/ELT workflows using PySpark and Spark SQL.

Implement and manage Delta Lake architecture, including Delta Tables, Delta Live Tables / Lakeflow Declarative Pipelines, and Delta Lake optimization.

Design and implement Unity Catalog for centralized data governance, access control, and metadata management.

Optimize Spark jobs, cluster configurations, partitioning, caching, and query performance.

Implement Databricks Workflows for job orchestration, scheduling, and pipeline automation.

Develop scalable batch and streaming data processing solutions.

Implement Medallion Architecture (Bronze, Silver, and Gold layers).

Integrate Databricks with AWS services such as S3, IAM, Glue, and cloud-native data services.

Establish best practices for data quality, reliability, monitoring, and performance optimization.

Lead Databricks platform architecture, technical design, and implementation.

Collaborate with data engineering, cloud, and enterprise architecture teams.

Required Technical Skills:

Strong hands-on experience withDatabricks on AWS.

Advanced expertise inApache Spark, PySpark, and Spark SQL.

Deep understanding ofDelta Lake and Lakehouse Architecture.

Strong experience implementingUnity Catalog.

Expertise in Databricks Workflows and job orchestration.

Experience with Delta Live Tables / Lakeflow Declarative Pipelines.

Strong knowledge of Medallion Architecture.

Hands-on experience building complex ETL/ELT data pipelines.

Experience with batch and real-time/streaming data processing.

Strong understanding of Spark performance tuning and optimization.

Experience integrating Databricks with AWS S3, IAM, and Glue.

Proficiency in Python and SQL.

Mandatory Requirements:

Must have extensive hands-on Databricks development and architecture experience.

Must have strong Databricks Lakehouse, Delta Lake, and PySpark expertise.

Must have experience architecting and implementing enterprise-scale Databricks solutions on AWS.

Candidates with only general AWS Cloud Security, IAM, or governance experience without strong Databricks hands-on expertise will not be considered.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.