Databricks Architect

Celebal Technologies; United States; Remote; USD 140,000 - 210,000; Full time

Benefits offered by this job

Career development

Cutting-edge Databricks tech exposure

Global collaboration

Job summary

Celebal Technologies is seeking an experienced Databricks Architect to lead the design and implementation of scalable data lakehouse solutions on AWS and/or Azure. You will define the technical architecture, establish data governance, and drive best practices across BI and analytics teams.

The role requires deep expertise with Delta Live Tables, Medallion Architecture, and Unity Catalog, plus strong PySpark/SQL skills. Remote with travel as needed, full-time, US-ready.

Qualifications

• Strong hands-on experience in Databricks architecture and Lakehouse implementation.
• Extensive experience with AWS and/or Microsoft Azure cloud platforms.
• Strong expertise in Delta Live Tables (DLT) and pipeline development.
• Hands-on experience implementing Medallion Architecture (Bronze, Silver, Gold).
• Strong experience with Databricks Auto Loader for incremental and scalable data ingestion.
• Proficiency in PySpark, Python, and Spark SQL.
• Experience with Delta Lake, Unity Catalog, and data governance.
• Strong understanding of batch and real-time/streaming data processing.
• Experience with performance tuning, optimization, and large-scale data processing.
• Knowledge of CI/CD, Git, and DevOps practices for Databricks deployments.
• Excellent communication, stakeholder management, and technical leadership skills.

Responsibilities

• Design and implement scalable, secure, and high-performance Lakehouse architectures using Databricks.
• Define and implement end-to-end data engineering solutions on AWS and/or Azure.
• Architect and implement Medallion Architecture (Bronze, Silver, and Gold layers).
• Develop and optimize Delta Live Tables (DLT) pipelines for batch and streaming workloads.
• Implement Databricks Auto Loader for efficient and incremental ingestion of structured and unstructured data.
• Establish data governance, security, access controls, and best practices using Unity Catalog.
• Optimize Spark workloads, cluster configurations, job performance, and compute costs.
• Design and implement data integration and transformation frameworks using PySpark and SQL.
• Collaborate with business stakeholders, data engineers, and cross-functional teams to translate business requirements into technical solutions.
• Provide technical leadership, architecture guidance, code reviews, and mentorship to engineering teams.
• Ensure best practices around CI/CD, deployment automation, monitoring, and data quality.

Skills

Databricks architecture

Lakehouse implementation

AWS/Azure cloud

Delta Live Tables

Medallion Architecture

Databricks Auto Loader

PySpark / Spark SQL

Unity Catalog / data governance

CI/CD / DevOps for Databricks

Leadership / communication

Tools

Terraform

Git

Unity Catalog

Job description

Location: Remote (Must be open to travel as per business requirements)

Work Authorization: Only US Citizens and Green Card holders

About Celebal Technologies

Celebal Technologies is a global technology company specializing in Data Engineering, Data Science, Artificial Intelligence, Generative AI, and Cloud Transformation. As a Databricks Elite Partner, we help enterprises accelerate their data and AI transformation journeys through innovative cloud-based solutions.

Role Overview

We are looking for an experienced Databricks Architect to lead the design, architecture, and implementation of scalable, secure, and high-performance data solutions using Databricks on AWS and/or Azure.

The ideal candidate should have strong expertise in Databricks Lakehouse architecture, Delta Live Tables (DLT), Medallion Architecture, and Auto Loader. The candidate will be responsible for defining technical architecture, driving data engineering best practices, and delivering enterprise-scale data platforms.

Key Responsibilities

• Design and implement scalable, secure, and high-performance Lakehouse architectures using Databricks.
• Define and implement end-to-end data engineering solutions on AWS and/or Azure.
• Architect and implement Medallion Architecture (Bronze, Silver, and Gold layers).
• Develop and optimize Delta Live Tables (DLT) pipelines for batch and streaming workloads.
• Implement Databricks Auto Loader for efficient and incremental ingestion of structured and unstructured data.
• Establish data governance, security, access controls, and best practices using Unity Catalog.
• Optimize Spark workloads, cluster configurations, job performance, and compute costs.
• Design and implement data integration and transformation frameworks using PySpark and SQL.
• Collaborate with business stakeholders, data engineers, and cross-functional teams to translate business requirements into technical solutions.
• Provide technical leadership, architecture guidance, code reviews, and mentorship to engineering teams.
• Ensure best practices around CI/CD, deployment automation, monitoring, and data quality.

Required Skills & Qualifications

• Strong hands-on experience in Databricks architecture and Lakehouse implementation.
• Extensive experience with AWS and/or Microsoft Azure cloud platforms.
• Strong expertise in Delta Live Tables (DLT) and pipeline development.
• Hands-on experience implementing Medallion Architecture (Bronze, Silver, Gold).
• Strong experience with Databricks Auto Loader for incremental and scalable data ingestion.
• Proficiency in PySpark, Python, and Spark SQL.
• Experience with Delta Lake, Unity Catalog, and data governance.
• Strong understanding of batch and real-time/streaming data processing.
• Experience with performance tuning, optimization, and large-scale data processing.
• Knowledge of CI/CD, Git, and DevOps practices for Databricks deployments.
• Excellent communication, stakeholder management, and technical leadership skills.

Preferred Qualifications

• Databricks certifications such as Databricks Certified Data Engineer Professional or Databricks Certified Data Engineer Associate.
• Experience with Terraform and Infrastructure as Code.
• Knowledge of data modeling, enterprise data architecture, and data quality frameworks.
• Experience with cloud-native services such as AWS S3, Glue, Lambda, Azure Data Lake Storage, Azure Data Factory, and Azure Key Vault.
• Experience leading enterprise-level Data & AI transformation initiatives.

Eligibility Criteria

• Must be a US Citizen or Green Card holder.
• Candidates must be comfortable working remotely and traveling as required by business needs.
• Must be available for full-time permanent employment.

What We Offer

• Opportunity to work on enterprise-scale Data & AI transformation projects.
• Exposure to cutting-edge Databricks and cloud technologies.
• Opportunity to collaborate with global teams and industry-leading clients.
• A growth-oriented environment with opportunities for technical leadership and career development.