Databricks Solution Architect

Jobtailor

New Jersey

On-site

USD 180,000 - 240,000

Full time

8 days ago

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

Jobtailor seeks a senior data architecture leader to own end-to-end Azure-based data platforms, focusing on Databricks Lakehouse, Delta Lake, and scalable cloud-native ecosystems. You will define target-state architecture, ingestion and transformation pipelines, and serving layers for reporting, analytics, and ML.

Responsibilities include leading ETL/ELT design, governance, security, and cost optimization, while mentoring teams and aligning with business goals.

Qualifications

10–15 years of experience in data engineering, cloud data platform design, or enterprise data architecture
At least 5+ years of hands-on experience with Databricks and Azure
Bachelor’s degree in computer science, IT, engineering, or related discipline; master’s preferred
Expertise in Lakehouse, medallion architecture, data modeling, data warehousing, and scalable ingestion and transformation frameworks
Proficiency in Databricks, PySpark, Python, SQL, Delta Lake, Databricks Workflows, Auto Loader, and Delta Live Tables
Experience with Azure Data Factory, Azure Data Lake Storage, Azure Key Vault, Azure DevOps, and enterprise cloud ecosystem integration
Experience defining architecture standards, reusable design patterns, CI/CD strategy, environment management, and delivery best practices
Experience with Unity Catalog, RBAC/ABAC controls, data governance, lineage, security frameworks, and Azure Purview
Ability to optimize large-scale Spark and Databricks workloads, including performance tuning, cluster sizing, workload management, and cost optimization
Experience translating business requirements into scalable solution designs and implementation roadmaps
Strong communication, leadership, and problem-solving skills; ability to mentor teams, review designs, and drive decisions
Insurance domain knowledge is preferred

Responsibilities

Lead the end-to-end architecture and solution design for enterprise data platforms on Azure
Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers
Design and implement ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables
Own architecture standards for data modeling, medallion design, reusable patterns, CI/CD, and release management
Drive platform governance and security using Unity Catalog, RBAC/ABAC, lineage, auditability, and Purview integration
Optimize Spark workloads, cluster policies, partitioning, file sizing, caching, and compute costs
Collaborate with stakeholders to translate requirements into scalable designs
Provide technical leadership through design reviews and best-practice establishment
Evaluate Databricks capabilities including Photon, serverless, Lakehouse Federation, and streaming
Ensure delivery governance through estimation, planning, dependency management, risk mitigation, and Agile execution

Skills

Databricks Lakehouse

PySpark

SQL

Delta Lake

Data Modeling

Data Warehousing

CI/CD

Performance Tuning

Agile Execution

Architecture Standards

Education

Bachelor's Degree

Master’s Degree Preferred

Tools

Azure Data Factory

Azure Data Lake Storage

Azure Key Vault

Azure DevOps

Unity Catalog

Azure Purview

Job description

Lead the end-to-end architecture and solution design for enterprise data platforms on Azure, focusing on Databricks Lakehouse, Delta Lake, and scalable cloud-native data ecosystems
Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers for reporting, advanced analytics, ML, and business-critical decisioning
Design and implement ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables
Own architecture standards for data modeling, medallion design, reusable engineering patterns, CI/CD, code quality, environment strategy, and release management
Drive platform governance and security using Unity Catalog, RBAC/ABAC controls, lineage, auditability, and Azure Purview integration
Optimize Spark workloads, cluster policies, partitioning, file sizing, caching, and compute costs
Collaborate with stakeholders, product owners, analysts, architects, and downstream consumers to translate requirements into scalable technical designs
Provide technical leadership through design reviews, implementation guidance, architectural issue resolution, and best-practice establishment
Evaluate Databricks capabilities including Photon, serverless compute, Lakehouse Federation, and streaming patterns
Ensure delivery governance through estimation, technical planning, dependency management, risk mitigation, and Agile execution

Requirements

10–15 years of experience in data engineering, cloud data platform design, or enterprise data architecture
At least 5+ years of hands-on experience with Databricks and Azure
Bachelor’s degree in computer science, Information Technology, Engineering, or a related discipline; master’s degree preferred
Expertise in Lakehouse, medallion architecture, data modeling, data warehousing, and scalable ingestion and transformation frameworks
Proficiency in Databricks, PySpark, Python, SQL, Delta Lake, Databricks Workflows, Auto Loader, and Delta Live Tables
Experience with Azure Data Factory, Azure Data Lake Storage, Azure Key Vault, Azure DevOps, and enterprise cloud ecosystem integration
Experience defining architecture standards, reusable design patterns, CI/CD strategy, environment management, and delivery best practices
Experience with Unity Catalog, role-based access controls, data governance, lineage, security frameworks, and Azure Purview
Ability to optimize large-scale Spark and Databricks workloads, including performance tuning, cluster sizing, workload management, and cost optimization
Experience translating business requirements into scalable solution designs and implementation roadmaps
Strong communication, leadership, and problem-solving skills; ability to mentor teams, review technical designs, and drive architecture decisions
Insurance domain knowledge is preferred

Core Competencies

Demonstrates expertise in designing and implementing enterprise data architectures on Azure, with a focus on Databricks Lakehouse and Delta Lake. Proficient in optimizing data ingestion, transformation frameworks, and ensuring data governance and security.

Highest-signal resume keywords