Senior Databricks SME

BuzzClan LLC; Texas; On-site; USD 150,000 - 210,000; Full time

Job summary

BuzzClan LLC is seeking a senior Databricks architect to lead end-to-end Lakehouse designs using Delta Lake, Unity Catalog, and medallion models. You will define reference architectures, oversee large ETL/ELT pipelines, and champion data quality and observability across cloud environments.

You will guide clients on multi-cloud deployments, ML workflows with MLflow, and MLOps patterns, mentoring teams and contributing to pre-sales demonstrations. A strong consulting background is valued.

Qualifications

• 10+ years in data engineering, analytics, or platform architecture.
• 5+ years with Databricks including enterprise production deployments.
• Expert in Spark, Delta Lake, SQL.
• Proven track record designing multi-cloud Lakehouse architectures.
• Strong Unity Catalog experience for governance and lineage.
• Ability to lead client engagements and communicate to executives.
• Consulting or professional services background with multiple concurrent projects.
• Bachelor's degree in CS/Engineering/Data Science or equivalent.

Responsibilities

• Lead end-to-end Databricks Lakehouse design including Delta Lake and Unity Catalog.
• Define reference architectures, best practices, and accelerators for engagements.
• Evaluate deployment patterns across Azure, AWS, and GCP to align with client needs.
• Architect large-scale ETL/ELT pipelines using Spark, Delta Live Tables, and Workflows.
• Design real-time and batch ingestion for structured to unstructured data.
• Establish data quality, observability, and lineage standards; guide ML workflows with MLflow and Feature Store.
• Advise on MLOps patterns including model versioning, A/B testing, monitoring, and retraining pipelines.
• Collaborate with data science teams to productionize models and integrate outputs into apps.
• Serve as trusted advisor on data platform strategy and Databricks adoption.

Skills

Databricks

Spark Py/Scala

Delta Lake

Unity Catalog

Lakehouse

Multi-cloud

ML workflows

Client advisory

Leadership

Executive communication

Education

Bachelor's degree in Computer Science, Engineering, Data Science, or related field

Tools

dbt

Terraform/Pulumi

CI/CD

MLflow

Feature Store

Model Serving

Photon engine

Job description

Must Have

The SME role calls for deep Databricks and Spark expertise, including Lakehouse architecture, Delta Lake, Unity Catalog, PySpark or Scala, and enterprise production deployments. Client advisory and leadership experience are also important.

Job Description

Solution Architecture & Design

• Lead end to end design of Databricks Lakehouse architectures, including Delta Lake, Unity Catalog, and medallion (bronze/silver/gold) data models.
• Define reference architectures, best practices, and reusable accelerators for client engagements.
• Evaluate and recommend deployment patterns across Azure Databricks, AWS Databricks, and GCP, aligning platform choices with client infrastructure and governance needs.
• Architect and oversee large scale ETL/ELT pipelines using Apache Spark, Delta Live Tables, and Databricks Workflows.
• Design real time and batch ingestion frameworks covering structured, semi structured, and unstructured data sources.
• Establish data quality, lineage, and observability standards using Unity Catalog, Delta Lake features, and complementary tooling.
• Guide clients on implementing end to end ML workflows using MLflow, Feature Store, and Model Serving on Databricks.
• Advise on MLOps patterns including model versioning, A/B testing, monitoring, and automated retraining pipelines.
• Collaborate with data science teams to productionize models and integrate ML outputs into business applications.

Client Advisory & Engagement Leadership

• Serve as the trusted advisor to senior client stakeholders on all matters related to data platform strategy and Databricks adoption.
• Lead discovery workshops, architecture reviews, and technical assessments to define solution roadmaps.
• Support pre sales efforts by developing proposals, estimating level of effort, and delivering technical demonstrations.

Thought Leadership & Practice Development

• Contribute to the firm's intellectual property by developing frameworks, white papers, and training curricula around Databricks.
• Mentor and upskill consultants across the data practice; foster a community of Databricks practitioners.
• Represent the firm at industry events, Databricks partner summits, and webinars.

Qualifications

Required Qualifications

• 10+ years of professional experience in data engineering, analytics, or platform architecture.
• 5+ years of deep, hands on experience with Databricks, including production deployments at enterprise scale.
• Expert level proficiency in Apache Spark (PySpark and/or Scala), Delta Lake, and SQL.
• Proven track record designing multi cloud Lakehouse architectures (Azure, AWS, or GCP).
• Strong experience with Unity Catalog for data governance, access control, and lineage tracking.
• Demonstrated ability to lead client facing engagements and communicate complex technical concepts to executive audiences.
• Experience in a consulting or professional services environment, with a history of managing multiple concurrent engagements.
• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field (or equivalent practical experience).

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Machine Learning Professional certification.
• Experience with Delta Live Tables, Databricks Workflows, and Photon engine optimization.
• Familiarity with complementary tools such as dbt, Terraform/Pulumi (IaC), and CI/CD pipelines for data platforms.
• Hands on experience with MLflow, Feature Store, and Model Serving in production environments.
• Prior experience as a Databricks partner or in a Databricks aligned practice within a consulting firm.
• Advanced degree (M.S. or Ph.D.) in a quantitative or technical discipline.

Additional Information

All your information will be kept confidential according to EEO guidelines.