Senior Databricks Solution Engineer

Blueflag-LLP; Salt Lake City (UT); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Competitive salary

Generous annual leave and holidays

Health and dental plans

401(k) with company match

Life insurance and AD&D

Ongoing training and development

Job summary

BlueFlag is hiring as a Databricks Solutions Engineer to establish the platform standards for a premier data and AI platform for the VA. You will shape workspace topology, security, governance, and reference implementations, guiding thousands of teams on analytics, reporting, and ML workloads.

You’ll work with Unity Catalog governance, medallion data flows, and high-performance lakehouse design, while building reusable templates and conducting workshops to accelerate adoption across departments.

Qualifications

• Experience building production data pipelines on Databricks.
• Ability to design platform governance and security controls.
• Proven track record delivering reference implementations and workshops.

Responsibilities

• Design the Databricks platform foundation in Azure Commercial with Terraform.
• Set Unity Catalog governance including security, lineage and access controls.
• Define lakehouse standards for medallion data flow and performance optimization.
• Build reference pipelines for batch, streaming, and CDC using Lakeflow and Auto Loader.
• Establish MLOps patterns with MLflow and feature stores; model serving.
• Implement guardrails for cost, tagging, monitoring, and audit logging in Azure Monitor.
• Create reusable templates and run workshops to enable teams to adopt them.
• Collaborate with security, networking, compliance, and VA stakeholders on standards and adoption.

Skills

Databricks

Unity Catalog

Spark (PySpark)

Structured Streaming

Delta Lake

MLflow

Terraform

Azure

CI/CD (Azure DevOps/GitHub Actions)

Security & Governance

FinOps/Chargeback

Data Modeling (dimensional)

Healthcare data (HL7/FHIR/OMOP)

Education

Bachelor's degree in IT or related field

Tools

Lakeflow Declarative Pipelines

Lakeflow Jobs

Auto Loader

Databricks Asset Bundles

Terraform Databricks provider

Job description

BlueFlag is hiring Databricks Solutions Engineers to set the standards and patterns for a premier data and artificial intelligence (AI) platform at the Department of Veterans Affairs. The platform is moving on-premises and existing cloud workloads to Databricks in Azure Commercial, where it will serve a thousand teams building reporting, analytics, and machine learning. You will define how the platform is built and governed, build the reference implementations teams copy, and teach those teams to deliver on their own.

We're looking for an engineer first: someone who built production pipelines on Databricks, then grew into setting standards and guiding customers. You'll join a friendly, collaborative team doing work that makes a real difference for Veterans and their families.

What You'll Do

• -Design the Databricks platform foundation in Azure Commercial: workspace topology, VNet injection and Private Link, and secure connectivity to ADLS Gen2, Azure SQL, and Event Hubs, all provisioned with Terraform.

• -Set Unity Catalog governance: catalog and schema design, data classification, row- and column-level security, attribute-based access, lineage, and Delta Sharing.
• -Define lakehouse standards for medallion data flow, table layout (liquid clustering, OPTIMIZE, VACUUM), and performance (Photon, serverless).
• -Build reference pipelines for batch, streaming, and change data capture (CDC) with Lakeflow Declarative Pipelines, Lakeflow Jobs, Auto Loader, and Structured Streaming.
• -Set MLOps patterns with MLflow, the Unity Catalog model registry and feature tables, and model serving.
• -Set guardrails for a platform that thousands of teams share: cluster and budget policies, tagging and chargeback, monitoring, alerting, and audit logging to Azure Monitor.
• -Build reusable templates (Databricks Asset Bundles, CI/CD with Azure DevOps or GitHub Actions), and run workshops, office hours, and playbooks that help teams adopt them.
• -Work with security, networking, compliance, and VA stakeholders to agree on standards, review exceptions, and report adoption.

Why Join BlueFlag

At BlueFlag, we're passionate about leveraging cutting-edge technology to make a real difference. You'll be at the forefront of cloud innovation, working on projects that directly impact people's lives. We offer a high-growth, entrepreneurial environment that values fresh ideas and authentic teamwork.

Desired

• -Experience running a platform for 500+ concurrent users or thousands of analytics teams
• -Experience with MLflow, feature stores, and model serving
• -FinOps and chargeback experience for data platforms
• -Experience with Synapse or Microsoft Fabric interoperability and data virtualization (e.g., Lakehouse Federation, Starburst)
• -Data modeling (dimensional, domain-driven) and data quality practice (expectations, SLAs)
• -Experience with healthcare data (HL7, FHIR, OMOP) or Azure Government and FedRAMP High environments
• -Certifications: Databricks Data Engineer Professional or Machine Learning Professional; Azure Solutions Architect Expert (AZ-305) or Azure Security Engineer (AZ-500); HashiCorp Terraform Associate

• Competitive salary
• Generous annual leave and paid holidays
• Comprehensive group health and dental plans
• 401(k) with company match
• Life insurance and AD&D coverage
• Ongoing training and professional development opportunities

• US Citizen: Must be a citizen of the United States
• Security Clearance: Must be able to obtain a public trust clearance. Must be eligible to work in the United States.

• -12+ years in data engineering or analytics, including 5+ years on cloud data platforms (Azure preferred) and 3+ years hands‑on with Databricks and Delta Lake
• -Proven Unity Catalog implementation with granular governance (row- and column-level security, attribute-based access, lineage)
• -Strong Spark skills (PySpark, Spark SQL), including Structured Streaming and performance tuning
• -Working knowledge of Azure networking (VNets, Private Link, NSGs), Entra ID, Key Vault, ADLS Gen2, Event Hubs, Azure SQL, and Azure Data Factory
• -Practical Terraform experience (Databricks and Azure providers) and CI/CD with Azure DevOps or GitHub Actions
• -A track record implementing audit logging, policy as code, and compliance controls for PII and PHI
• -Customer-facing experience standardizing, teaching, and influencing across many teams, with clear written and verbal communication
• -Bachelor's degree in information technology or a related field, or commensurate experience