Senior Databricks Engineer - Cloud Data & ML Migrations

BlueFlag LLC; Dallas (TX); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Competitive salary

Health benefits

401(k) with company match

Paid time off

Job summary

BlueFlag LLC seeks a Senior Databricks Engineer to help VA teams migrate on-prem data workloads to a Databricks-based platform in Azure Commercial. The role is hands-on and customer-facing, including workshops and reusable templates for rapid migration.

You’ll build and optimize pipelines, apply Unity Catalog governance, and support ML and analytics focus areas across ETL modernization and reporting.

Qualifications

• 7+ years in data engineering, including 2+ years building production workloads on Databricks
• Strong proficiency in Python, SQL, PySpark, and Spark SQL
• Hands-on experience with Delta Lake, lakehouse (medallion) design, and Spark performance tuning
• Hands-on Unity Catalog experience: catalogs, grants, lineage, and row- and column-level security
• Experience migrating on-prem workloads (SQL Server, SSIS, stored procedures) to the cloud
• Experience with Azure services, including ADLS Gen2, Azure Data Factory, Entra ID, and Key Vault
• Experience with Git, CI/CD, and infrastructure as code (Databricks Asset Bundles, Terraform)
• Customer-facing consulting or professional services experience, including requirements sessions, workshops, and training
• Clear written and verbal communication with technical and non-technical audiences
• US Citizenship: Must be a citizen of the United States
• Security Clearance: Must be able to obtain a public trust clearance; must be eligible to work in the United States
• Bachelor’s degree in computer science, information systems, or a related field

Responsibilities

• Work directly with VA teams to scope use cases, plan migrations, and deliver working solutions on Databricks
• Assess on-premises workloads (SQL Server, SSIS packages, stored procedures, scheduled jobs) and adapt to Databricks patterns in Azure Commercial
• Build and tune pipelines with PySpark, Spark SQL, Delta Lake, Lakeflow, and Azure Data Factory
• Use Unity Catalog for access, lineage, and sharing to keep data governed
• Teach customers through workshops and documentation so they can own their workloads
• Package reusable templates and code to speed up migrations
• Build end-to-end ML workflows: feature engineering, training, evaluation, MLflow tracking, Unity Catalog model registry, and model serving
• Help data scientists move models from notebooks and on-prem servers into governed production
• Support generative AI use cases (RAG, agents) alongside predictive models
• Convert SSIS and stored procedure ETL into Databricks pipelines with reconciliation
• Replace on-prem reporting with Databricks SQL, AI/BI dashboards, and Power BI
• Publish reporting data from Databricks SQL warehouses with semantic models and tuning

Skills

Databricks

Python

SQL

PySpark

Spark SQL

Delta Lake

Azure Data Factory

Unity Catalog

Git

CI/CD

Terraform

Education

Bachelor's degree in computer science/information systems/related field

Tools

Databricks Asset Bundles

Terraform

Job description

BlueFlag LLC seeks a Senior Databricks Engineer to help VA teams migrate on-prem data workloads to a Databricks-based platform in Azure Commercial. The role is hands-on and customer-facing, including workshops and reusable templates for rapid migration.

You’ll build and optimize pipelines, apply Unity Catalog governance, and support ML and analytics focus areas across ETL modernization and reporting.