Senior Databricks Engineer: ML & Analytics Solutions

BlueFlag LLC; Pittsburgh (Allegheny County); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Competitive salary

Generous leave and holidays

Comprehensive health plans

401(k) with company match

Job summary

BlueFlag LLC is hiring Senior Databricks Engineers to help VA teams move their data work to a premier data and AI platform built on Databricks in Azure Commercial. The role is hands-on and customer-facing, collaborating with VA teams to modernize workloads and teach adoption.

Two focus areas exist: ML workflows and Analytics pipelines, both requiring production data pipelines, strong Python/SQL skills, and experience with Delta Lake, Unity Catalog, and Azure services.

Qualifications

• 7+ years in data engineering, including 2+ years on Databricks production workloads.
• Strong proficiency in Python, SQL, PySpark, and Spark SQL.
• Hands-on experience with Delta Lake, lakehouse design, and Spark performance tuning.
• Hands-on Unity Catalog experience: catalogs, grants, lineage, and security.
• Experience migrating on-prem workloads (SQL Server, SSIS) to the cloud.
• Experience with Azure services including ADLS Gen2, Data Factory, and Key Vault.
• Git, CI/CD, and infrastructure as code (Terraform, Databricks Asset Bundles).
• Customer-facing consulting or professional services experience with requirements sessions and training.

Responsibilities

• Scope use cases with VA teams and plan migrations to Databricks on Azure Commercial.
• Assess on-prem workloads and migrate them to Databricks patterns.
• Build and tune pipelines with PySpark, Spark SQL, Delta Lake, Lakeflow, and Data Factory.
• Use Unity Catalog for access control, lineage, and data sharing.
• Teach customers via workshops and documentation to run their workloads independently.
• Package reusable templates and code to accelerate future migrations.
• Develop end-to-end ML workflows including feature engineering, training, evaluation, and deployment.
• Replace on-premise reporting with Databricks SQL, dashboards, and Power BI.

Skills

Python

SQL

PySpark

Spark SQL

Unity Catalog

Databricks

Education

Bachelor's degree in computer science or information systems

Tools

Azure Data Factory

Git

CI/CD

Terraform

Job description

BlueFlag LLC is hiring Senior Databricks Engineers to help VA teams move their data work to a premier data and AI platform built on Databricks in Azure Commercial. The role is hands-on and customer-facing, collaborating with VA teams to modernize workloads and teach adoption.

Two focus areas exist: ML workflows and Analytics pipelines, both requiring production data pipelines, strong Python/SQL skills, and experience with Delta Lake, Unity Catalog, and Azure services.