Senior Databricks Engineer — AI/ML & Analytics Migration Lead

Carbon Data Solutions; United States; Remote; USD 140,000 - 190,000; Full time

Benefits offered by this job

Competitive salary

Generous annual leave

Job summary

BlueFlag is hiring Senior Databricks Engineers to help Department of Veterans Affairs teams move their data workloads to a premier Databricks-based platform on Azure Commercial. The role is hands-on and customer-facing, blending data engineering with professional services.

You will work with VA teams to scope migrations, rebuild on Databricks, and teach users to operate the platform confidently. Expect production pipelines, collaboration with data scientists, and the creation of reusable

Qualifications

• 7+ years in data engineering
• 2+ years building production workloads on Databricks
• Proficient in Python, SQL, PySpark
• Hands-on Unity Catalog experience with governance
• Experience migrating on-prem workloads to the cloud
• Familiar with Azure services (ADLS Gen2, Key Vault, Entra ID)
• Git, CI/CD, and infrastructure as code
• Customer-facing consulting experience
• Clear written and verbal communication
• Bachelor’s degree in CS or related field
• US Citizenship and ability to obtain clearance

Responsibilities

• Work with VA teams to scope use cases and plan migrations
• Assess on-prem workloads and adapt to Databricks patterns
• Build and tune pipelines using PySpark, Spark SQL, and Delta Lake
• Use Unity Catalog for access control, lineage, and sharing
• Teach customers through workshops and documentation
• Package reusable templates and code to speed future migrations

Skills

Python

SQL

PySpark

Spark SQL

Communication

Customer-facing

Education

Bachelor's degree in computer science or related field

Tools

Databricks

Delta Lake

Unity Catalog

Azure Data Factory

Terraform

Git

CI/CD

Job description

BlueFlag is hiring Senior Databricks Engineers to help Department of Veterans Affairs teams move their data workloads to a premier Databricks-based platform on Azure Commercial. The role is hands-on and customer-facing, blending data engineering with professional services.

You will work with VA teams to scope migrations, rebuild on Databricks, and teach users to operate the platform confidently. Expect production pipelines, collaboration with data scientists, and the creation of reusable