Remote Databricks SME for Data Modernization

GovCIO; Harrisburg (Dauphin County); Remote; USD 175,000 - 195,000; Full time

Benefits offered by this job

Employee Assistance Program (EAP)

Corporate Discounts

Learning & Development platform

Training, Education and Certification

Referral Bonus Program

Internal Mobility Program

Pet Insurance

Flexible Work Environment

Job summary

GovCIO is seeking a Databricks Subject Matter Expert to support the VA Data Modernization initiative. You will guide data scientists and engineers to deliver production-ready data solutions and drive workspace automation, onboarding support, and user success entirely remotely within the United States.

You will implement linked services, develop PySpark jobs, build Delta Lake-based pipelines, and ensure data quality and security while collaborating across teams to migrate workloads to the cloud.

Qualifications

• Bachelor’s degree in Information Technology or a related field (or commensurate experience).
• 12+ years of experience in business analysis, project management, or a similar role.
• Strong experience with Databricks and Azure data services, including workspace administration, catalog creation, linked services, storage mounts, and access configuration.
• Proficiency in ETL/ELT pipeline development, data modeling, and managing structured and unstructured datasets.
• Ability to automate provisioning workflows and infrastructure using scripting or infrastructure‑as‑code tools (Python, PowerShell, Bash, Terraform, ARM, or Bicep).
• Strong troubleshooting and problem solving for diagnosing platform, data access, and pipeline issues.
• Effective communication: documenting processes, creating reusable templates, and onboarding users.

Responsibilities

• Implement and manage linked services, mount points, and catalog configuration in Databricks and VA’s data lake.
• Develop and optimize PySpark jobs, notebooks, and workflows in Databricks.
• Build ETL/ELT pipelines using Databricks Lakehouse features.
• Design and implement Spark-based data processing for analytics workloads.
• Optimize Spark jobs for performance, cost, and scalability.
• Collaborate with data scientists, analysts, and engineers to deliver production-ready data solutions.
• Ensure data quality, governance, and security best practices.
• Lead migration guidance and self-service resources to accelerate platform adoption.

Skills

Databricks

Azure data services

Workspace administration

ETL/ELT pipelines

Python

PowerShell

Bash

Terraform

ARM

Bicep

Troubleshooting

Documentation

Education

Bachelor’s degree in Information Technology or related field

Tools

Databricks

Azure

Job description

GovCIO is seeking a Databricks Subject Matter Expert to support the VA Data Modernization initiative. You will guide data scientists and engineers to deliver production-ready data solutions and drive workspace automation, onboarding support, and user success entirely remotely within the United States.

You will implement linked services, develop PySpark jobs, build Delta Lake-based pipelines, and ensure data quality and security while collaborating across teams to migrate workloads to the cloud.