Azure Databricks Platform Engineer | Data Pipelines & Automation

Vanderbilt Health; United States; Remote; USD 110,000 - 145,000; Full time

Job summary

Vanderbilt Health in the United States is seeking a Data Engineer / Platform Administrator to deepen their technical skills on an Azure Databricks platform with 100+ workspaces. This role blends platform administration with hands-on data engineering to build and optimize data pipelines.

You will own defined platform areas, collaborate with senior engineers, and work toward scalable, cost-aware solutions. Note: this role can be performed fully remote from most US locations.

Qualifications

• Bachelor's degree in a relevant field.
• 1+ year of Databricks experience.
• 2+ years Python in an object-oriented paradigm.
• 2+ years implementing cloud solutions with Azure, AWS, or GCP.
• 1+ year of SQL experience.

Responsibilities

• Administer our Azure Databricks environment of 100+ workspaces, including compute policies, access controls, and governance, working with senior team members on larger architectural decisions.
• Build automation and Infrastructure as Code to provision and manage workspaces at scale.
• Build solutions for proactive monitoring of platform health, performance, and costs.
• Test new Databricks features and promote best practices to teams across the organization.
• Develop tools and services that other teams use to create, test, and deliver data-related assets.
• Design and develop performance-focused data pipelines that support multiple source systems.
• Troubleshoot issues with production pipelines and work with internal stakeholders to deploy fixes.
• Build reusable solutions for recurring problems and help ensure data accuracy and integrity.
• Work with stakeholders to identify data needs and look for opportunities to improve platform scalability.

Skills

Python (OO)

Data engineering concepts

Communication

SQL fundamentals

Education

Bachelor's degree

Tools

Databricks

Terraform

GitHub/GitLab/Azure DevOps CI/CD

Docker/Podman

REST APIs

Job description

Vanderbilt Health in the United States is seeking a Data Engineer / Platform Administrator to deepen their technical skills on an Azure Databricks platform with 100+ workspaces. This role blends platform administration with hands-on data engineering to build and optimize data pipelines.

You will own defined platform areas, collaborate with senior engineers, and work toward scalable, cost-aware solutions. Note: this role can be performed fully remote from most US locations.