Data Engineer / Platform Administrator (Azure Databricks)

Vanderbilt Health; United States; A distancia; USD 110.000 - 145.000; Jornada completa

Descripción de la vacante

Vanderbilt Health in the United States is seeking a Data Engineer / Platform Administrator to deepen their technical skills on an Azure Databricks platform with 100+ workspaces. This role blends platform administration with hands-on data engineering to build and optimize data pipelines.

You will own defined platform areas, collaborate with senior engineers, and work toward scalable, cost-aware solutions. Note: this role can be performed fully remote from most US locations.

Formación

• Bachelor's degree in a relevant field.
• 1+ year of Databricks experience.
• 2+ years Python in an object-oriented paradigm.
• 2+ years implementing cloud solutions with Azure, AWS, or GCP.
• 1+ year of SQL experience.

Responsabilidades

• Administer our Azure Databricks environment of 100+ workspaces, including compute policies, access controls, and governance, working with senior team members on larger architectural decisions.
• Build automation and Infrastructure as Code to provision and manage workspaces at scale.
• Build solutions for proactive monitoring of platform health, performance, and costs.
• Test new Databricks features and promote best practices to teams across the organization.
• Develop tools and services that other teams use to create, test, and deliver data-related assets.
• Design and develop performance-focused data pipelines that support multiple source systems.
• Troubleshoot issues with production pipelines and work with internal stakeholders to deploy fixes.
• Build reusable solutions for recurring problems and help ensure data accuracy and integrity.
• Work with stakeholders to identify data needs and look for opportunities to improve platform scalability.

Conocimientos

Python (OO)

Data engineering concepts

Communication

SQL fundamentals

Educación

Bachelor's degree

Herramientas

Databricks

Terraform

GitHub/GitLab/Azure DevOps CI/CD

Docker/Podman

REST APIs

Descripción del empleo

Discover Vanderbilt Health : With eight hospitals and over 180 clinics, we are the region's largest comprehensive academic health system. Vanderbilt Health has a long tradition of providing world-class care to the people of the Mid-South, living out our mission of moving medicine forward through preeminent programs in patient care, education and research. Here you'll find a spirit of collaboration and a supportive community where your expertise is valued, your knowledge expanded and your uniqueness celebrated. We are committed to fostering an environment where talented, compassionate people are united in bringing the most advanced care to our neighbors throughout the region.

Organization:

HealthIT Data Platform Services

Job Summary:

The Data Platform Services team is looking for a Data Engineer / Platform Administrator with an inquisitive, analytical mindset who wants to deepen their technical skills. About 80% of your time will go to administering our Azure Databricks platform of 100+ workspaces: building automation, monitoring performance and costs, managing configuration, testing new features, and promoting best practices. The remaining 20% will be spent as a data engineer, building and optimizing data pipelines alongside the rest of the team.

You'll own defined areas of the platform from the start, with support from senior engineers and room to take on broader scope as you grow.

Note: This role can be performed fully remote from most US locations

KEY RESPONSIBILITIES

Platform Administration (~80%)

• Administer our Azure Databricks environment of 100+ workspaces, including compute policies, access controls, and governance, working with senior team members on larger architectural decisions
• Build automation and Infrastructure as Code to provision and manage workspaces at scale
• Build solutions for proactive monitoring of platform health, performance, and costs
• Test new Databricks features and promote best practices to teams across the organization
• Develop tools and services that other teams use to create, test, and deliver data-related assets

Data Engineering (~20%)

• Design and develop performance-focused data pipelines that support a variety of source systems (flat files, APIs, databases, and more)
• Troubleshoot issues with production pipelines and work with internal stakeholders to deploy fixes
• Build reusable solutions for recurring problems and help ensure data accuracy and integrity
• Work with stakeholders to identify data needs and look for opportunities to improve platform scalability

Technical Qualifications

Minimum Qualifications

• Bachelor's degree
• 1+ year of experience with Databricks (certifications are a plus)
• 2+ years of Python in an object-oriented paradigm
• 2+ years implementing cloud solutions with Azure, AWS, or GCP
• 1+ year of SQL experience

Preferred Qualifications

• Experience with Infrastructure as Code solutions such as Terraform
• Experience with CI/CD automation in GitLab, GitHub, or Azure DevOps
• Experience with REST data services, APIs, and microservices
• Experience with containerization using Docker or Podman

Bonus Qualifications

• Experience with PowerShell and/or Bash
• A link to your GitHub profile or other personal code examples

TECHNICAL CAPABILITIES

CORPORATE SYSTEMS ARCH STANDARDS (PROFICIENT): Has basic knowledge of the architecture of the systems which support more than one core business area. Demonstrates knowledge of how the core business areas' applications and support systems are structured, how various components and parts interact, as well as what protocols and interfaces are used for communications and cooperation between modules and components. Participates in the overall planning and design of the specific core business systems. Able to train and educate others in the structure and components of systems architecture for those core business areas. BUSINESS ARCHITECTURE (PROFICIENT): Has basic knowledge of the architecture of the business systems which support more than one core business area. Demonstrates knowledge of how the core business areas' applications and support systems are structured, how various components and parts interact, as well as what protocols and interfaces are used for communications and cooperation between components. CLOUD ARCHITECTURE (PROFICIENT): Has basic knowledge of leveraging cloud-based architecture to support the life cycle of a data lake. Demonstrates an understanding of when and how to scale cloud components to provide the required level of performance and support of a data lake. DATA LAKES (PROFICIENT): Demonstrates a basic understanding of the effective application, operation, or practice in complex data lakes. Experienced in all core features and functions. Demonstrates a thorough understanding of advanced concepts. Applies concepts and tools consistently in delivering data lakes. May have successfully completed training in advanced techniques, features, and tools. DATA MARTs (PROFICIENT): Proficient understanding of fundamental