Data Engineer / Platform Administrator (Azure Databricks)

Vanderbilt University Medical Center; United States; Remote; USD 110,000 - 150,000; Full time

Job summary

Vanderbilt University Medical Center seeks a Data Engineer / Platform Administrator to manage the Azure Databricks platform (100+ workspaces) and build scalable data pipelines. The role combines platform governance with hands-on data engineering to optimize performance, costs, and data quality.

The position offers a remote-friendly setup from most US locations, with an 80/20 mix between platform administration and data engineering responsibilities.

Qualifications

• Bachelor's degree required.
• 1+ year of experience with Databricks (certifications are a plus).
• 2+ years of Python in an object-oriented paradigm.
• 2+ years implementing cloud solutions with Azure, AWS, or GCP.
• 1+ year of SQL experience.

Responsibilities

• Platform Administration (~80%): administer Azure Databricks environment with 100+ workspaces, governance, access controls, and compute policies.
• Build automation and Infrastructure as Code to provision/manage workspaces at scale.
• Monitor platform health, performance, and costs; test new Databricks features and promote best practices.
• Data Engineering (~20%): design/develop data pipelines across multiple source systems; troubleshoot production pipelines; ensure data quality and scalability.

Skills

Databricks

Python

Cloud platforms

SQL

Education

Bachelor's degree

Tools

Terraform

CI/CD (GitLab/GitHub/Azure DevOps)

REST APIs

Docker/Podman

Job description

Discover Vanderbilt Health :With eight hospitals and over 180 clinics, we are the region’s largest comprehensive academic health system. Vanderbilt Health has a long tradition of providing world-class care to the people of the Mid-South, living out our mission of moving medicine forward through preeminent programs in patient care, education and research. Here you’ll find a spirit of collaboration and a supportive community where your expertise is valued, your knowledge expanded and your uniquenesscelebrated. We are committed to fostering an environment where talented, compassionate people are united in bringing the most advanced care to our neighbors throughout the region.

Organization:

HealthIT Data Platform Services

Job Summary:

The Data Platform Services team is looking for a Data Engineer / Platform Administrator with an inquisitive, analytical mindset who wants to deepen their technical skills. About 80% of your time will go to administering our Azure Databricks platform of 100+ workspaces: building automation, monitoring performance and costs, managing configuration, testing new features, and promoting best practices. The remaining 20% will be spent as a data engineer, building and optimizing data pipelines alongside the rest of the team.

You’ll own defined areas of the platform from the start, with support from senior engineers and room to take on broader scope as you grow.

Note: This role can be performed fully remote from most US locations

.

KEY RESPONSIBILITIES

Platform Administration (~80%)

• Administer our Azure Databricks environment of 100+ workspaces, including compute policies, access controls, and governance, working with senior team members on larger architectural decisions
• Build automation and Infrastructure as Code to provision and manage workspaces at scale
• Build solutions for proactive monitoring of platform health, performance, and costs
• Test new Databricks features and promote best practices to teams across the organization
• Develop tools and services that other teams use to create, test, and deliver data-related assets

Data Engineering (~20%)

• Design and develop performance-focused data pipelines that support a variety of source systems (flat files, APIs, databases, and more)
• Troubleshoot issues with production pipelines and work with internal stakeholders to deploy fixes
• Build reusable solutions for recurring problems and help ensure data accuracy and integrity
• Work with stakeholders to identify data needs and look for opportunities to improve platform scalability

Technical Qualifications

Minimum Qualifications

• Bachelor’s degree
• 1+ year of experience with Databricks (certifications are a plus)
• 2+ years of Python in an object-oriented paradigm
• 2+ years implementing cloud solutions with Azure, AWS, or GCP
• 1+ year of SQL experience

Preferred Qualifications

• Experience with Infrastructure as Code solutions such as Terraform
• Experience with CI/CD automation in GitLab, GitHub, or Azure DevOps
• Experience with REST data services, APIs, and microservices
• Experience with containerization using Docker or Podman

Bonus Qualifications

• Experience with PowerShell and/or Bash
• A link to your GitHub profile or other personal code examples

Technical Capabilities

• CORPORATE SYSTEMS ARCH STANDARDS (PROFICIENT): Has basic knowledge of the architecture of the systems which support more than one core business area. Demonstrates knowledge of how the core business areas’ applications and support systems are structured, how various components and parts interact, as well as what protocols and interfaces are used for communications and cooperation between modules and components. Participates in the overall planning and design of the specific core business systems. Able to train and educate others in the structure and components of systems architecture for those core business areas.
• BUSINESS ARCHITECTURE (PROFICIENT): Has basic knowledge of the architecture of the business systems which support more than one core business area. Demonstrates knowledge of how the core business areas’ applications and support systems are structured, how various components and parts interact, as well as what protocols and interfaces are used for communications and cooperation between components.
• CLOUD ARCHITECTURE (PROFICIENT): Has basic knowledge of leveraging cloud-based architecture to support the life cycle of a data lake. Demonstrates an understanding of when and how to scale cloud components to provide the required level of performance and support of a data lake.
• DATA LAKES (PROFICIENT): Demonstrates a basic understanding of the effective application, operation, or practice in complex data lakes. Experienced in all core features and functions. Demonstrates a thorough understanding of advanced concepts. Applies concepts and tools consistently in delivering data lakes. May have successfully completed training in advanced techniques, features, and tools.
• DATA MARTS (PROFICIENT): Proficient understanding of fundamental data mart concepts, encompassing their purpose, significance within data warehousing, and the advantages they provide in supporting data analytics efforts.
• DATA WAREHOUSING (PROFICIENT): Demonstrates a foundational understanding of data warehousing and its role within an organization. Possesses knowledge of basic corporate data warehouse models and their purposes. Understands the core components of data warehousing systems. Familiar with fundamental data warehouse structures both in theory and basic practical application. Capable of basic performance considerations for data warehousing components.

About the Department:

Health IT

HealthITprovides the best health information technology tools that support Vanderbilt University Medical Center’s mission of:

• Delivering distinctively personalized care
• Improving the health care of individuals and communities regionally, nationally and internationally
• Providing transformative learning programs
• Supporting compelling discoveries

Our tools, which form the digital arteries of VUMC, are either developed in-house by our innovative product teams or selected from the most cutting-edge solutions available in today’s ever-changing marketplace. Our 500 colleagues provide ongoing support over each product’s entire lifespan, ensuring that the tools are meeting the evolving needs of the Medical Center’s 24,000 colleagues.

Our solutions are driven by the incredible work and research of our colleagues throughout Vanderbiltand supported through a close partnership with VUMCInformation Technology (VUMC IT). The strong collaboration among our teams means that VUMC can respond to clinical and operational issues with agility and innovation. Together, we ensure VUMC remains a leader in its pioneering use of healthcare information technology.

Our core values of accountability,transparencyand execution, delivered with a strong Partner Promise.

Our professional administrative functions include critical supporting roles in information technology and informatics, finance, administration, legal and community affairs, human resources, communications and marketing, development, facilities, and many more.

At our growing health system, we support each other and encourage excellence among all who are part of our workforce. High-achieving employees stay at Vanderbilt Health for professional growth, appreciation of benefits, and a sense of community and purpose.

Core Accountabilities:

Organizational Impact: Executes job responsibilities with the understanding of how output would affect and impact other areas related to own job area/team with occasional guidance. Problem Solving/ Complexity of work: Analyzes moderately complex problems using technical experience and judgment. Breadth of Knowledge: Has expanded knowledge gained through experience within a professional area. Team Interaction: Provides informal guidance and support to team members.

Core Capabilities :

Supporting Colleagues:- Develops Self and Others: Invests time, energy, and enthusiasm in developing self/others to help improve performance e and gain knowledge in new areas.- Builds and Maintains Relationships: Maintains regular contact with key colleagues and stakeholders using formal and informal opportunities to expand and strengthen relationships.- Communicates Effectively: Recognizes group interactions and modifies one’s own communication style to suit different situations and audiences. Delivering Excellent Services:- Serves Others with Compassion: Seeks to understand current and future needs of relevant stakeholders and customizes services to better address them.- Solves Complex Problems: Approaches problems from different angles; Identifies new possibilities to interpret opportunities and develop concrete solutions.- Offers Meaningful Advice and Support: Provides ongoing support and coaching in a constructive manner to increase employees’ effectiveness. Ensuring High Quality: - Performs Excellent Work: Engages regularly in formal and informal dialogue about quality; directly addresses quality issues promptly.- Ensures Continuous Improvement: Applies various learning experiences by looking beyond symptoms to uncover underlying causes of problems and identifies ways to resolve them. - Fulfills Safety and Regulatory Requirements: Understands all aspects of providing a safe environment and performs routine safety checks to prevent safety hazards from occurring. Managing Resources Effectively: - Demonstrates Accountability: Demonstrates a sense of ownership, focusing on and driving critical issues to closure.- Stewards Organizational Resources: Applies understanding of the departmental work to effectively manage resources for a department/area.- Makes Data Driven Decisions: Demonstrates strong understanding of the information or data to identify and elevate opportunities. Fostering Innovation:- Generates New Ideas: Proactively identifies new ideas/opportunities from multiple sources or methods to improve processes beyond conventional approaches.- Applies Technology: Demonstrates an enthusiasm for learning new technologies, tools, and procedures to address short-term challenges.- Adapts to Change: Views difficult situations and/or problems as opportunities for improvement; actively embraces change instead of emphasizing negative elements.

Experience Level:

2 years

Education:

Bachelor’s

This role offers the opportunity to make a meaningful impact within Vanderbilt Health, supported by a comprehensive benefits package which mayinclude health, disability, retirement and/or wellness offerings to enhance your well-being and professional growth.

Vanderbilt Health is committed tofosteringan environment where everyone has the chance to thrive and is committed to the principles of equal opportunity. EOE/Vets/Disabled.