Data Platform Engineer

Clinician Nexus; United States; On-site; USD 120,000 - 190,000; Full time

Job summary

Clinician Nexus is hiring a Data Platform Engineer to design, implement, and maintain scalable data infrastructure. You will build data lakes and lake houses, create streaming and batch pipelines, and integrate them with software products across teams.

The role requires strong Python expertise, Spark proficiency, and experience with ELT/ETL in distributed compute environments, preferably Databricks, while mentoring junior engineers and upholding data governance and quality standards.

Qualifications

• 5+ years of relevant experience.
• Degree in Computer Science, Software Engineering, Information Systems, Information Technology or a related computer degree or equivalent experience.
• Master's degree is a plus
• Must know and have proficiency in one object or object/functional programing language. Preferably Python
• Must know common object and object/functional design patterns. Builder, factory, façade, context, etc.
• Must have proficiency with Apache Spark
• Must know data lake and lake house design principles, OLTP (Online Transaction Processing) design principles, document data stores, and graph
• Must know and understand how to build ELT/ETL patterns in a distributed compute system. Preferably Databricks
• Must know and have proficiency with common data quality tooling and the design of configurable systems to front end that tooling
• Must know and have proficiency with common systems and data observability tooling

Responsibilities

• Collaborate with data platform architects to design tooling, services, and integrations.
• Build data lake and lake house architectures.
• Build streaming data architectures to support data platform tooling and services.
• Collaborate with software teams to integrate streaming and batch architectures with software products.
• Collaborate with data scientists, analysts, and business stakeholders to MVP optimal solutions.
• Collaborate with data governance teams and data platform architects to build governance into technical solutions.
• Maintain systems and services that provide transparency and observability into our critical systems.
• Implement and promote engineering and architectural patterns, perform code reviews, and collaborate in architectural reviews.
• Collaborate with product team data engineers and architects to MVP data products, maintain data models, and promote data modeling best practice.
• Provide technical leadership and mentorship to product and data engineers, guiding their growth and professional development and enabling their ability to use platform tooling and services.
• Lead by example through hands-on contributions to designing, coding, and troubleshooting complex data systems.
• Identify and address performance bottlenecks and optimization opportunities within data pipelines, databases, and processing frameworks.
• Optimize data processing workflows to improve efficiency and reduce latency.
• Lead efforts to diagnose and resolve data-related incidents in a timely manner.
• Will participate in the grooming of stories
• Will be responsible for their own tasking towards the completion of stories
• Will mentor junior members of the team and guide junior members on best practice
• Will participate in design
• Will be responsible for the quality of their own code and will participate in code review of others product
• Will be responsible for integrating their own code with the team's DevOps plan and implementation

Skills

Python

Software design patterns

Apache Spark

Data lake / lake house

ETL/ELT patterns

Data quality tooling

Observability tooling

Education

Bachelor's degree in Computer Science, Software Engineering, Information Systems, Information Technology or related

Master's degree is a plus

Tools

Databricks

SQL

Java

Scala

Job description

ABOUT US AND ABOUT YOU

Clinician Nexus enables health care organizations to build thriving clinician teams with industry-leading technology products, workforce and compensation analytics, and automated workflow solutions. Backed by extensive technical expertise and industry-leading data, we deliver innovative approaches to help clients to plan, educate, and engage their clinical workforce at every stage of the lifecycle. We are committed to providing our clients with outstanding guidance and support as they focus on shaping the future of health care. As a Data Platform Engineer your core responsibility revolves around crafting, advancing, and maintaining the infrastructure and systems essential for managing and optimizing data as an enterprise asset. In this capacity, you will create and maintain tooling, abstractions, and services that allow enterprise users to meticulously oversee the entire lifecycle of their data product, encompassing the efficient collection, storage, processing, and retrieval of data, all while upholding stringent standards for security, governance, and data quality. You will possess extensive expertise in working with diverse data storage technologies, including databases, data lakes, and data warehouses, as well as streaming architectures and service layer technologies. You are proficient in programming languages such as Python, Scala, Java, and SQL. Your skillset extends to encompass data integration, ETL (Extract, Transform, Load) processes, ELT (Extract, Load, Transform), Restful/GRPC services, Streaming, and the construction and abstraction of robust data pipelines. The overarching objective as a Data Platform Engineer is to enable enterprise product teams to deliver quality data products that drive business insight and value to our customers. This includes building tooling and services that connects data sources to data consumers and ensures the accessibility, structural integrity, and reliability of data to fulfill analytical and business imperatives.

PRIMARY ACCOUNTABILITIES

• Collaborate with data platform architects to design and specify platform tooling, services, and integrations
• Build data lake and lake house architectures
• Build streaming data architectures to support data platform tooling and services
• Collaborate with software teams to integrate streaming and batch architectures with software products
• Collaborate with data scientists, analysts, and business stakeholders to MVP optimal solutions
• Collaborate with data governance teams and data platform architects to build governance into technical solutions
• Maintain systems and services that provide transparency and observability into our critical systems
• Implement and promote engineering and architectural patterns, perform code reviews, and collaborate in architectural reviews
• Collaborate with product team data engineers and architects to MVP data products, maintain data models, and promote data modeling best practice
• Provide technical leadership and mentorship to product and data engineers, guiding their growth and professional development and enabling their ability to use platform tooling and services
• Lead by example through hands-on contributions to designing, coding, and troubleshooting complex data systems
• Identify and address performance bottlenecks and optimization opportunities within data pipelines, databases, and processing frameworks
• Optimize data processing workflows to improve efficiency and reduce latency
• Lead efforts to diagnose and resolve data-related incidents in a timely manner
• Will participate in the grooming of stories
• Will be responsible for their own tasking towards the completion of stories
• Will mentor junior members of the team and guide junior members on best practice
• Will participate in design
• Will be responsible for the quality of their own code and will participate in code review of others product
• Will be responsible for integrating their own code with the team's DevOps plan and implementation

KNOWLEDGE, SKILLS & ABILITIES

Minimum Required Qualifications

• 5+ years relevant experience
• Degree in Computer Science, Software Engineering, Information Systems, Information Technology or a related computer degree or equivalent experience.
• Master's degree is a plus
• Must know and have proficiency in one object or object/functional programing language. Preferably Python
• Must know common object and object/functional design patterns. Builder, factory, façade, context, etc.
• Must have proficiency with Apache Spark
• Must know data lake and lake house design principles, OLTP (Online Transaction Processing) design principles, document data stores, and graph
• Must know and understand how to build ELT/ETL patterns in a distributed compute system. Preferably Databricks
• Must know and have proficiency with common data quality tooling and the design of configurable systems to front end that tooling
• Must know and have proficiency with common systems and data observability tooling
• Must know standard practices of the Sof