Staff Data Engineer - Cloud Spark Architect (Remote)

H1; New York (NY); On-site; USD 170,000 - 190,000; Full time

Benefits offered by this job

Full suite of health insurance options

Generous paid time off

Flexible work hours

Stock options

Job summary

H1 in New York is seeking a Staff Data Engineer to shape their healthcare entity resolution platform. You will lead a small team, ensuring scalability and optimization of the data architecture.

The ideal candidate has strong expertise in distributed data systems, particularly with Apache Spark, and will work closely with AI/ML teams. A competitive salary and a full suite of health benefits are provided.

Qualifications

• 8+ years of experience in large-scale distributed data systems.
• Strong proficiency in Python, Scala, or Java for distributed processing.
• Experience with Apache Spark and AWS big data technologies.

Responsibilities

• Lead design of distributed Spark pipelines for entity resolution.
• Own systems supporting automatching and identity mapping.
• Collaborate with Product and AI/ML teams for platform improvements.

Skills

Distributed data processing frameworks

Apache Spark

Python (PySpark)

Cloud-native environments

ETL/ELT frameworks

Kafka

Docker

Kubernetes

Tools

AWS

Terraform

Argo

Job description

H1 in New York is seeking a Staff Data Engineer to shape their healthcare entity resolution platform. You will lead a small team, ensuring scalability and optimization of the data architecture.

The ideal candidate has strong expertise in distributed data systems, particularly with Apache Spark, and will work closely with AI/ML teams. A competitive salary and a full suite of health benefits are provided.