Staff Backend Engineer – Distributed Systems, Data Platform

Jobtailor; New York (NY); On-site; USD 180,000 - 260,000; Full time

Job summary

Jobtailor invites an experienced data engineer to lead end-to-end data initiatives in the NY area. You will own architecture for core data assets, design large-scale pipelines, and collaborate across Product, Data Science, and Engineering to deliver high-impact outcomes.

You will mentor engineers, drive cross-team initiatives, and help shape our data products while operating in a fast-paced environment.

Qualifications

• 8+ years as a software, data, or backend engineer building scalable, production-grade systems.
• Deep experience with large-scale data processing and distributed backends; familiarity with our stack (PySpark, EMR, Hudi/Delta).
• Strong SQL skills with the ability to write and optimize complex queries over large datasets.
• Proficient Python programming and rapid ramp-up in Python if needed.
• Experience designing systems for performance, reliability and maintainability.

Responsibilities

• Act as a self-starter who drives execution independently and takes ownership with minimal direction.
• Lead high-visibility RWE projects, starting with claims data, keeping initiatives moving by unblocking teams.
• Own end-to-end architecture for critical data assets; ensure scalable, reliable solutions aligned with long-term vision.
• Design, build, and optimize large-scale data pipelines (hundreds of TBs) for performance and cost efficiency.
• Partner with Product, Data Science, and downstream engineering to align priorities and deliver high-value outcomes.
• Represent engineering in cross-functional forums, shaping roadmaps and reducing reliance on senior leadership for daily decisions.
• Develop domain expertise and mentor engineers to raise technical bar and influence data product evolution.

Skills

Python

SQL

Spark/PySpark

Distributed systems

Data processing

Tools

Git

JIRA

CircleCI

AWS EMR

Kubernetes

Airflow

Argo

Kafka

Kinesis

GitHub Copilot

Claude Code

Job description

Responsibilities

• Act as a self‑starter who drives execution independently, taking ownership and initiative with minimal need for day‑to‑day direction.
• Lead high‑visibility RWE projects, starting with claims data, and keep multiple initiatives moving by proactively unblocking teams.
• Own the end‑to‑end architecture for critical data assets, ensuring solutions are scalable, reliable, and aligned with H1’s long‑term vision.
• Design, build, and optimize large‑scale data pipelines (hundreds of TBs) for performance, reliability, and cost efficiency.
• Partner with Product, Data Science, and downstream engineering teams to align priorities, manage dependencies, and deliver high‑value outcomes.
• Represent engineering in cross‑functional forums, shaping roadmaps and reducing reliance on senior leadership for day‑to‑day decisions.
• Develop deep domain expertise and mentor other engineers, helping raise the technical bar and influence the evolution of our data products.

Requirements

• 8+ years as a software, data, or backend engineer building and operating scalable, production‑grade systems.
• Experience with large‑scale data processing (e.g., Spark/PySpark on EMR or similar) or scalable distributed backend systems, with the ability to quickly deepen expertise in our data stack (PySpark, EMR, Hudi/Delta).
• Strong proficiency in SQL, including writing and optimizing complex queries over large datasets.
• Strong programming experience in Python (or a modern language with the ability to quickly ramp up in Python).
• Experience designing systems or large‑scale datasets/pipelines with attention to performance, reliability, and maintainability.
• Hands‑on experience with modern engineering workflows and tooling such as Git, JIRA, and CI/CD systems (e.g., CircleCI).
• Comfort deploying and troubleshooting distributed workloads in cloud environments such as AWS EMR or Kubernetes.
• Experience with workflow orchestration or job scheduling tools (e.g., Airflow, Argo).
• Demonstrated ability to independently drive complex, cross‑team technical initiatives and influence stakeholders without formal authority.
• Experience with streaming/messaging technologies (e.g., Kafka, Kinesis) nice to have.
• Background in RWE, healthcare data, or other complex/regulated data domains is preferred.
• Experience using AI‑assisted coding tools (e.g., GitHub Copilot, Claude Code) to accelerate development while maintaining quality is encouraged.

Core Competencies

Demonstrates expertise in building and operating scalable data systems, with strong proficiency in Python and SQL. Capable of leading cross‑functional initiatives and mentoring engineers while ensuring high‑performance data pipelines and architecture.