Senior Backend Engineer — Distributed Data Platform

Jobtailor; New York (NY); On-site; USD 180,000 - 260,000; Full time

Job summary

Jobtailor invites an experienced data engineer to lead end-to-end data initiatives in the NY area. You will own architecture for core data assets, design large-scale pipelines, and collaborate across Product, Data Science, and Engineering to deliver high-impact outcomes.

You will mentor engineers, drive cross-team initiatives, and help shape our data products while operating in a fast-paced environment.

Qualifications

• 8+ years as a software, data, or backend engineer building scalable, production-grade systems.
• Deep experience with large-scale data processing and distributed backends; familiarity with our stack (PySpark, EMR, Hudi/Delta).
• Strong SQL skills with the ability to write and optimize complex queries over large datasets.
• Proficient Python programming and rapid ramp-up in Python if needed.
• Experience designing systems for performance, reliability and maintainability.

Responsibilities

• Act as a self-starter who drives execution independently and takes ownership with minimal direction.
• Lead high-visibility RWE projects, starting with claims data, keeping initiatives moving by unblocking teams.
• Own end-to-end architecture for critical data assets; ensure scalable, reliable solutions aligned with long-term vision.
• Design, build, and optimize large-scale data pipelines (hundreds of TBs) for performance and cost efficiency.
• Partner with Product, Data Science, and downstream engineering to align priorities and deliver high-value outcomes.
• Represent engineering in cross-functional forums, shaping roadmaps and reducing reliance on senior leadership for daily decisions.
• Develop domain expertise and mentor engineers to raise technical bar and influence data product evolution.

Skills

Python

SQL

Spark/PySpark

Distributed systems

Data processing

Tools

Git

JIRA

CircleCI

AWS EMR

Kubernetes

Airflow

Argo

Kafka

Kinesis

GitHub Copilot

Claude Code

Job description

Jobtailor invites an experienced data engineer to lead end-to-end data initiatives in the NY area. You will own architecture for core data assets, design large-scale pipelines, and collaborate across Product, Data Science, and Engineering to deliver high-impact outcomes.

You will mentor engineers, drive cross-team initiatives, and help shape our data products while operating in a fast-paced environment.