Staff Software Engineer, Data Platform Engineering

Paradigm; New York (NY, NC); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Paradigm Health is building a scalable data platform for clinical care information, ingesting data from EHRs, labs, Health Information Exchanges, and more. As Staff Engineer on the Data team, you’ll shape end-to-end data architecture, drive architectural decisions, and mentor engineers across teams.

You’ll partner with product, AI, and infra to raise data quality, governance, and security while delivering reliable batch and streaming pipelines on AWS.

Qualifications

• 10+ years of experience in data engineering or related leadership roles.
• Expertise in SQL, data modeling, and distributed systems.
• Experience with Spark or Flink for batch and streaming analytics.
• Proficiency in Python and cloud infrastructure (AWS) with IaC tooling like Terraform.
• Ability to lead cross-functional initiatives and drive architectural decisions.

Responsibilities

• Drive large cross-team initiatives within engineering from start to production.
• Own cross-team effects, break down work and drive with stakeholders.
• Architectural alignment at the company level and define best practices.
• Influence product roadmap and ensure data platform scalability and impact.
• Build and maintain robust batch and streaming data pipelines on cloud platforms.
• Mentor engineers cross-functionally and manage technical debt.
• Shape the data models and toolchain for Paradigm's data platform.

Skills

SQL

Data modeling

Distributed systems

Python

AWS

Terraform

Spark

Flink

Data mesh

AI tooling

Tools

Spark

Flink

Terraform

AWS

Kubernetes

Job description

Paradigm Health is rebuilding the clinical research ecosystem by enabling equitable access to trials for all patients. Our platform enhances trial efficiency and reduces the barriers to participation for healthcare providers. Incubated by ARCH Venture Partners and backed by leading healthcare and life sciences investors, Paradigm’s seamless infrastructure implemented at healthcare provider organizations, will bring potentially life-saving therapies to patients faster.

Our team hails from a broad range of disciplines and is committed to the company’s mission to create equitable access to clinical trials for any patient, anywhere. Join us, and bring your expertise, passion, creativity, and drive as we work together to realize this mission.

The Data Engineering team is building Paradigm’s platform for clinical care information, ingesting data from multiple sources, including Electronic Health Records, lab providers, Health Information Exchanges, and more. As Paradigm’s data needs evolve with an expanding population, research, and health system footprint, Data Engineering will scale its architecture to support downstream application, analytics and AI use cases.

As a Staff Engineer on the Data team, you’ll operate at the intersection of software, data and AI, partnering closely with product, AI and infra teams to improve the breadth, quality, and completeness of our data. We're seeking someone with experience in database technologies, distributed systems, and cloud computing, who can focus on the end-to-end technical architecture, drive operational excellence and design extensible platform abstractions to support the future growth of the company.

This is a high visibility and high impact technical leadership role. You will be working with our health systems integration, AI and business intelligence teams to maintain a high technical bar, make the right business and technology tradeoffs, manage technical debt and mentor engineers cross functionally. You will set the technical strategy and incremental plan to redesign the current legacy and messy data pipelines into a coherent modular and reusable data platform.

This role also requires someone willing to learn and dive deep into the clinical data domain and embodies customer obsession as a core leadership principle.

What you’ll do:

• Drive large cross team initiatives within engineering from start to production

• Owns cross team effects of their work, breaks down work and drives with the appropriate stakeholders

• Communicates effectively across the organization

• Architectural alignment at the company level

• Defines and champions your teams best practices to the organization

• Influences product roadmap and shapes what we should be building to be impactful for customer

• Works following the lean execution model through iterations and feedback loops to deliver business value.

• Shape the engineering vision while owning the data models and toolchain for our data platform.

• Build and maintain robust batch and streaming pipelines, leveraging cloud technologies and distributed systems.

• Develop tools and processes to deploy configurational data and self-service tools.

• Enhance the maturity of our monitoring systems to improve visibility and failure detection within our infrastructure.

• Become a subject matter expert on our diverse datasets and associated tooling.

• Ensure privacy and security in the governance of our data in collaboration with cross-functional teams.

• Advocate for a scalable, maintainable data platform that supports our evolving business needs.

• Understands business priorities / deadlines and executes on initiatives that are most impactful, while communicating to the various engineering teams your goals and set expectations correctly.

Who you are:

• 10+ Years of experience

• Curious - you look at your work beyond the surface, and care about why and how things work

• Get Stuff Done - Say what you will do, and do that. Deliver work timely and reliably.

• Growth Mindset - Believe in potential, embrace challenges, learn from mistakes.

• Teachable - Willing to learn from anyone and want to teach others

• Respectful - All viewpoints are valuable - we listen and provide appropriate feedback

• Collaborative - Prioritizes unblocking teammates, users, stakeholders

• Extensive experience driving technical projects, prioritizing work, identifying dependencies, and facilitating technical decisions across teams.

• Deep expertise in SQL, data analysis, and data modeling, with a strong understanding of distributed systems and cloud environments.

• Experience with data processing technologies like Spark or Flink - batch and microbatch analytics

• Able to evangelize and use data mesh design principles to build data products.

• Proven track record in designing systems that are scalable, available, maintainable, and performant.

• Proficiency in coding (Python or other languages) and knows when to apply AI assisted development tools

• Proficiency building infrastructure using major cloud providers e.g. AWS and infrastructure as code solutions e.g. Terraform

At Paradigm Health, we are committed to providing equal employment opportunities to all qualified individuals. We encourage and welcome candidates from all backgrounds and perspectives to apply for our open positions. We are interested in all qualified individuals and ensure that all employment decisions are based on job-related factors such as skills, experience, and qualifications.