Senior Data Engineer (33142)

United States Digital Space LLC; United States; Hybrid; USD 120,000 - 170,000; Full time

Benefits offered by this job

Hybrid/remote work options

Excellent benefits package

Job summary

PANTHERx Rare Pharmacy seeks a Senior Data Engineer to own the EDW and core data pipelines on Databricks, implementing Bronze-Silver-Gold layers and event-driven ingestion. You’ll drive architecture, governance and reliability while mentoring engineers and partnering with QA and Data Governance.

This role supports a hybrid home/office work arrangement in the Pittsburgh area, with emphasis on scalable, low-latency data platforms and production-grade data services across the enterprise.

Qualifications

• 6+ years of progressive data engineering experience with production ownership.
• Hands-on Databricks experience in production: Delta Lake and medallion architecture.
• Advanced SQL with PySpark and Python to build complex transformations.
• Experience with Azure data services and CI/CD in Azure DevOps.

Responsibilities

• Design, build, and operate enterprise data warehouse pipelines in the Databricks medallion architecture.
• Own production systems, manage data model integrity and SLAs.
• Develop Delta Lake pipelines and event-driven ingestion patterns.
• Document decisions and maintain runbooks; mentor engineers.

Skills

Databricks

Delta Lake

Unity Catalog

PySpark

Python

SQL

Azure

Data Modeling

Documentation

Leadership/Mentoring

Education

Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field, or equivalent experience

Tools

Azure Databricks

Azure Data Lake Storage

Azure Data Factory

CI/CD in Azure DevOps

Job description

*7,000 Diseases - 500 Treatments - 1 Rare Pharmacy*

*PANTHERx is the nation’s largest rare disease pharmacy, and we put the patient experience at the top of everything that we do.*

*If you are looking for a career in the healthcare field that embraces authentic dedication to patient care, you don’t need to look beyond PANTHERx. In every line of service, in every position and area of expertise, PANTHERx associates are driven to provide the highest quality outcomes for our patients.*

*We are seeking team members who:*

*Are inspired and compassionate problem solvers;*

*Produce high quality work;*

*Thrive in the excitement of the ever-challenging environment of modern medicine; and*

*Are committed to achieving superior health outcomes for people living with rare and devastating diseases.*

*At PANTHERx, we know our employees are the driving force in what we do. We cultivate talent and encourage growth within PANTHERx so that our associates can continue to explore their interests and expand their careers. Guided by our mission to provide uncompromising quality every day, we continue our strategic growth to further reach those affected by rare diseases.*

*Join the PANTHERx team, and define your own RxARE future in healthcare!*

Location: Pittsburgh, PA (Hybrid)Classification: ExemptStatus: Full-TimeReports to: Manager, Data EngineeringPurpose

The Senior Data Engineer builds, operates, and owns the enterprise data warehouse and core data pipelines on PANTHERx’s Databricks platform. This role is responsible for EDW architecture, ingress patterns into Bronze, transformation logic from Bronze to Silver, and Gold harmonization rules. The Senior Data Engineer significantly contributes to the platform’s evolution to a simplified medallion architecture with event-driven, low-latency ingestion. This role serves as a senior practitioner within the Data Engineering track, setting the technical bar for pipeline quality, documentation discipline, and reliability, as well as mentoring Data Engineers as the team scales.

ResponsibilitiesEDW Engineering \& Ownership

• Designs, builds, and operates enterprise data warehouse pipelines and data models across the Databricks medallion architecture (Bronze, Silver, Gold), including Silver conformance logic and Gold harmonization and survivorship rules.
• Assumes named ownership of assigned EDW domains as production systems: data model integrity, pipeline reliability, ingestion SLAs, and incident response.
• Leads structured documentation of knowledge for assigned domains, absorbing architecture, transformation logic, and feed SQL.
• Executes data model simplification work, consolidating legacy transformation layers into the target architecture with validated output parity.

Pipeline Development \& Streaming

• Develops and maintains production Delta Lake pipelines, evolving ingestion from batch watermark patterns to event-driven architectures using Zerobus Ingestion and Spark Declarative Pipelines for high-SLA source systems.
• Implements row-level reconciliation, validation checkpoints, and quality gates at ingestion, transformation, and delivery layers, operationalizing governance-defined data quality dimensions in partnership with QA.
• Builds Gold-layer tables and promotion logic that support Unity Catalog metric views, lineage capture, and access controls in partnership with Analytics Engineering and Data Governance.
• Optimizes pipeline performance, latency, and compute cost across all layers of the platform.

Documentation, Quality \& Observability

• Documents architecture decisions, data models, pipeline logic, and operational runbooks to team standards, ensuring no critical platform capability depends on a single point of failure.
• Operates pipeline observability tooling: monitors anomaly detection, triages data reliability incidents, and drives root cause remediation for assigned domains.
• Implements data contract validation (ODCS or equivalent) in pipelines supporting external partner feed domains, ensuring contract failures halt delivery before reaching consumers.

Mentorship \& Cross-Functional Collaboration

• Mentors Data Engineers on Databricks development patterns, SQL and PySpark craft, and documentation discipline; reviews pull requests in Azure DevOps \& Git to maintain code quality.
• Partners with Data Governance on lineage capture and metadata standards, with QA on Tier 1 and Tier 2 pipeline test suite development, and with Partner Data Services on feed engineering requiring pipeline work.
• Works from structured requirements and acceptance criteria entering through the Informatics intake process, and flags requirements gaps before build work begins.

Required Qualifications

• 6+ years of progressive data engineering experience, including production ownership of an enterprise data warehouse or large-scale transformation pipelines with defined SLAs.
• Deep, hands-on expertise with Databricks in production environments: Delta Lake, medallion architecture, Unity Catalog, and pipeline performance optimization.
• Demonstrated data warehousing depth: dimensional and harmonized data modeling, conformance and survivorship logic, and operating a warehouse as a production system.
• Advanced SQL and strong PySpark and Python proficiency, sufficient to build, review, and optimize complex transformation logic independently.
• Proven experience with Azure data services: Azure Databricks, Azure Data Lake Storage, Azure Data Factory, and CI/CD practices in Azure DevOps.
• Demonstrated ability to absorb complex, under-documented systems through structured knowledge transfer and produce documentation that makes that knowledge durable and transferable.
• Strong communication and collaboration skills; able to work directly with QA, Governance, Informatics, and business-facing teams without an intermediary.

Preferred Qualifications

• Healthcare, specialty pharmacy, or regulated industry experience, with exposure to clinical or operational data environments.
• Production experience with event-driven ingestion: Azure Event Hubs, Kafka, or equivalent, consumed via Structured Streaming.
• Familiarity with data contract standards (ODCS or equivalent) and pipeline observability platforms (Monte Carlo or similar).
• Familiarity with enterprise data catalog tooling (Atlan, Collibra, or equivalent) and designing pipelines as catalogued, governed assets.
• Experience leading through influence: mentoring engineers, setting standards, and driving adoption without formal management authority.
• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field, or equivalent experience.

Work Environment

This position works in a home office and professional office environment. When in-office this role routinely uses standard office equipment such as computers, phones, photocopiers, filing cabinets and fax machines, and communications via MS Teams.

Physical Demands

While performing the duties of this job, the employee is regularly required to sit, see, talk or hear. The employee frequently is required to stand; walk; use hands and fingers to handle or feel; and reach with hands and arms. Visual acuity is necessary for tasks such as reading and working with various forms of data on a screen. Reasonable accommodation may be made to enable individuals with disabilities to perform

Benefits:

Hybrid, remote and flexible on-site work schedules are available, based on the position. PANTHERx Rare Pharmacy also affords an excellent benefit package, including but not limited to medical, dental, vision, health savings and flexible spending accounts, 401K with employer matching, employer-paid life insurance and short/long term disability coverage, and an Employee Assistance Program! Generous paid time off is also available to all full-time employees, as well as limited paid time off for part-time employees. Of course we offer paid holidays too!

Equal Opportunity:

PANTHERx Rare Pharmacy is an equal opportunity employer, and does not discriminate in recruiting, hiring, promotions or any term or condition of employment based on race, age, religion, gender, ethnicity, sexual orientation, gender identity, disability, protected veteran's status, or any other characteristic protected by federal, state or local laws.