Digital Platforms, Senior Data Engineer - PCI Pharma Services

OpenTalent; Philadelphia (Philadelphia County); On-site; USD 130,000 - 170,000; Full time

Job summary

PCI Senior Data Engineer (Databricks Platform) to design, build, and optimize scalable data pipelines on Databricks and AWS. Own data models, ETL/ELT workflows, and governance with a focus on reliability and performance.

You will collaborate with the Data Platform Lead and cross-functional teams to deliver analytics, reporting, and AI/ML use cases, applying Delta Lake architecture and 3NF/star schema modelling. Experience with AWS services and CI/CD is a plus.

Qualifications

• Experience designing scalable data pipelines on Databricks.
• Proficiency with PySpark and Databricks SQL.
• Knowledge of medallion architecture (Bronze/Silver/Gold).
• Experience with data governance and data quality.
• Collaborative, able to work with cross-functional teams.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks (Spark / Delta Lake).
• Build robust ETL/ELT workflows with PySpark and Databricks SQL.
• Implement and support medallion architecture (Bronze, Silver, Gold).
• Design efficient data models (3NF, star schema).
• Support Data Catalog and Data Governance for quality and compliance.
• Build and maintain CI/CD pipelines for reliable deployments.
• Collaborate with external partners for development and production support.

Skills

Databricks

PySpark

Delta Lake

Data Modeling

CI/CD

AWS

Data Governance

Tools

AWS Glue

S3

Lambda

Job description

Life changing therapies. Global impact. Bridge to thousands of biopharma companies and their patients.

We are PCI.

Our investment is in People who make an impact, drive progress and create a better tomorrow. Our strategy includes building teams across our global network to pioneer and shape the future of PCI.

Summary: The Senior Data Engineer (Databricks Platform) is a senior individual contributor responsible for the hands-on design, development, optimization, and operational support of the enterprise data platform built on Databricks and AWS technologies. This role ensures scalable, secure, and high-performance data pipelines, models, and integrations that power analytics, reporting, and advanced AI/ML use cases across the organization.

The role brings deep expertise in distributed data processing, Delta Lake architecture, data modeling, ETL/ELT frameworks, and cloud environments (AWS). This role focuses on execution excellence, platform reliability, and technical problem resolution, while collaborating closely with Data Platform Lead, Product, Project Management, Testing, Customer Success, and external development partners.

Responsibilities:

Data Platform Engineering

• Design, develop, and maintain scalable data pipelines using Databricks (Spark / Delta Lake).
• Build robust ETL/ELT workflows leveraging PySpark and Databricks SQL.
• Implement and support medallion architecture (Bronze, Silver, Gold layers).
• Design and maintain efficient data models, including 3NF, star schema, and canonical models, as appropriate.
• Support and maintain Data Catalog and Data Governance frameworks to ensure data quality and compliance.
• Build and maintain CI/CD pipelines to enable reliable deployment across environments.
• Collaborate with external partners for development activities and ongoing production support.

Cloud & AWS Environment Support

• Demonstrate hands-on experience with AWS Glue, S3, and Lambda.
• Provide operational support for AWS-hosted infrastructure, including application services, data services, and system integrations.
• Monitor cloud environments for availability, performance, security, and cost efficiency; elevate risks and issues to the Technical Lead when required.
• Support deployment activities and ensure environment readiness in collaboration with DevOps and technical stakeholders.
• Contribute to cloud operational best practices focused on scalability, resilience, security, and incident response.

Testing Support

• Partner with Testing and QA teams to support User Acceptance Testing (UAT) and validation activities by:
• Reviewing test cases for technical accuracy and risk coverage
• Triaging defects identified during testing cycles
• Ensuring test environment readiness and stability