Senior Data Engineer - Databricks & AWS for Life Sciences

PCI Pharma Services; Philadelphia (Philadelphia County); On-site; USD 120,000 - 180,000; Full time

Job summary

PCI Pharma Services is seeking a Senior Data Engineer (Databricks Platform) to design, develop, and optimize our enterprise data platform on Databricks and AWS. You will build scalable data pipelines and models powering analytics, reporting, and AI/ML use cases across the organization.

The role requires hands-on implementation of medallion architecture, data modeling, and governance, plus collaboration with Data Platform Lead, Product, Testing, and DevOps to ensure reliability and security.

Qualifications

• Hands-on experience building data platforms on Databricks and AWS.
• Experience with data governance and data catalog concepts.
• Strong collaboration with product, QA, and DevOps teams.
• Experience with CI/CD for data pipelines.
• Knowledge of medallion architecture is a plus.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks (Spark / Delta Lake).
• Build robust ETL/ELT workflows leveraging PySpark and Databricks SQL.
• Implement and support medallion architecture (Bronze, Silver, Gold layers).
• Design and maintain efficient data models, including 3NF, star schema, and canonical models.
• Support and maintain Data Catalog and Data Governance frameworks to ensure data quality and compliance.
• Build and maintain CI/CD pipelines to enable reliable deployment across environments.
• Collaborate with external partners for development activities and ongoing production support.
• Demonstrate hands-on experience with AWS Glue, S3, and Lambda.
• Provide operational support for AWS-hosted infrastructure, including application services, data services, and system integrations.
• Monitor cloud environments for availability, performance, security, and cost efficiency.
• Support deployment activities and ensure environment readiness in collaboration with DevOps and technical stakeholders.
• Contribute to cloud operational best practices focused on scalability, resilience, security, and incident response.
• Partner with Testing and QA teams to support UAT and validation activities by reviewing test cases, triaging defects, and ensuring test environment readiness.
• Collaborate closely with Product Managers, Project Managers, QA, DevOps, and external development partners to ensure successful solution delivery.
• Translate business requirements into scalable technical solutions aligned with architecture and standards.
• Ensure adherence to PCI, cGMP, GCP, and regulatory policies and support occasional overtime for critical releases.

Skills

Databricks Platform

PySpark

Delta Lake

AWS (Glue, S3, Lambda)

ETL/ELT design

Data Modeling

CI/CD for data

Data Governance

Tools

Databricks

Spark

Delta Lake

AWS Glue

S3

AWS Lambda

Job description

PCI Pharma Services is seeking a Senior Data Engineer (Databricks Platform) to design, develop, and optimize our enterprise data platform on Databricks and AWS. You will build scalable data pipelines and models powering analytics, reporting, and AI/ML use cases across the organization.

The role requires hands-on implementation of medallion architecture, data modeling, and governance, plus collaboration with Data Platform Lead, Product, Testing, and DevOps to ensure reliability and security.