Senior Data Engineer - Databricks & AWS Platform

PCI Pharma Services; Rockford (IL); On-site; USD 120,000 - 180,000; Full time

Job summary

PCI Pharma Services is seeking a Senior Data Engineer to lead hands-on design, development, and operations for our Databricks-based data platform on AWS. You will ensure scalable, secure data pipelines and models powering analytics, reporting, and AI/ML initiatives.

The role requires deep expertise in distributed data processing, Delta Lake architecture, data modeling, and ETL/ELT frameworks, with collaboration across Data Platform Lead, Product, Testing, and external partners.

Qualifications

• Extensive experience building scalable data pipelines on Databricks and AWS.
• Deep knowledge of Delta Lake, data modeling, and data governance.
• Strong ETL/ELT design and optimization skills.
• Experience collaborating with cross-functional teams including Product and QA.
• Familiarity with UAT/testing support and production readiness.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks (Spark / Delta Lake).
• Build ETL/ELT workflows with PySpark and Databricks SQL.
• Implement medallion architecture (Bronze, Silver, Gold).
• Design data models (3NF, star schema).
• Support Data Catalog and Governance frameworks to ensure quality and compliance.
• Build and maintain CI/CD pipelines for deployments.
• Collaborate with external partners for production support.

Skills

Databricks Platform

AWS

PySpark

Data Modeling

ETL/ELT

CI/CD

3NF/Star Schema

Data Governance

Python

Tools

AWS S3

AWS Glue

Delta Lake

CI/CD Pipelines

Lambda

Job description

PCI Pharma Services is seeking a Senior Data Engineer to lead hands-on design, development, and operations for our Databricks-based data platform on AWS. You will ensure scalable, secure data pipelines and models powering analytics, reporting, and AI/ML initiatives.

The role requires deep expertise in distributed data processing, Delta Lake architecture, data modeling, and ETL/ELT frameworks, with collaboration across Data Platform Lead, Product, Testing, and external partners.