Senior Data Platform Engineer: Databricks on AWS

OpenTalent; Philadelphia (Philadelphia County); On-site; USD 130,000 - 170,000; Full time

Job summary

PCI Senior Data Engineer (Databricks Platform) to design, build, and optimize scalable data pipelines on Databricks and AWS. Own data models, ETL/ELT workflows, and governance with a focus on reliability and performance.

You will collaborate with the Data Platform Lead and cross-functional teams to deliver analytics, reporting, and AI/ML use cases, applying Delta Lake architecture and 3NF/star schema modelling. Experience with AWS services and CI/CD is a plus.

Qualifications

• Experience designing scalable data pipelines on Databricks.
• Proficiency with PySpark and Databricks SQL.
• Knowledge of medallion architecture (Bronze/Silver/Gold).
• Experience with data governance and data quality.
• Collaborative, able to work with cross-functional teams.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks (Spark / Delta Lake).
• Build robust ETL/ELT workflows with PySpark and Databricks SQL.
• Implement and support medallion architecture (Bronze, Silver, Gold).
• Design efficient data models (3NF, star schema).
• Support Data Catalog and Data Governance for quality and compliance.
• Build and maintain CI/CD pipelines for reliable deployments.
• Collaborate with external partners for development and production support.

Skills

Databricks

PySpark

Delta Lake

Data Modeling

CI/CD

AWS

Data Governance

Tools

AWS Glue

S3

Lambda

Job description

PCI Senior Data Engineer (Databricks Platform) to design, build, and optimize scalable data pipelines on Databricks and AWS. Own data models, ETL/ELT workflows, and governance with a focus on reliability and performance.

You will collaborate with the Data Platform Lead and cross-functional teams to deliver analytics, reporting, and AI/ML use cases, applying Delta Lake architecture and 3NF/star schema modelling. Experience with AWS services and CI/CD is a plus.