Senior Data Engineer: AI-Driven Azure Data Pipelines

PrePass; Phoenix (AZ); On-site; USD 120,000 - 190,000; Full time

Benefits offered by this job

Medical, dental, and vision coverage

Paid time off

401(k) with employer match

Tuition reimbursement

Job summary

PrePass is seeking a Senior Data Engineer to architect and scale the core data platform. You will design high-throughput ETL pipelines, lakehouse architectures on Databricks, and mentor engineers in software craftsmanship and TDD/XP practices.

You will own production reliability, data quality, and observability, collaborating across teams while applying AI-augmented workflows to accelerate delivery in a cloud-first environment.

Qualifications

• Senior-level data engineer with distributed data pipelines and cloud data platforms experience.
• Expertise in TDD for data pipelines including unit and integration tests.
• Experience applying XP practices like pairing and refactoring.
• Strong production operations and observability expectations.
• Leadership in establishing data engineering standards and mentorship.

Responsibilities

• Build & Scale Production ETL: design and operate scalable ETL/ELT pipelines on Azure with Databricks and Delta Lake.
• Modernize Legacy Integrations: re-architect SSIS workloads into cloud-native pipelines and lead prototyping.
• Champion TDD & XP Practices: write clean, modular code with strict TDD and CI/CD pipelines.
• Drive AI-Augmented Engineering: use AI tools to accelerate coding, testing, and data-quality checks.
• Own Production Reliability & Data Quality: monitor data systems against SLAs, ensure observability and traceability.
• Elevate Engineering Standards: define and evolve data engineering standards and documentation.

Skills

Data pipelines design

Cloud data platforms

TDD & XP practices

Leadership / mentorship

Observability

Tools

Databricks

Azure

Delta Lake

Python (PySpark)

SQL

CI/CD pipelines

Job description

PrePass is seeking a Senior Data Engineer to architect and scale the core data platform. You will design high-throughput ETL pipelines, lakehouse architectures on Databricks, and mentor engineers in software craftsmanship and TDD/XP practices.

You will own production reliability, data quality, and observability, collaborating across teams while applying AI-augmented workflows to accelerate delivery in a cloud-first environment.