Senior Data Engineer: Build Scalable AI-Driven Pipelines

PrePass Ag; Phoenix (AZ); On-site; USD 120,000 - 180,000; Full time

Job summary

PrePass is seeking a Senior Data Engineer to architect, build, and scale our core data platform. You will lead high-throughput data pipelines and lakehouse architectures, mentoring engineers in modern software craftsmanship with TDD and XP practices, and applying AI-assisted workflows to deliver reliable data systems in Databricks.

You will own production reliability, data quality, and observability, driving cost-effective performance and scalable solutions while elevating engineering standards

Qualifications

• Senior-level data engineering experience with distributed pipelines and cloud data platforms.
• Hands-on expertise in TDD for data pipelines with unit/integration tests.
• Experience integrating AI coding assistants to accelerate delivery and testing.
• Production operations/observability with uptime and SLAs.
• Technical leadership, architectural reviews, and mentorship in Agile teams.

Responsibilities

• Design, build, and scale core data pipelines and lakehouse architectures.
• Modernize legacy data integrations into cloud-native pipelines.
• Lead by example in TDD/XP practices and CI/CD.
• Leverage AI augmentation to accelerate code generation and testing.
• Own production reliability, data quality, and observability.
• Evolve data engineering standards and documentation within Agile teams.

Skills

Data pipelines

Azure

SQL

Apache Spark

Databricks

TDD

XP

AI augmentation

Observability

Mentorship

Tools

Terraform

Bicep

CI/CD

Databricks Workflows

Unity Catalog

Job description

About PrePass

PrePass® is North America's most trusted weigh station bypass and toll management platform. We’re transforming how the transportation industry operates—creating solutions that keep trucks moving safely, efficiently, and compliantly. This means making bold decisions and building systems that support not only fleets but the broader economy. It all starts with enabling commercial vehicles to keep rolling with seamless toll management, weigh station bypass, and safety solutions. It’s what we do best, and we do it to meet the demands of the road every day.

That’s why people join us: our solutions are implemented in real-time, on highways and interstates across the nation, helping fleets go farther, faster. This work challenges and rewards, presenting complex problems that need ambitious answers. We hire bold thinkers with a heart for impact, a passion for progress, and the optimism to shape the future of transportation.

About the Role

PrePass is seeking a Senior Data Engineer to architect, build, and scale our core data platform. In this role, you will lead the design of high-throughput data pipelines and lakehouse architectures while mentoring fellow engineers in modern software craftsmanship. We are looking for a technical leader who treats data engineering with the rigor of software engineering—relying on strict Test-Driven Development (TDD), extreme programming (XP) practices, and AI-augmented workflows to deliver reliable, observable, and mission-critical data systems in Databricks.

Key Responsibilities

• Build & Scale Production ETL: Design, build, and operate scalable ETL/ELT pipelines on Azure that ingest, transform, and integrate data from multiple sources. Use advanced SQL, Apache Spark, and Lakehouse technologies such as Databricks and Delta Lake to support analytics, reporting, and real-time operational workloads.
• Modernize Legacy Integrations: Partner with the team to define and execute a strategy for re-architecting legacy SQL Server Integration Services (SSIS) workloads into modern, cloud-native pipelines. Lead proof-of-concept and prototyping work and turn the findings into clear recommendations.
• Champion TDD & XP Practices: Lead by example in writing clean, modular, and maintainable code using strict Test-Driven Development (TDD), pair programming, continuous refactoring, and automated CI/CD pipelines.
• Drive AI-Augmented Engineering: Leverage AI-assisted development tools to accelerate code generation, automate unit and data-quality test creation, streamline legacy refactoring, and enhance pipeline anomaly detection.
• Own Production Reliability & Data Quality: Operate and monitor mission-critical data systems against strict SLAs for data freshness, uptime, and reliability. Diagnose and resolve integration and data-quality failures, build end-to-end traceability and observability, and tune pipeline and compute performance for cost and scalability.
• Elevate Engineering Standards: Define and evolve team-wide data engineering standards, architectural documentation, and continuous improvement initiatives within an Agile environment.

Qualifications

• Senior-Level Data Engineering Experience: Proven track record of designing, building, and operating distributed data pipelines and cloud data platforms at scale.
• Mastery of Test-Driven Development (TDD): Demonstrated hands-on expertise in TDD for data pipelines—writing unit, integration, and data-contract tests prior to implementation—alongside XP practices such as pairing, refactoring, and continuous integration.
• AI Augmentation Proficiency: Practical experience integrating AI coding assistants and LLM-based tooling into daily engineering workflows to accelerate delivery, improve test coverage, and automate documentation.
• Production Operations & Observability: Strong experience operating high-availability systems in production with clear uptime, reliability, and observability expectations.
• Technical Leadership: Experience driving engineering standards, architectural reviews, documentation, and mentorship within an Agile team.

Bonus Points For

• Experience in Data Science or Machine Learning, particularly in MLOps, model deployment, Feature Store management, or feature engineering on Databricks.
• Experience with Infrastructure as Code (e.g., Terraform / Bicep) for managing Databricks workspaces and cloud resources.
• Familiarity of event driven architecture – queues, batches, and pub/sub models.
• Databricks Expertise:Production experience with the Databricks platform (Delta Lake, PySpark/Spark SQL, Databricks Workflows, and Unity Catalog)
• Background in high-volume IoT, telematics, or transportation industry data.

How We Will Take Care of You

• Robust benefit package that includes medical, dental, and vision that start on date of hire.
• Paid Time Off, to include vacation, sick, holidays, and floating holidays.
• 401(k) plan with employer match.
• Company-funded “lifestyle account” upon date of hire for you to apply toward your physical and mental well-being (i.e., ski passes, retreats, gym memberships).
• Tuition Reimbursement Program.
• Voluntary benefits, to include but not limited to Legal and Pet Discounts.
• Employee Assistance Program (available at no cost to you).
• Company-sponsored and funded “Culture Team” that focuses on the Physical, Mental, and Professional well-being of employees.
• Community Give-Back initiatives.
• Culture that focuses on employee development initiatives.

Join Us

At PrePass, our mission drives us.

We invest in relationships. We challenge ourselves to innovate and improve. We win together. Simply put, we live our Core Values.

Ready to help move the transportation industry forward? Join us and let’s drive progress—together.