Senior Data Platform & Ingestion Leader

PDI Technologies; Temple (TX); Hybrid; USD 150,000 - 190,000; Full time

Benefits offered by this job

Competitive salary

Market-competitive benefits

Hybrid working arrangements

Learning & leadership development

Job summary

PDI Technologies is seeking an experienced data engineering leader to unify ingestion across batch, CDC, and streaming sources, and to own the lakehouse architecture and query layer. You will drive a team of 10+ engineers, shape the data platform, and ensure reliable, scalable pipelines with SLOs.

Role requires hands-on leadership, deep Spark/PySpark expertise, and strong collaboration with analytics and product teams. Hybrid work options available in Texas.

Qualifications

• Bachelor's degree or equivalent practical experience in CS/Engineering.
• 8+ years in data engineering with at least 3+ years in people leadership while remaining hands-on.
• Production experience with Spark Structured Streaming at scale and performance tuning in PySpark.
• Experience building or owning a reusable ingestion framework across multiple source types.
• Open table formats (Iceberg) with schema and partition evolution, compaction, and migrations.
• Experience with dbt as modeling standard including tests and CI.
• Experience with Trino/Presto operations and query optimization.
• Experience running reliable, high-scale platform systems with 24x7 on-call and fast restoration.

Responsibilities

• Lead and grow a data engineering team of 10+ engineers.
• Own the architecture of a unified ingestion framework for batch, CDC, and streaming sources.
• Maintain production Spark Structured Streaming pipelines with proper checkpointing and exactly-once sinks.
• Oversee the Apache Iceberg lakehouse, including partitioning, compaction, and evolution.
• Set dbt modeling standards, tests, contracts, and CI enforcement.
• Design and manage the Trino catalog for fast, isolated queries.
• Define and meet freshness, completeness, and latency SLOs; run 24x7 on-call rotation.
• Own cost metrics per pipeline and per dataset; optimize spend.
• Collaborate with product/analytics/architecture to sequence roadmap and run pilots.

Skills

Data engineering leadership

Spark Structured Streaming

PySpark performance

Ingestion framework design

Apache Iceberg

dbt modeling

Trino / Presto

SRE & 24x7 on-call

Education

Bachelor's degree in CS/Engineering

Tools

Apache Iceberg

Trino / Presto

dbt

Job description

PDI Technologies is seeking an experienced data engineering leader to unify ingestion across batch, CDC, and streaming sources, and to own the lakehouse architecture and query layer. You will drive a team of 10+ engineers, shape the data platform, and ensure reliable, scalable pipelines with SLOs.

Role requires hands-on leadership, deep Spark/PySpark expertise, and strong collaboration with analytics and product teams. Hybrid work options available in Texas.