Lead Data Engineer: Spark & Data Pipelines Architect

Prevalent Ai India; United States; À distance; USD 150 000 - 210 000; Plein temps

Résumé du poste

Prevalent AI in the United States is seeking a Lead Data Engineer to design and deliver advanced data pipelines for clients. You will write code, review teammates' work, and help ensure production performance on large-scale workloads.

The role emphasizes deep Spark internals, time-series data, and building reusable, configuration-driven components, with mentorship of junior engineers and collaboration with client teams.

Qualifications

• 10+ years in data engineering with hands-on PySpark on large-volume workloads.
• Deep understanding of Spark internals: execution model, stages and tasks, shuffle, partitioning, joins, and memory behavior.
• Experience tuning long-running jobs to meet SLAs and handling data skew.
• Hands-on experience with time-series data at scale: event-time semantics and windowing.
• Strong Delta Lake or Iceberg experience including partitioning and data pruning.

Responsabilités

• Lead the design and delivery of client data engineering projects from requirement to production.
• Design pipelines that meet SLA and scale with data growth.
• Diagnose failures and performance issues using logs and metrics.
• Set data layout and storage strategy to optimize reads.
• Build reusable, configuration-driven components to reduce development effort.
• Review team code and establish coding standards.
• Collaborate with client teams to define requirements and data structures.
• Contribute to platform work around delivery, including object storage and CI/CD.
• Mentor developers and document design decisions and runbooks.

Connaissances

PySpark

Spark internals

Delta Lake

Iceberg

Performance tuning

Time-series data

Kubernetes

Python

Code reviews

Outils

S3

Spark UI

Description du poste

Prevalent AI in the United States is seeking a Lead Data Engineer to design and deliver advanced data pipelines for clients. You will write code, review teammates' work, and help ensure production performance on large-scale workloads.

The role emphasizes deep Spark internals, time-series data, and building reusable, configuration-driven components, with mentorship of junior engineers and collaboration with client teams.