Lead Big Data Engineer — Spark, Hadoop, Hybrid

EXL; Dallas (TX); Hybrid; USD 120,000 - 180,000; Full time

Job summary

EXL is seeking an experienced Data Engineering Lead to drive architecture decisions and mentor distributed teams. You will own end-to-end data pipelines, focusing on on-prem Hadoop ecosystems, Spark processing, and effective scheduling with CA7/Control-M.

The role requires strong SQL, ETL capabilities, and a passion for AI/ML concepts, with a focus on scalable data platforms that accelerate business insights.

Qualifications

• 7-10 years of experience in data engineering and technical leadership.
• Hands-on Hadoop ecosystem (Hadoop, Hive, Impala, HDFS) in on-prem environments.
• Solid Apache Spark experience with large-scale jobs.
• Hands-on with job scheduling/orchestration (CA7 or Control-M).
• Strong SQL, ETL skills and data architecture ability.
• Interest in AI/ML concepts and willingness to learn.

Responsibilities

• Lead design and architecture decisions for data platforms and pipelines.
• Mentor and guide development teams through implementation and best practices.
• Own end-to-end data engineering pipelines from ingestion to consumption.
• Collaborate with analytics and product teams to optimize data flows and performance.

Skills

Data engineering

Big Data / Hadoop

Apache Spark

ETL

SQL

Distributed systems

Mentoring

AI/ML awareness

Tools

CA7/Control-M

Job description

EXL is seeking an experienced Data Engineering Lead to drive architecture decisions and mentor distributed teams. You will own end-to-end data pipelines, focusing on on-prem Hadoop ecosystems, Spark processing, and effective scheduling with CA7/Control-M.

The role requires strong SQL, ETL capabilities, and a passion for AI/ML concepts, with a focus on scalable data platforms that accelerate business insights.