Lead Data Engineer: Big Data & ML Pipelines (Hybrid)

Placements24; Riverton (UT); Hybrid; USD 61,000 - 85,000; Full time

Job summary

Placements24 is seeking a Lead Data Engineer to spearhead big data solutions, data pipelines, and data warehousing. The role involves directing a team, setting best practices, and ensuring scalable data platforms that support analytics and ML initiatives.

The successful candidate will lead design and implementation, collaborate with stakeholders, and drive governance and performance improvements in a hybrid work setting based in Upington, Northern Cape.

Qualifications

• 7+ years in data engineering, with at least 2 years in a lead role.
• Experience with Hadoop, Spark, Kafka, and Flink.
• Proficient in Python, Scala, or Java for data processing.
• Strong SQL and experience with relational and NoSQL databases.
• Cloud experience (AWS, Azure, GCP) and big data services.

Responsibilities

• Lead the design, development, and implementation of scalable big data solutions and data warehousing systems.
• Build and maintain ETL/ELT pipelines and manage data lakes and warehouses.
• Mentor a team of data engineers and ensure data quality, security, and governance.
• Collaborate with data scientists, analysts, and stakeholders to deliver solutions.
• Evaluate and adopt new big data technologies and optimize data processing performance.

Skills

Python

Scala

Java

SQL

Leadership

Communication

Education

Bachelor's or Master's in CS/Engineering/related field

Tools

Hadoop

Spark

Kafka

Flink

Job description

Placements24 is seeking a Lead Data Engineer to spearhead big data solutions, data pipelines, and data warehousing. The role involves directing a team, setting best practices, and ensuring scalable data platforms that support analytics and ML initiatives.

The successful candidate will lead design and implementation, collaborate with stakeholders, and drive governance and performance improvements in a hybrid work setting based in Upington, Northern Cape.