Lead Data Engineer - Big Data Solutions

Placements24; Riverton (UT); Hybrid; USD 61,000 - 85,000; Full time

Job summary

Placements24 is seeking a Lead Data Engineer to spearhead big data solutions, data pipelines, and data warehousing. The role involves directing a team, setting best practices, and ensuring scalable data platforms that support analytics and ML initiatives.

The successful candidate will lead design and implementation, collaborate with stakeholders, and drive governance and performance improvements in a hybrid work setting based in Upington, Northern Cape.

Qualifications

• 7+ years in data engineering, with at least 2 years in a lead role.
• Experience with Hadoop, Spark, Kafka, and Flink.
• Proficient in Python, Scala, or Java for data processing.
• Strong SQL and experience with relational and NoSQL databases.
• Cloud experience (AWS, Azure, GCP) and big data services.

Responsibilities

• Lead the design, development, and implementation of scalable big data solutions and data warehousing systems.
• Build and maintain ETL/ELT pipelines and manage data lakes and warehouses.
• Mentor a team of data engineers and ensure data quality, security, and governance.
• Collaborate with data scientists, analysts, and stakeholders to deliver solutions.
• Evaluate and adopt new big data technologies and optimize data processing performance.

Skills

Python

Scala

Java

SQL

Leadership

Communication

Education

Bachelor's or Master's in CS/Engineering/related field

Tools

Hadoop

Spark

Kafka

Flink

Job description

About the Role

Our client, a leader in data analytics and business intelligence, is seeking a visionary Lead Data Engineer to spearhead their Big Data Solutions initiatives. This pivotal role is based in Upington, Northern Cape , and involves designing, building, and maintaining robust data pipelines and infrastructure to support advanced analytics and machine learning capabilities. You will lead a team of data engineers, define best practices, and ensure the efficient and reliable flow of data across the organization. The position offers a hybrid work arrangement, combining on-site collaboration with remote flexibility. We are looking for an experienced leader passionate about leveraging data to drive business insights and innovation.

Key Responsibilities

Lead the design, development, and implementation of scalable big data solutions and data warehousing systems. Build and maintain efficient, reliable, and robust ETL/ELT data pipelines using various big data technologies. Oversee the architecture and implementation of data lakes and data warehouses. Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions. Mentor and guide a team of data engineers, fostering technical excellence and professional development. Ensure data quality, integrity, and security across all data platforms. Optimize data processing performance and data retrieval efficiency. Evaluate and adopt new big data technologies and tools. Develop and enforce data governance policies and best practices. Troubleshoot and resolve complex data pipeline and infrastructure issues.

Requirements

Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field. 7+ years of experience in data engineering, with at least 2 years in a lead or senior capacity. Extensive experience with big data technologies such as Hadoop, Spark, Kafka, and Flink. Proficiency in programming languages like Python, Scala, or Java for data processing. Strong SQL skills and experience with relational and NoSQL databases. Experience with cloud platforms (AWS, Azure, GCP) and their big data services (e.g., S3, Redshift, BigQuery, Databricks). Familiarity with data warehousing concepts and dimensional modeling. Experience with data pipeline orchestration tools (e.g., Airflow, Luigi). Excellent leadership, communication, and problem-solving skills. Experience working in the Upington, Northern Cape region or willingness to work a hybrid schedule is preferred.

Benefits

Our client offers a highly competitive salary, performance-based bonuses, a comprehensive health benefits package, and significant opportunities for professional growth and skill development in the rapidly evolving field of big data. The hybrid work model supports work-life balance. Employees will have the chance to work with cutting-edge technologies, lead impactful data initiatives, and contribute to data-driven decision-making within a forward-thinking organization.