Lead Data Engineer (Big Data Specialist)

Placements24; Gauteng; Hybrid; ZAR 1,996,000 - 2,995,000; Full time

Benefits offered by this job

Competitive compensation

Full remote work

Learning and development opportunities

Stock options or equity (if offered)

Access to latest Big Data tools

Job summary

Placements24 is seeking a visionary Lead Data Engineer for a cutting-edge data analytics firm. This fully remote role focuses on architecting, building, and maintaining scalable data pipelines and infrastructure to enable advanced analytics and ML initiatives.

You will lead a team of data engineers, driving innovation across the data platform. You will work with Spark, Hadoop, Kafka, and cloud platforms, mentor teammates, ensure data quality and security, and collaborate with data scientists and

Qualifications

• Master's or Bachelor's degree in CS, Eng, or related quantitative field.
• 5+ years in data engineering with Big Data focus.
• Experience designing data pipelines with Spark.
• Knowledge of Hadoop, Hive, HDFS, Kafka.
• Proficient in Python, Scala or Java; strong SQL and DB skills.
• Experience with cloud data platforms (AWS, Azure, GCP).
• Familiar with data warehousing concepts and data modelling.
• Excellent leadership, problem-solving, and communication skills.
• Ability to work remotely and collaborate across disciplines.

Responsibilities

• Lead design, development, and optimization of large-scale data pipelines.
• Build scalable, fault-tolerant data infrastructure using Spark, Hadoop, Kafka.
• Develop data warehousing and data lakes solutions.
• Implement ETL/ELT processes from diverse sources.
• Ensure data quality, integrity, and security across systems.
• Collaborate with data scientists, analysts, and software engineers.
• Mentor junior data engineers and promote best practices.
• Automate data processes and monitor system performance.
• Stay current with emerging Big Data technologies and practices.

Skills

Leadership

Remote collaboration

Communication

Problem solving

Mentorship

Education

Master's degree in Computer Science / Engineering

Bachelor's degree in Computer Science / Engineering

Tools

Apache Spark

Hadoop ecosystem

Kafka

Python

Scala

Java

SQL

Relational & NoSQL databases

AWS

Azure

GCP

Job description

About the Role

Our client, a cutting-edge data analytics firm, is seeking a visionary Lead Data Engineer with specialized expertise in Big Data technologies to join their distributed team. This pivotal role involves architecting, building, and maintaining scalable data pipelines and infrastructure that enable advanced analytics and machine learning initiatives. You will lead a team of talented data engineers, drive innovation in data processing, and ensure the reliability and efficiency of our client's data ecosystem. This is a fully remote position, offering the ultimate flexibility to contribute to critical projects from anywhere in the world. If you are passionate about harnessing the power of Big Data and have a proven track record of building robust data solutions, this is an exceptional opportunity to shape the future of data engineering.

Key Responsibilities

• Lead the design, development, and optimization of large-scale batch and real-time data pipelines.
• Build and manage robust, scalable, and fault-tolerant data infrastructure using Big Data technologies (e.g., Spark, Hadoop ecosystem, Kafka).
• Develop and maintain data warehousing solutions and data lakes.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources.
• Ensure data quality, integrity, and security across all data systems.
• Collaborate with data scientists, analysts, and software engineers to understand data needs and deliver effective solutions.
• Mentor and guide junior data engineers, fostering best practices and technical excellence.
• Automate data processes and monitor system performance for optimal efficiency.
• Stay current with emerging Big Data technologies and best practices.

Requirements

• Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• 5+ years of experience in data engineering, with a strong focus on Big Data technologies.
• Proven experience designing and implementing data pipelines using frameworks like Apache Spark.
• In-depth knowledge of distributed systems and Big Data technologies (Hadoop, Hive, HDFS, Kafka).
• Proficiency in programming languages such as Python, Scala, or Java.
• Strong SQL skills and experience with various database technologies (relational and NoSQL).
• Experience with cloud data platforms (AWS, Azure, GCP) is highly desirable.
• Familiarity with data warehousing concepts and data modeling techniques.
• Excellent leadership, problem-solving, and communication skills.
• Ability to work effectively in a remote, collaborative team environment.

Benefits

• Highly competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance.
• Generous stock options or equity grants.
• Full remote work flexibility, allowing you to work from anywhere.
• Opportunities for continuous learning and professional development.
• Access to the latest Big Data tools and technologies.
• Collaborative and innovative team culture.
• Regular virtual team events and professional networking opportunities.