Senior Data Engineer - Big Data Solutions

Placements24; East London; Hybrid; ZAR 900,000 - 1,300,000; Full time

Benefits offered by this job

Hybrid work model

Career growth opportunities

Industry conferences

Job summary

Placements24 in East London is seeking a Senior Data Engineer to lead design and implementation of scalable Big Data solutions. You will build robust data pipelines, ensure data quality, and enable advanced analytics across large datasets, collaborating with data scientists and analysts.

The role offers a hybrid work arrangement in East London, with opportunities for professional growth, training, and involvement in cutting-edge projects that drive data-driven decision making.

Qualifications

• Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• 5+ years of experience in data engineering, with a strong focus on big data technologies.
• Proficiency in Python, Scala, or Java.

Responsibilities

• Design, build, and maintain scalable and efficient data pipelines using big data technologies (e.g., Spark, Hadoop ecosystem).
• Develop ETL/ELT processes to ingest, transform, and load data into data warehouses or data lakes.
• Ensure data quality, integrity, and reliability across all data platforms.
• Optimize data storage, processing, and query performance for large datasets.
• Collaborate with data scientists and business analysts to understand data needs and provide data solutions.
• Implement monitoring and alerting for data pipeline health and performance.
• Stay current with emerging big data technologies and best practices.

Skills

Big data

Spark

Hadoop

Python

Scala

Java

SQL

NoSQL

Data warehousing

Cloud

Education

Master's or Bachelor's in Computer Science / Engineering

Tools

Apache Spark

Apache Hadoop

AWS/Azure/GCP data services

Job description

About the Role

Our client is seeking an accomplished Senior Data Engineer to lead the design and implementation of sophisticated Big Data Solutions in East London. This pivotal role involves building and optimizing robust data pipelines, ensuring data quality, and enabling advanced analytics capabilities. You will work with large-scale datasets, cutting-edge technologies, and collaborate closely with data scientists and analysts to unlock valuable business insights. If you are passionate about data architecture and thrive in a challenging, dynamic environment, this is the role for you.

Key Responsibilities

• Design, build, and maintain scalable and efficient data pipelines using big data technologies (e.g., Spark, Hadoop ecosystem).
• Develop ETL/ELT processes to ingest, transform, and load data from various sources into data warehouses or data lakes.
• Ensure data quality, integrity, and reliability across all data platforms.
• Optimize data storage, processing, and query performance for large datasets.
• Collaborate with data scientists and business analysts to understand data needs and provide data solutions.
• Implement monitoring and alerting for data pipeline health and performance.
• Stay current with emerging big data technologies and best practices.

Requirements

• Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• 5+ years of experience in data engineering, with a strong focus on big data technologies.
• Proficiency in programming languages such as Python, Scala, or Java.
• Hands-on experience with distributed computing frameworks like Apache Spark.
• Experience with SQL and NoSQL databases, as well as data warehousing concepts.
• Familiarity with cloud platforms (AWS, Azure, GCP) and their data services.
• Strong understanding of data modeling and data architecture principles.

Benefits

• Highly competitive salary and comprehensive benefits package.
• Hybrid work model offering flexibility in East London.
• Opportunities for professional growth, including training and attending industry conferences.
• Work with a talented team on challenging and impactful data engineering projects.
• Be part of an organization committed to data-driven innovation.