Senior Data Engineer - Cloud Platforms

Placements24; Benoni; On-site; ZAR 900,000 - 1,200,000; Full time

Benefits offered by this job

Competitive salary

Performance-based bonuses

Health, dental, and vision benefits

Retirement savings plan

Paid time off

Job summary

Placements24 in Benoni, South Africa, is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines and data infrastructure. You will transform raw data into actionable insights to support BI and ML initiatives, in a modern office setting.

The ideal candidate has 5+ years in data engineering, strong cloud data services (AWS/Azure/GCP), and expert Python/Scala/Java skills with Spark and Hadoop. Collaboration with data scientists and business stakeholders is essential.

Qualifications

• Requires degree in a quantitative field and 5+ years in data engineering.
• Strong cloud data services experience (AWS/Azure/GCP) and BI/ML support knowledge.
• Proficiency in Python/Scala/Java with Spark and Hadoop experience.

Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL/ELT processes.
• Build and maintain data warehouses and data lakes on cloud platforms.
• Implement data governance and data quality best practices.
• Collaborate with data scientists, analysts, and stakeholders to define data requirements.
• Develop and manage infrastructure for big data processing (Spark, Kafka, Flink).
• Monitor performance, troubleshoot, and ensure high availability and efficiency.
• Automate data processes and deployments using CI/CD concepts.
• Stay current with emerging data technologies and document standards.
• Contribute to architectural design of data platforms.

Skills

Analytical thinking

Communication skills

Team collaboration

Agile development

Education

Bachelor's degree in Computer Science / Engineering / IT

Master's degree is a plus

Tools

Python

Scala

Java

Apache Spark

Apache Hadoop

Airflow

Databricks

SQL

Cloud data services

Job description

About the Role

Our client, a leading enterprise in data analytics, is seeking a highly skilled Senior Data Engineer to join their robust technology team. This role is critical for designing, building, and maintaining scalable and reliable data pipelines and infrastructure. You will be instrumental in transforming raw data into actionable insights, supporting various business intelligence and machine learning initiatives. This position is based in our modern Benoni office and requires a strong background in cloud-based data solutions and big data technologies. The ideal candidate is passionate about data, possesses exceptional problem-solving skills, and thrives in a collaborative, fast-paced environment. You will work with a team of talented engineers and data scientists to architect and implement cutting-edge data solutions.

Key Responsibilities

• Design, develop, and optimize robust, scalable data pipelines and ETL/ELT processes.
• Build and maintain data warehousing solutions and data lakes on cloud platforms (AWS, Azure, or GCP).
• Implement data governance, data quality, and data security best practices.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Develop and manage infrastructure for big data processing using technologies like Spark, Kafka, or Flink.
• Monitor data system performance, troubleshoot issues, and implement solutions to ensure high availability and efficiency.
• Automate data processes and deployments using CI/CD principles.
• Stay current with emerging data technologies and best practices, recommending and implementing new tools and techniques.
• Document data architecture, processes, and standards.
• Contribute to the architectural design of data platforms and solutions.

Requirements

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related quantitative field.
• Master's degree is a plus.
• 5+ years of experience in data engineering, with a strong focus on building and managing data infrastructure.
• Proven experience with cloud data services (e.g., AWS S3, Redshift, Glue; Azure Data Lake, Databricks, Synapse; GCP Cloud Storage, BigQuery, Dataflow).
• Proficiency in programming languages such as Python, Scala, or Java.
• Strong SQL skills and experience with various database technologies (relational and NoSQL).
• Experience with big data technologies like Apache Spark, Hadoop ecosystem.
• Familiarity with data orchestration tools (e.g., Airflow, Prefect).
• Understanding of data warehousing concepts and dimensional modeling.
• Excellent problem-solving, analytical, and communication skills.
• Experience working in an Agile development environment.
• Ability to work effectively in a team and collaborate with cross-functional stakeholders.

Benefits

Our client offers a competitive salary, performance-based bonuses, and comprehensive health, dental, and vision benefits. We provide a retirement savings plan, ample paid time off, and continuous opportunities for professional development, including training, certifications, and conference attendance. This role is based in our state-of-the-art Benoni office, fostering a collaborative and innovative work environment. Join our client and contribute to building the future of data infrastructure in a company that values innovation and employee growth.