Senior Data Engineer (Big Data Pipelines)

Placements24; Noord-Kaap; Hybrid; ZAR 900,000 - 1,500,000; Full time

Job summary

Placements24 is seeking a Senior Data Engineer to join a distributed, remote-first team. You will design, build, and maintain scalable data pipelines, data warehouses, and data lakes using Spark, Hadoop, and cloud services.

The role emphasizes data quality, accessibility for data scientists, and collaboration across a remote team. You will work with Python/Scala/Java, ETL/ELT tooling, and modern orchestration systems to enable real-time and batch processing.

Qualifications

• Experience designing and building large-scale data pipelines using big data technologies such as Spark, Hadoop, Hive.
• Proficiency in SQL and experience with relational and NoSQL databases.
• Cloud data platform experience (AWS, Azure, GCP) and data warehousing knowledge.
• Strong programming skills in Python, Scala, or Java; data modeling and ETL/ELT theory.
• Excellent problem-solving, analytical, and debugging skills; strong collaboration in remote settings.

Responsibilities

• Design, develop, and optimize scalable data pipelines for batch and real-time data processing.
• Build and maintain data warehousing solutions and data lakes on cloud platforms.
• Implement ETL/ELT processes to ingest, transform, and load data from sources.
• Ensure data quality, integrity, and reliability across data systems.
• Collaborate with data scientists and analysts to provide efficient data access.
• Develop and maintain data infrastructure, monitoring performance, and addressing issues.

Skills

Programming languages

SQL proficiency

Remote work communication

Education

Bachelor's degree in Computer Science, Engineering, or a related quantitative field

Tools

Spark

Hadoop

Hive

Airflow

Prefect

Redshift

Glue

BigQuery

Dataflow

Job description

About the Role

Our client, a cutting-edge data analytics firm, is seeking a highly skilled Senior Data Engineer to join their distributed, fully remote team. This role is critical for designing, building, and maintaining robust data pipelines and infrastructure that support the company's advanced analytics and machine learning initiatives. You will work with vast datasets, implement efficient data processing solutions, and ensure data quality and accessibility for data scientists and analysts. As a remote-first organization, strong communication, self-discipline, and experience with collaborative remote tools are paramount. This is an exceptional opportunity to work with state-of-the-art big data technologies from anywhere.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines for batch and real-time data processing.
• Build and maintain data warehousing solutions and data lakes on cloud platforms.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources.
• Ensure data quality, integrity, and reliability across all data systems.
• Collaborate with data scientists and analysts to understand their data requirements and provide efficient data access.
• Develop and maintain data infrastructure, monitoring performance and proactively addressing issues.
• Implement data governance best practices and ensure compliance with data privacy regulations.
• Automate data processes and workflows using scripting and orchestration tools.
• Evaluate and integrate new data technologies and tools to improve data processing capabilities.
• Participate in code reviews and contribute to architectural decisions for data platforms.
• Effectively communicate technical solutions and data insights within a remote team environment.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• Minimum of 6 years of experience in data engineering or a related role.
• Proven experience in designing and building large-scale data pipelines using big data technologies (e.g., Spark, Hadoop, Hive).
• Proficiency in SQL and experience with various relational and NoSQL databases.
• Experience with cloud data platforms (e.g., AWS EMR, Redshift, Glue; Azure Data Factory, Synapse; GCP Dataflow, BigQuery).
• Strong programming skills in Python, Scala, or Java.
• Experience with data warehousing concepts and ETL/ELT tools.
• Familiarity with data orchestration tools like Airflow or Prefect.
• Understanding of data modeling and database design principles.
• Excellent problem-solving, analytical, and debugging skills.
• Strong communication and collaboration skills, essential for remote work.
• Demonstrated ability to work independently and manage priorities effectively in a remote setting.

Benefits

Our client offers a highly competitive salary, comprehensive health benefits, performance bonuses, and ample opportunities for professional development, including advanced training and certifications. This is a fully remote position, allowing for excellent work-life balance and flexibility. The company cultivates a strong remote work culture with regular virtual team meetings, collaborative online platforms, and opportunities for significant career growth. Join a leading data-driven organization and build the future of data infrastructure from anywhere.