Senior Data Engineer (Big Data Technologies)

Placements24; Upington; On-site; ZAR 900,000 - 1,200,000; Full time

Benefits offered by this job

Competitive salary

In-office Upington location

Professional development

Access to modern data tech

Generous leave

Job summary

Placements24 is seeking a Senior Data Engineer to join the team in Upington, Northern Cape, in-office. The role focuses on building scalable data pipelines, managing data warehouses and data lakes, and ensuring data quality for BI initiatives.

You will work with data scientists, analysts, and stakeholders to translate data needs into robust solutions, leveraging Spark, Hadoop, and cloud platforms. A strong programming background and experience with large datasets are essential for success.

Qualifications

• Bachelor's or higher in CS/Engineering; Master's preferred.
• 5+ years in data engineering or similar role.
• Proficient in Python, Scala or Java and big data tech.

Responsibilities

• Design, build, test, and maintain scalable data pipelines and ETL/ELT processes.
• Develop data warehousing solutions and data lakes.
• Optimize data retrieval for performance and cost.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, and security across data systems.

Skills

Python/Scala/Java

Big data technologies

SQL proficiency

Data warehousing concepts

Communication skills

Independent/team collaboration

Education

Bachelor's degree in CS/Engineering

Master's degree preferred

Tools

Apache Spark

Hadoop ecosystem

Hive

Kafka

PostgreSQL/MySQL

AWS Redshift/S3

Azure Data Lake/Google BigQuery

Docker

Job description

About the Role

Our client, a leading firm in data analytics and business intelligence, is seeking a highly skilled Senior Data Engineer to join their dynamic team based in Upington, Northern Cape . This is an in-office, full-time role focused on building and maintaining robust, scalable data pipelines and infrastructure. You will play a critical role in managing large datasets, optimizing data flow, and ensuring data quality and availability for various business intelligence and analytical initiatives. The ideal candidate possesses extensive experience with big data technologies, strong programming skills, and a deep understanding of data warehousing concepts. Join an innovative company that is leveraging data to drive strategic decisions and offers a challenging and rewarding work environment in Upington .

Key Responsibilities

• Design, build, test, and maintain scalable data pipelines and ETL/ELT processes.
• Develop and manage data warehousing solutions and data lakes.
• Optimize data retrieval and processing for performance and cost-efficiency.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, and security across all data systems.
• Implement data governance policies and best practices.
• Work with distributed computing systems and big data technologies (e.g., Spark, Hadoop ecosystem).
• Develop and maintain data models for various analytical and reporting needs.
• Monitor data systems for performance issues and implement solutions.
• Stay current with emerging data engineering trends and technologies.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related quantitative field; Master's degree preferred.
• 5+ years of experience in data engineering or a similar role.
• Proficiency in programming languages such as Python, Scala, or Java.
• Hands-on experience with big data technologies and frameworks (e.g., Apache Spark, Hadoop, Hive, Kafka).
• Strong experience with SQL and relational database systems (e.g., PostgreSQL, MySQL).
• Experience with cloud data platforms (e.g., AWS Redshift, S3, Azure Data Lake, Google BigQuery) is a plus.
• Understanding of data warehousing concepts and ETL/ELT best practices.
• Experience with data modeling and database design.
• Familiarity with containerization technologies (e.g., Docker) is desirable.
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and collaboratively in a team environment.

Benefits

• Competitive salary and benefits package.
• Full-time, in-office position within a collaborative team in Upington .
• Opportunities for professional development and skill enhancement.
• Work with cutting-edge big data technologies and complex datasets.
• Contribute to data-driven decision-making within the organization.
• Comprehensive health and wellness benefits.
• Retirement savings plan with company contribution.
• Generous annual leave and company holidays.
• Access to modern office facilities and resources.
• Regular opportunities for learning and knowledge sharing.