Data Engineer - Big Data & Cloud Platforms

Placements24; Kimberley; On-site; CAD 60,000 - 82,000; Full time

Benefits offered by this job

Medical aid

Retirement fund

Paid time off

Professional development

Dynamic work in Kimberley

Big data projects exposure

Team building

Career advancement

Job summary

Placements24 is seeking an experienced Data Engineer to join our team in Kimberley. You will design, build, and maintain scalable data pipelines and cloud-based data infrastructure to support big data initiatives.

The role requires expertise in ETL, data warehousing, and proficiency in Python or Scala, with hands-on experience on AWS/Azure/GCP and related services. You will collaborate with data scientists and analysts to deliver robust data solutions.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related technical field, or equivalent practical experience.
• Minimum of 4 years of experience in data engineering, data warehousing, or related role.
• Proven experience designing and implementing complex ETL processes.
• Proficiency in SQL and experience with relational database management systems.
• Experience with big data technologies such as Hadoop, Spark, or Kafka.
• Hands-on experience with cloud platforms (AWS, Azure, or GCP) and their data services (e.g., S3, Redshift, BigQuery, Data Factory).
• Strong programming skills in Python or Scala.
• Experience with data modeling techniques.
• Familiarity with data visualization tools is a plus.
• Excellent problem-solving and analytical skills.
• Strong communication and teamwork abilities.

Responsibilities

• Design, construct, install, test, and maintain highly scalable data management systems and pipelines.
• Develop ETL (Extract, Transform, Load) processes to consolidate data from various sources into data warehouses or data lakes.
• Optimize data retrieval and query performance for analytical needs.
• Build and maintain data infrastructure on cloud platforms such as AWS, Azure, or GCP.
• Collaborate with data scientists and analysts to understand their data requirements.
• Implement data governance, data quality, and data security best practices.
• Monitor data systems for performance, reliability, and efficiency.
• Automate data processes and deployments using scripting and CI/CD tools.
• Troubleshoot and resolve data-related issues.
• Stay current with emerging big data and cloud technologies.

Skills

SQL

Python

Scala

Data modeling

ETL design

Problem solving

Education

Bachelor's degree in CS/Engineering or equivalent

Tools

Hadoop

Spark

Kafka

AWS

Azure

GCP

Data Factory

BigQuery

Job description

About the Role

Our client, a prominent data analytics firm, is seeking an experienced Data Engineer to join their team in Kimberley, Northern Cape . This role is crucial for designing, building, and maintaining scalable data pipelines and infrastructure that support the organization's big data initiatives. You will work with large, complex datasets, transforming raw data into actionable insights that drive business decisions. This is an exciting opportunity to leverage your expertise in cloud platforms and big data technologies, collaborating with data scientists and analysts to deliver robust data solutions. The ideal candidate has a strong foundation in data warehousing, ETL processes, and experience with cloud-based data services.

Key Responsibilities

• Design, construct, install, test, and maintain highly scalable data management systems and pipelines.
• Develop ETL (Extract, Transform, Load) processes to consolidate data from various sources into data warehouses or data lakes.
• Optimize data retrieval and query performance for analytical needs.
• Build and maintain data infrastructure on cloud platforms such as AWS, Azure, or GCP.
• Collaborate with data scientists and analysts to understand their data requirements.
• Implement data governance, data quality, and data security best practices.
• Monitor data systems for performance, reliability, and efficiency.
• Automate data processes and deployments using scripting and CI/CD tools.
• Troubleshoot and resolve data-related issues.
• Stay current with emerging big data and cloud technologies.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
• Minimum of 4 years of experience in data engineering, data warehousing, or a related role.
• Proven experience designing and implementing complex ETL processes.
• Proficiency in SQL and experience with relational database management systems.
• Experience with big data technologies such as Hadoop, Spark, or Kafka.
• Hands-on experience with cloud platforms (AWS, Azure, or GCP) and their data services (e.g., S3, Redshift, BigQuery, Data Factory).
• Strong programming skills in Python or Scala.
• Experience with data modeling techniques.
• Familiarity with data visualization tools is a plus.
• Excellent problem-solving and analytical skills.
• Strong communication and teamwork abilities.

Benefits

• Competitive salary and annual performance bonuses.
• Comprehensive medical aid and retirement fund benefits.
• Generous paid time off.
• Opportunities for professional development and certifications in cloud and big data technologies.
• A dynamic work environment in Kimberley, Northern Cape .
• Exposure to large-scale data projects and modern tech stacks.
• Team-building activities and a supportive work culture.
• Potential for career advancement within the data analytics domain.