Data Engineer - Big Data Platforms

Placements24; Kimberley; Hybrid; CAD 90,000 - 130,000; Full time

Benefits offered by this job

Bonus program

Health benefits

Growth opportunities

Flexible remote work

Cutting-edge data tools

Job summary

Placements24 in Kimberley is seeking a Data Engineer to design, develop and optimize data pipelines and data warehouses, enabling analytics and ML teams to access reliable data. You will work with Spark, Hadoop, Hive and Kafka, and leverage cloud platforms (AWS/Azure/GCP) with services like S3, Redshift and BigQuery.

Strong SQL skills and Python or Scala are required, along with 3+ years in data engineering.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related quantitative field.
• 3+ years of experience as a Data Engineer or in a similar role.
• Proficiency with big data technologies such as Apache Spark, Hadoop, Hive, or Kafka.
• Experience with cloud data platforms (AWS, Azure, GCP) and related services (e.g., S3, Redshift, BigQuery).
• Strong SQL skills and experience with data warehousing concepts.
• Proficiency in programming languages like Python or Scala.

Responsibilities

• Design, build, and maintain scalable and reliable data pipelines using big data technologies (e.g., Spark, Hadoop, Kafka).
• Develop and manage data warehouses and data lakes, optimizing for performance and cost.
• Implement ETL/ELT processes to integrate data from various sources.
• Ensure data quality, integrity, and security across all data platforms.
• Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
• Monitor data pipeline performance and troubleshoot any issues that arise.

Skills

SQL

Python

Scala

Education

Bachelor's degree in Computer Science, Engineering, or related quantitative field

Tools

Apache Spark

Hadoop

Hive

Kafka

AWS

Azure

GCP

S3

Redshift

BigQuery

Job description

About the Role

Our client is seeking a skilled Data Engineer to join their team in Kimberley and play a key role in building and maintaining their big data infrastructure. You will be responsible for designing, developing, and optimizing data pipelines, ensuring efficient data flow and accessibility for analytics and machine learning teams. This role is essential for transforming raw data into actionable insights, requiring a strong understanding of distributed systems and data warehousing concepts. Join a team dedicated to harnessing the power of data.

Key Responsibilities

• Design, build, and maintain scalable and reliable data pipelines using big data technologies (e.g., Spark, Hadoop, Kafka).
• Develop and manage data warehouses and data lakes, optimizing for performance and cost.
• Implement ETL/ELT processes to integrate data from various sources.
• Ensure data quality, integrity, and security across all data platforms.
• Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
• Monitor data pipeline performance and troubleshoot any issues that arise.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• 3+ years of experience as a Data Engineer or in a similar role.
• Proficiency with big data technologies such as Apache Spark, Hadoop, Hive, or Kafka.
• Experience with cloud data platforms (AWS, Azure, GCP) and related services (e.g., S3, Redshift, BigQuery).
• Strong SQL skills and experience with data warehousing concepts.
• Proficiency in programming languages like Python or Scala.

Benefits

• Competitive salary and performance-based bonuses.
• Comprehensive health and wellness benefits.
• Opportunities for professional growth and training in big data technologies.
• Flexible remote work options.
• Collaborative team environment working with cutting-edge data tools.