Data Engineer

BigTalent; Johannesburg; On-site; ZAR 600,000 - 900,000; Full time

Job summary

Unknown company in Johannesburg is seeking a Data Engineer specialized in Google Cloud Platform (GCP) to design, develop, and maintain robust data pipelines and cloud infrastructure solutions.

You will design and implement data pipelines using Dataflow, Pub/Sub, Composer and BigQuery, optimize ETL/ELT workflows, and ensure secure, governed data processing. Collaborate with stakeholders, monitor performance, and mentor junior staff in GCP technologies.

Qualifications

• Proven experience as a Data Engineer with extensive GCP expertise.
• Hands-on experience with Dataflow, Pub/Sub, Composer and BigQuery.
• Strong skills in ETL/ELT optimization, performance tuning, and security best practices.

Responsibilities

• Design and build scalable data pipelines on GCP.
• Optimize ETL/ELT processes for reliable ingestion and storage.
• Monitor pipelines, troubleshoot performance bottlenecks and data anomalies.
• Ensure data governance, security, and compliance within the GCP environment.
• Document cloud architectures, pipelines and deployment processes; mentor junior team members.

Skills

GCP expertise

Data pipeline design

ETL/ELT optimization

Problem solving

Tools

Dataflow

Pub/Sub

Composer

BigQuery

Job description

Join our clients dynamic team that collaborates closely with Finance and Investment teams to drive data automation, optimize operational processes, and execute high-impact analytics projects.

As a Data Engineer specialized in Google Cloud Platform (GCP), your primary role will be to design, develop, and maintain robust data pipelines and cloud infrastructure solutions.

Key Responsibilities:

• Design and build scalable, robust data pipelines using GCP services like Dataflow, Pub/Sub, Composer, and BigQuery.
• Optimize ETL/ELT processes to ensure reliable data ingestion and efficient storage solutions.

Collaboration & Stakeholder Engagement:

• Partner closely with relevant stakeholders to understand strategic project goals, effectively prioritizing tasks that deliver measurable outcomes.
• Coordinate efforts to uphold code quality, security, and architectural integrity.

Performance Tuning & Monitoring:

• Implement comprehensive monitoring, logging, and alerting solutions to maintain pipeline performance and data reliability.
• Actively troubleshoot and resolve performance bottlenecks or data anomalies.

Security & Compliance:

• Adhere strictly to established data governance and security protocols, particularly around sensitive financial and investment data.
• Ensure best practices for access control, encryption, and network security within the GCP environment.
• Proactively identify automation and cost-efficiency opportunities within the GCP framework.
• Explore and pilot innovative GCP tools and services to enhance the client's data analytics capabilities.

Documentation & Knowledge Sharing:

• Maintain accurate and thorough documentation for cloud architecture, data pipelines, and deployment processes.
• Actively contribute to team knowledge-sharing initiatives and provide mentorship to junior team members on GCP technologies.

Your Profile:

• Proven experience as a Data Engineer with extensive GCP expertise.
• Demonstrable experience with GCP tools: Dataflow, Pub/Sub, Composer, and BigQuery.
• Strong proficiency in ETL/ELT pipeline optimization, performance tuning, and security best practices.
• Excellent collaboration, problem-solving, and analytical skills.

Join our clients innovative team to build impactful solutions and advance your career within a strategic, data-driven environment.