Senior Cloud Data Engineer - Remote

Recru-it; South Africa; Hybrid; ZAR 600 000 - 1 000 000; Full time

Job summary

An established industry player is seeking a Senior Cloud Engineer to enhance their dynamic Data Engineering team. This pivotal role involves designing and optimizing scalable cloud-agnostic data solutions using cutting-edge technologies like Azure, Databricks, and Spark. The ideal candidate will possess extensive experience in cloud-based data engineering, showcasing advanced skills in T-SQL and Python. You'll lead projects, mentor junior engineers, and collaborate with diverse teams to meet business objectives. Enjoy a flexible work environment, comprehensive benefits, and a culture that values transparency and work-life balance while making a significant impact in the data engineering landscape.

Benefits

Flexible Working Arrangements

Comprehensive Benefits

Team Culture

Performance Incentives

Qualifications

• 6+ years in cloud-based data engineering with Azure expertise.
• Advanced skills in T-SQL and Python for data transformations.

Responsibilities

• Architect and optimize cloud-agnostic data solutions using Azure.
• Lead projects and mentor junior engineers in data engineering.

Skills

Cloud-based Data Engineering

Azure

Databricks

Spark

T-SQL

Python

ETL/ELT Processes

Data Modeling

Big Data Processing

DevOps Practices

Education

Computer Science Degree

Data Engineering Certification

Tools

Azure Synapse

Azure Data Factory

ADLS

Azure DevOps

Git

Docker

Kubernetes

Job description

Our client is seeking a Senior Cloud Engineer to join their dynamic and growing Engineering team.

This individual will be a key contributor within the Data Engineering team, playing a pivotal role in designing, developing, and optimizing scalable cloud-agnostic data solutions.

The ideal candidate will have strong expertise in Databricks, Spark, Lakehouse architecture, Delta Lake tables, and other modern data engineering technologies. They should be proficient in T-SQL and Python, with advanced coding abilities, and possess a deep understanding of cloud-based and cloud-agnostic data architectures.

This is a senior role that requires the ability to lead projects, mentor junior team members, and work independently on complex data engineering challenges.
Key Responsibilities:

1. Data Engineering & Cloud Solutions:
   - Architect, build, and optimize scalable and cloud-agnostic data solutions using Azure, Databricks, Spark, Lakehouse, and Delta Lake tables.
   - Develop, implement, and maintain big data pipelines for ingesting, processing, and storing large volumes of structured and unstructured data.
   - Manage and optimize data lake and data warehouse architectures for performance, cost, and scalability.
2. Cloud & DevOps:
   - Work within Azure environments (Azure Synapse, Data Factory, ADLS, etc.) to develop and maintain cloud-based data solutions.
   - Implement best DevOps practices for CI/CD pipelines, infrastructure-as-code, and automation.
   - Utilize Azure DevOps and Git for managing code repositories, version control, and continuous integration/deployment.
   - Ensure high levels of security, compliance, and data governance across data engineering processes.
3. Big Data Processing & Development:
   - Utilize Spark, Databricks, and distributed computing to process and analyze large datasets efficiently.
   - Write advanced Python and T-SQL scripts for data transformations, ETL/ELT processes, and real-time data processing.
   - Optimize performance for data pipelines and SQL queries for efficiency and cost-effectiveness.
   - Experience with Graph databases is a big advantage.
4. Collaboration & Leadership:
   - Work closely with data scientists, analysts, and business stakeholders to understand data requirements and develop solutions that meet business objectives.
   - Lead initiatives to enhance data engineering capabilities, introduce new technologies, and drive best practices.
   - Mentor junior engineers, conduct code reviews, and contribute to building a culture of technical excellence.
   - Communicate effectively with technical and non-technical stakeholders, translating complex data concepts into actionable insights.

Required Qualifications:

1. 6+ years of extensive experience.
2. Education / Degrees: Computer science degree or comparable data engineering certification in one of the cloud platforms.

Minimum required skills and experience:

1. Extensive experience in cloud-based data engineering, with expertise in Azure (Azure Synapse, Azure Data Factory, ADLS, etc.).
2. Strong expertise in cloud-agnostic data tools such as Databricks, Spark, Delta Lake, and Lakehouse architectures.
3. Advanced proficiency in T-SQL and Python for developing complex data pipelines, transformations, and optimizations.
4. Hands-on experience in big data processing, ETL/ELT, and data pipeline orchestration.
5. Solid understanding of data modeling, warehouse design, and data lakehouse architecture.
6. Experience in setting up, managing, and processing with Azure DevOps (or similar DevOps tools).
7. Strong knowledge of DevOps, CI/CD, Git, and infrastructure-as-code for automated deployments.
8. Excellent verbal and written communication skills, with the ability to collaborate across teams and explain complex data concepts clearly.
9. Ability to work independently on high-impact projects while maintaining a team-oriented approach.

Added Advantage:

1. Experience with containerization technologies (Docker, Kubernetes) for data processing workloads.
2. Knowledge of data security, encryption, and compliance regulations related to big data.
3. Exposure to machine learning and AI pipelines within data engineering workflows.
4. Experience working with multi-cloud environments (AWS, GCP) alongside Azure.

Why you’ll love working for this company:

The company believes in taking care of their team and creating an environment where you can thrive. As part of our company, you’ll enjoy:

1. Flexible Working Arrangements: Whether you’re a night owl or an early bird, they offer hybrid and remote options to suit your lifestyle.
2. Comprehensive Benefits: From a wellness program to home office reimbursements and continuous learning opportunities, they have got you covered.
3. Team Culture: Fun team-building activities, regular socials, and a supportive, inclusive culture that values transparency, accountability, and work-life balance.
4. Performance Incentives: Competitive salaries, ESOP, and recognition for your hard work.

Desired Skills:

• Azure (Azure Synapse)
• Azure Data Factory
• ADLS
• T-SQL and Python
• Data tools