Azure Data Platform Engineer

IDC Technologies; Abu Dhabi; On-site; AED 120,000 - 200,000; Full time

Job summary

A leading company in Abu Dhabi is seeking a Senior Platform Engineer with extensive experience in Azure and data engineering. The role entails implementing and managing distributed systems, automating infrastructure, and utilizing services like Azure Data Factory and Databricks. Ideal candidates will have a strong background in Azure administration and experience with DevOps methodologies, with responsibilities ranging from infrastructure management to CI/CD pipeline setups.

Qualifications

• Proven experience in Azure platform services and data engineering.
• Strong understanding of distributed system architectures and DevOps practices.
• Experience in automating infrastructure provisioning and management.

Responsibilities

• Implement and manage infrastructure using IaC tools like Terraform.
• Manage containerized apps with Docker and Azure Kubernetes Service.
• Automate processes with Python, shell scripting, and Azure DevOps.

Skills

Azure Administration

Data Engineering

Infrastructure as Code (IaC)

Docker

Azure Kubernetes Service (AKS)

Python

Terraform

Azure DevOps

Education

Bachelor's Degree in Computer Science or related field

Tools

Azure Data Factory

Azure Monitor

Databricks

Event Hub

Azure CI/CD

Job description

Senior Platform Engineer - Azure

We are seeking a Senior Platform Engineer proficient in distributed system architectures and Azure Data Engineering services, including:

• Event Hub
• Data Factory
• ADLS Gen2
• Cosmos DB
• Synapse
• Databricks
• APIM, Function App, Logic App, and App Services

Responsibilities include:

1. Implementing and managing infrastructure using IaC tools such as ARM templates and Terraform.
2. Managing containerized applications with Docker and orchestrating with Azure Kubernetes Service (AKS).
3. Setting up and managing monitoring, logging, and alerting systems using Azure Monitor, Log Analytics, and Application Insights.
4. Implementing disaster recovery strategies, backups, and failover mechanisms for critical workloads.
5. Automating infrastructure provisioning, scaling, and management for high availability and efficiency.
6. Managing and maintaining clusters across various environments on Azure.
7. Defining scheduling and monitoring job flows with proactive alert setup.
8. Troubleshooting failed jobs in Azure tools like Databricks and Data Factory, performing root cause analysis, and applying corrective measures.
9. Hands-on experience with distributed streaming tools like Event Hub.
10. Designing and managing backup and disaster recovery solutions using IaC with Terraform.
11. Automating processes using Python, shell scripting, Jenkins, and Azure DevOps.
12. Designing and maintaining Azure CI/CD pipelines for seamless code integration, testing, and deployment.
13. Monitoring and troubleshooting VM resources and applications, advising developers on performance improvements.
14. Reviewing and resolving log file issues for system and application components.
15. Adapting to evolving technologies with a strong sense of responsibility and accomplishment.
16. Applying agile methodologies for software delivery.

We require a Data Platform Engineer with a strong understanding of Azure administration and Azure DevOps, with expertise in:

• Storage Account
• Data Lake (ADLS)
• Databricks Unity Catalog, including table types, Parquet vs Delta, Optimize, Vacuum, Security, Segregation, job slowness issues, version control
• Azure Data Factory (ADF)
• Event Hub and Event Grid, message flow issues, Terraform setup
• Key Vault
• Cosmos DB
• Synapse

Location: Abu Dhabi, United Arab Emirates