Senior Data Engineer - Real-Time Pipelines & Cloud

AI71; Abu Dhabi; On-site; AED 600,000 - 900,000; Full time

Job summary

AI71 is looking for a Senior Data Engineer to design, develop, and maintain scalable data systems powering critical decisions. You will lead robust pipelines, ensure data quality and governance, and collaborate across cross-functional teams to deliver production-grade platforms.

Ideal candidates have 8+ years in data engineering, strong Python and SQL/NoSQL skills, and experience with Kafka, Docker, Kubernetes, and cloud services (AWS/Azure). This is an on-site role in Abu Dhabi, UAE.

Qualifications

• 8+ years of experience in data engineering in production environments.
• Proficient in Python and Linux shell scripting for data manipulation.
• Strong SQL/NoSQL database expertise (PostgreSQL, MongoDB).
• Experience building stream processing with Apache Kafka.
• Proficient with Docker and Kubernetes for containerized data workflows.
• Good understanding of cloud services (AWS or Azure).
• Hands-on experience with ELK stack for search and logging.
• Familiarity with Lakehouse architectures and data denormalization.
• Understanding of data governance, metadata, lineage and security practices.
• Experience collaborating with product managers, data scientists, and DevOps.

Responsibilities

• Design, develop, and maintain scalable data pipelines for diverse data sources including real-time feeds.
• Build high-performance data processing systems with denormalization, partitioning, caching, and parallelization.
• Develop and deploy data workflows on AWS or GCP and containerize tasks with Docker/Kubernetes.
• Write and optimize SQL queries on relational and NoSQL databases; support ELK for analytics.
• Ensure data governance, quality, lineage, and auditability across pipelines.
• Collaborate with data scientists to prepare features and support ML model deployment.
• Document data flows and continuously improve the engineering stack with automation.

Skills

Python

Linux shell

SQL/NoSQL

Kafka

Docker

Kubernetes

Cloud platforms

ELK stack

Lakehouse

Data governance

Cross-functional

Education

Bachelor's or Master's in CS/Engineering/Data Science

Tools

PostgreSQL

MongoDB

BigQuery

Redshift

Elasticsearch

Job description

AI71 is looking for a Senior Data Engineer to design, develop, and maintain scalable data systems powering critical decisions. You will lead robust pipelines, ensure data quality and governance, and collaborate across cross-functional teams to deliver production-grade platforms.

Ideal candidates have 8+ years in data engineering, strong Python and SQL/NoSQL skills, and experience with Kafka, Docker, Kubernetes, and cloud services (AWS/Azure). This is an on-site role in Abu Dhabi, UAE.