Senior Data Engineer [UAE Based]

AI71; London; On-site; GBP 60,000 - 90,000; Full time

Job summary

A leading technology company is seeking a Senior Data Engineer to design and maintain scalable data systems. This role will focus on developing data pipelines, ensuring data integrity, and implementing cloud solutions, requiring extensive experience and expertise in modern data engineering practices.

Qualifications

• 8+ years of experience in data engineering within a production environment.
• Advanced knowledge of Python and Linux shell scripting.
• Experience building stream processing systems using Apache Kafka.

Responsibilities

• Design scalable data pipelines for structured and unstructured data.
• Implement data quality tools and validation frameworks.
• Develop and deploy data workflows on AWS or GCP.

Skills

Python

SQL

Data Governance

Data Integrity

Cloud Services

Stream Processing

Data Quality Tools

Education

Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science

Tools

Docker

Kubernetes

ELK Stack

Apache Kafka

AWS

GCP

Job description

As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining advanced, scalable data systems that power critical business decisions. You will lead the development of robust data pipelines, ensure data quality and governance, and collaborate across cross-functional teams to deliver high-performance data platforms in production environments. This role requires a deep understanding of modern data engineering practices, real-time processing, and cloud-native solutions.

Key Responsibilities:

• Design, implement, and maintain scalable and reliable data pipelines to ingest, transform, and load structured, unstructured, and real-time data feeds from diverse sources.
• Manage data pipelines for analytics and operational use, ensuring data integrity, timeliness, and accuracy across systems.
• Implement data quality tools and validation frameworks within transformation pipelines.

Data Processing & Optimization:

• Build efficient, high-performance systems by leveraging techniques like data denormalization, partitioning, caching, and parallel processing.
• Develop stream-processing applications using Apache Kafka and optimize performance for large-scale datasets.
• Enable data enrichment and correlation across primary, secondary, and tertiary sources.

Cloud, Infrastructure, and Platform Engineering:

• Develop and deploy data workflows on AWS or GCP, using services such as S3, Redshift, Pub/Sub, or BigQuery.
• Containerize data processing tasks using Docker, orchestrate with Kubernetes, and ensure production-grade deployment.
• Collaborate with platform teams to ensure scalability, resilience, and observability of data pipelines.
• Write and optimize complex SQL queries on relational (Redshift, PostgreSQL) and NoSQL (MongoDB) databases.
• Work with ELK stack (Elasticsearch, Logstash, Kibana) for search, logging, and real-time analytics.
• Support Lakehouse architectures and hybrid data storage models for unified access and processing.

Data Governance & Stewardship:

• Implement robust data governance, access control, and stewardship policies aligned with compliance and security best practices.
• Establish metadata management, data lineage, and auditability across pipelines and environments.

Machine Learning & Advanced Analytics Enablement:

• Collaborate with data scientists to prepare and serve features for ML models.
• Maintain awareness of ML pipeline integration and ensure data readiness for experimentation and deployment.
• Maintain thorough documentation including technical specifications, data flow diagrams, and operational procedures.
• Continuously evaluate and improve the data engineering stack by adopting new technologies and automation strategies.

Required Skills & Qualifications:

• 8+ years of experience in data engineering within a production environment.
• Advanced knowledge of Python and Linux shell scripting for data manipulation and automation.
• Strong expertise in SQL/NoSQL databases such as PostgreSQL and MongoDB.
• Experience building stream processing systems using Apache Kafka.
• Proficiency with Docker and Kubernetes in deploying containerized data workflows.
• Good understanding of cloud services (AWS or Azure).
• Hands-on experience with ELK stack (Elasticsearch, Logstash, Kibana) for scalable search and logging.
• Familiarity with AI models supporting data management.
• Experience working with Lakehouse systems, data denormalization, and data labeling practices.

Preferred Qualifications:

• Working knowledge of data quality tools, lineage tracking, and data observability solutions.
• Experience in data correlation, enrichment from external sources, and managing data integrity at scale.
• Understanding of data governance frameworks and enterprise compliance protocols.
• Exposure to CI/CD pipelines for data deployments and infrastructure-as-code.

Education & Experience:

• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field.
• Demonstrated success in designing, scaling, and operating data systems in cloud-native and distributed environments.
• Proven ability to work collaboratively with cross-functional teams including product managers, data scientists, and DevOps.

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Job function

  Information Technology

• Industries

  Technology, Information and Internet

Referrals increase your chances of interviewing at AI71 by 2x

Sign in to set job alerts for “Data Engineer” roles.

Greater London, England, United Kingdom 2 days ago

London, England, United Kingdom 2 weeks ago

London, England, United Kingdom 1 week ago

London, England, United Kingdom 2 weeks ago

London, England, United Kingdom 1 day ago

London, England, United Kingdom 1 week ago

London, England, United Kingdom 2 weeks ago

London, England, United Kingdom 1 week ago

City Of London, England, United Kingdom £75,000.00-£85,000.00 6 hours ago

London, England, United Kingdom 1 day ago

London, England, United Kingdom 1 week ago

Greater London, England, United Kingdom 3 weeks ago

London, England, United Kingdom 3 weeks ago

London, England, United Kingdom 5 days ago

City Of London, England, United Kingdom £60,000.00-£70,000.00 2 weeks ago

London, England, United Kingdom 1 day ago

London Area, United Kingdom £42,000.00-£45,000.00 2 hours ago

London, England, United Kingdom 1 week ago

London, England, United Kingdom 3 hours ago

London, England, United Kingdom 1 month ago

London, England, United Kingdom 2 days ago

London, England, United Kingdom 1 week ago

London, England, United Kingdom 1 week ago

London Area, United Kingdom £90,000.00-£90,000.00 8 hours ago

London, England, United Kingdom 1 week ago

London, England, United Kingdom 1 month ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.