Data Engineer - Scalable AI Data Pipelines

Discovered MENA; Abu Dhabi; On-site; AED 240,000 - 420,000; Full time

Job summary

Discovered MENA in Abu Dhabi is seeking an experienced Data Engineer to design and build scalable data infrastructure and ETL/ELT pipelines that underpin AI and ML initiatives across a major financial services client.

You will collaborate with AI/ML and software teams, ensure data quality, governance and security, and document architecture and lineage for reliable data access at scale.

Qualifications

• 5+ years of experience in Data Engineering, ETL development, data warehousing or similar positions.
• Strong hands‑on experience designing and managing ETL/ELT workflows using technologies such as Airflow, dbt, Luigi or similar.
• Excellent Python and SQL skills, alongside experience with Scala, Java or another relevant programming language.
• Experience working with big data architectures and optimising large-scale data pipelines.
• Hands‑on experience with cloud data environments across Azure, AWS or GCP
• Experience processing and managing large-scale structured and unstructured datasets
• Strong understanding of data quality, governance, monitoring and engineering best practices
• Excellent problem‑solving skills with the ability to diagnose performance and data issues
• Bachelor’s degree in Computer Science, Engineering or a related technical discipline

Responsibilities

• Design, build and optimise scalable ETL/ELT pipelines to ingest, process and transform structured and unstructured data
• Develop reliable data infrastructure capable of supporting large-scale AI, machine learning and analytics workloads
• Continuously test, monitor and optimise data pipelines to improve performance, reliability and data integrity
• Work closely with AI/ML and software engineering teams to make high-quality data easily accessible for downstream applications
• Implement strong data quality, validation, governance and security practices across the data lifecycle
• Maintain clear documentation around data architecture, lineage and metadata

Skills

Python

SQL

ETL pipelines

Airflow

dbt

Luigi

Scala/Java

Big data

Azure / AWS / GCP

Data governance

Problem solving

Education

Bachelor's degree in Computer Science, Engineering or related field

Tools

Airflow

dbt

Luigi

Python

SQL

Scala

Java

Azure

AWS

GCP

Job description

Discovered MENA in Abu Dhabi is seeking an experienced Data Engineer to design and build scalable data infrastructure and ETL/ELT pipelines that underpin AI and ML initiatives across a major financial services client.

You will collaborate with AI/ML and software teams, ensure data quality, governance and security, and document architecture and lineage for reliable data access at scale.