Data Engineer

Deeplight; Abu Dhabi; On-site; AED 180,000 - 240,000; Full time

Job summary

A leading AI consultancy in Abu Dhabi is seeking a data solutions architect to design and optimise scalable data solutions using Lakehouse architecture. The ideal candidate will have experience with MLOps principles, data ingestion techniques, and big data tools. This role offers competitive salary, health insurance, and opportunities for career advancement in an innovative environment.

Benefits

Competitive salary

Performance bonuses

Health insurance

Professional development support

Flexible working arrangements

Career advancement opportunities

Qualifications

• Experience designing and optimising solutions using Data Lakehouse architectures.
• Hands-on experience with data ingestion and management.
• Proficient in SQL and scripting languages like Python or Scala.
• Experience implementing MLOps pipelines for production systems.
• Strong background with major cloud platforms (AWS, Azure, GCP).
• Analytical and excellent communication skills.

Responsibilities

• Design and optimise data solutions using Lakehouse architecture.
• Develop and maintain data pipelines for diverse datasets.
• Implement data governance frameworks ensuring quality and compliance.
• Optimise storage and compute resources for BI and ML workloads.
• Develop CI/CD pipelines for machine learning lifecycle.
• Deploy and manage machine learning models in production.

Skills

Data Lakehouse architectures

Data ingestion

Big data technologies

MLOps pipelines

SQL and scripting languages

CI/CD tools

Cloud platforms (AWS, Azure, GCP)

Problem-solving

Tools

Databricks

Spark

Kafka

Docker

Kubernetes

TensorFlow Serving

MLflow

Job description

DeepLightAI is a specialist AI and data consultancy with extensive experience implementing intelligent enterprise systems across multiple industries, withparticular depthin financial services and banking. Our team combines deepexpertisein data science, statisticalmodeling, AI/ML technologies, workflow automation, and systems integration with a practical understanding of complex business operations.

Responsibilities of the role:

• Design, build, and optimises scalable data solutions, primarily utilising the Lakehouse architecture to unify data warehousing and data lake capabilities. Advise stakeholders on the strategic choice between Data Warehouse, Data Lake, and Lakehouse architectures based on specific business needs, cost, and latency requirements.
• Design, develop, and maintain scalable and reliable datapipelines to ingest, transform, and load diverse datasets from various sources, including structured and unstructured data, streaming data, and real‑time feeds.
• Implement standards and tooling to ensure ACID properties, schema evolution, and high data quality within the Lakehouse environment. Implement robust data governance frameworks (security, privacy, integrity, compliance, auditing).
• Continuously optimisedata storage, compute resources, and query performance across the data platform to reduce costs and improve latency for both BI and ML workloads, leveraging techniques such as indexing, partitioning, and parallel processing.
• Develop and maintain CI/CD pipelines to automate the entire machine learning lifecycle, from data validation and model training to deployment and infrastructure provisioning.
• Deploy, manage, and scale machine learning models into production environments, utilizing MLOps principles for reliable and repeatable operations.
• Establish and manage monitoring systems to track model performance metrics, detect data drift (changes in input data), and model decay (degradation in prediction accuracy).
• Ensure rigorous version control and tracking for all components: code, datasets, and trained model artifacts (using tools like MLflow or similar).
• Create comprehensive documentation, including technical specifications, data flow diagrams, and operational procedures, to facilitate understanding, collaboration, and knowledge sharing.

Qualifications

• Proven practical experience in designing, building, and optimising solutions using Data Lakehouse architectures (e.g., Databricks, Delta Lake).
• Strong hands‑on experience with managing data ingestion, schema enforcement, ACID properties, and utilizing big data technologies/frameworks like Spark and Kafka.
• Expertise in data modelling, ETL/ELT processes, and data warehousing concepts. Proficiency in SQL and scripting languages (e.g., Python, Scala).
• Demonstrated practical experience implementing MLOps pipelines for production systems. This includes a solid understanding and implementation experience with MLOps principles: automation, governance, and monitoring of ML models throughout the entire lifecycle.
• Experience with CI/CD tools, containerization/orchestration technologies (e.g., Docker, Kubernetes), model serving frameworks (e.g., TensorFlow Serving, SageMaker), and experiment tracking (e.g., MLflow).
• Experience with production monitoring tools to detect data drift or model decay.
• Strong hands‑on experience with major cloud platforms (e.g., AWS, Azure, GCP) and familiarity with DevOps practices.
• Excellent analytical, problem‑solving, and communication skills, with the ability to translate complex technical concepts into clear and actionable insights.
• Proven ability to work effectively in a fast‑paced, collaborative environment, with a passion for innovation and continuous learning.

Benefits & Growth Opportunities:

• Competitive salary and performance bonuses
• Comprehensive health insurance
• Professional development and certification support
• Opportunity to work on cutting‑edge AI projects
• Flexible working arrangements
• Career advancement opportunities in a rapidly growing AI company

This position offers a unique opportunity to shape the future of AI implementation while working with a talented team of professionals at the forefront of technological innovation. The successful candidate will play a crucial role in driving our company's success in delivering transformative AI solutions to our clients.