Senior Data Engineer

Open Innovation AI; Abu Dhabi; On-site; AED 240,000 - 400,000; Full time

Job summary

Open Innovation AI in Abu Dhabi seeks a Senior Data Engineer to advance our MLOps platform and deploy Large Language Models. You will maintain scalable data infrastructure and enhance data pipelines for efficient AI workloads.

Join a global technology team focusing on reducing costs and accelerating value from AI initiatives while collaborating across stakeholders and cloud environments in a fast-paced setting.

Qualifications

• Bachelor’s or master’s degree in CS, Engineering or related field.
• Hands-on with Apache Airflow and ClickHouse; proficient in Python/Java/C++/Scala.
• Experience as a Data Engineer focused on MLOps platforms and big data technologies.

Responsibilities

• Design, construct, install, test, and maintain scalable data management systems.
• Contribute to the development and enhancement of the MLOps platform.
• Deploy and manage Large Language Models (LLMs) for scalable workloads.
• Develop datasets for data modeling, mining, and production processes.
• Automate deployment of machine learning models with data pipelines.

Skills

Python

SQL

NoSQL

Big data

Education

Bachelor’s or Master’s degree in CS/Engineering

Tools

Apache Airflow

ClickHouse

Java

C++

Scala

Cloud services (AWS, Azure, GCP)

Job description

Company Description

Open Innovation AI is a global technology company that specializes in developing advanced solutions for managing AI workloads. Its flagship product, the Open Innovation Cluster Manager (OICM), orchestrates complex AI tasks efficiently across diverse infrastructures. The platform is hardware-agnostic, optimized for various GPUs and accelerators hardware, and facilitates seamless integration and scalability for enterprise AI applications. Open Innovation AI focuses on optimizing and simplifying AI workload management and making AI technologies accessible to organizations of all sizes. With its innovative solutions, companies can reduce operational costs, accelerate time to value, and maximize their return on investment, ensuring that their AI strategies contribute directly to enhanced business outcomes.

Role Overview:

Open Innovation AI is seeking an experienced Senior Data Engineer to contribute to the development of our MLOps platform and to work on deploying Large Language Models (LLMs). This role involves both maintaining scalable data infrastructure and actively participating in the enhancement of our machine learning operations.

Role Requirements:

• Design, construct, install, test, and maintain scalable data management systems
• Actively contribute to the development and enhancement of our MLOps platform
• Work on deploying and managing Large Language Models, ensuring efficiency and scalability
• Develop data set processes for data modeling, mining, and production
• Integrate new data management technologies and software engineering tools into existing structures
• Collaborate with the engineering team and other stakeholders in automating the deployment of machine learning models
• Recommend improvements to data reliability, efficiency, and quality
• Research industry trends and apply this knowledge to data infrastructure projects
• Utilize data to identify tasks that can be automated and streamline processes

Required Qualifications

• Bachelor’s or master’s degree in computer science, Engineering, or a related field
• Hands-on experience with Apache Airflow and ClickHouse (database), along with proficiency in programming languages such as Python, Java, C++, or Scala
• Proven experience as a Data Engineer with a focus on MLOps platforms and big data technologies
• Experience in deploying and managing Large Language Models (LLMs)
• Strong proficiency in SQL and NoSQL databases, data warehousing solutions
• Skilled in data pipeline, data modeling, ETL and workflow management tools
• Knowledge of cloud services (AWS, Azure, GCP) and their application in data engineering
• Excellent problem-solving skills and ability to work in a collaborative environment
• Strong project management and organizational skills