VLM Engineer

Technology Innovation Institute; Dubai; On-site; AED 150,000 - 210,000; Full time

Job summary

Technology Innovation Institute (TII) is a publicly funded research institute based in Abu Dhabi, UAE, driving cutting-edge AI research. The Extreme-Scale Language Model team is advancing deep learning for vision-language tasks, including CLIP-based architectures, with strong emphasis on data and model evaluation.

The role focuses on ablation studies, data curation, and training of multimodal models, contributing to high-impact publications and collaborations with international institutions.

Qualifications

• PhD in deep learning is required.
• Proven track record in vision-language models and related research.
• Publication record in top-tier conferences highly desirable.

Responsibilities

• Conduct comprehensive ablation studies on vision models and report findings.
• Perform data ablation studies to optimize training data and inputs.
• Develop robust evaluation protocols for vision-language models.
• Engage in model training and integration of LLMs with vision models (e.g., CLIP).

Skills

Vision-language models

CLIP architectures

Distributed training

PyTorch

Model evaluation

Dataset curation

Education

PhD in deep learning

Job description

Job Description

Technology Innovation Institute (TII) is a publicly funded research institute, based in Abu Dhabi, United Arab Emirates. It is home to a diverse community of leading scientists, engineers, mathematicians, and researchers from across the globe, transforming problems and roadblocks into pioneering research and technology prototypes that help move society ahead.

Artificial Intelligence and Digital Research Centre

As part of TII’s Artificial Intelligence Research Center, the Extreme-Scale Language Model team is developing and implementing innovative deep learning technologies with a broad range of applications, from Natural Language Processing to Perception and Vision. Our team has developed the Falcon models and is now planning to continue our journey into cutting-edge applied research in the topic of large language models.

Key Responsibilities

• Vision Model Ablation Studies: Conduct comprehensive ablation studies on vision models to assess the impact of various components and configurations. Collaborate with researchers to analyze and report on the effectiveness of different model architectures and settings.
• Data Ablation Research: Partner with team members to perform data ablation studies, identifying optimal data types and structures for training vision-language models. Also, analyze the impact of different data inputs on model performance, particularly focusing on vision-language alignment.
• Model Evaluation: Develop and implement robust evaluation protocols for vision language models. Assess model performance across diverse benchmarks and real-world scenarios.
• Model Training and Optimization: Engage in model training, with an emphasis on integrating LLMs with vision models like CLIP. Technical

Skills Required

• Expertise in machine learning, particularly in vision-language models and LLMs.
• Strong understanding of model architectures like CLIP and their application in vision-language tasks.
• Proficiency in distributed training techniques and multi-GPU optimization.
• Experience with deep learning frameworks (e.g., "PyTorch").
• Strong analytical skills for conducting ablation studies and evaluating model performance.
• Familiarity with dataset curation and processing for vision and language tasks.

Qualifications

• PhD degree in deep learning.
• Proven track record of research and development in vision-language models.
• Publication record in top-tier conferences is highly desirable

At TII, we help society to overcome its biggest hurdles through a rigorous approach to scientific discovery and inquiry, using state-of-the-art facilities and collaboration with leading international institutions. Our rigorous discovery and inquiry-based approach helps to forge new and disruptive breakthroughs in AI, advanced materials, autonomous robotics, cryptography, digital security, directed energy, quantum computing and secure systems.

Requirements