Senior AI Data Engineer: LLM Pipelines & Vector Retrieval

Business Umbrella; Abu Dhabi; On-site; AED 446,000 - 670,000; Full time

Job summary

Business Umbrella seeks a senior Data Engineer in Abu Dhabi to build robust ingestion pipelines for structured and unstructured data using Python. You will prepare data for LLM fine-tuning and create high-quality datasets for training and evaluation.

You will implement dataset versioning with DVC or LakeFS, generate embeddings with HuggingFace or Sentence Transformers, and manage vector indexes with FAISS or Weaviate to optimize retrieval.

Qualifications

• 10+ years in Data Engineering
• 2+ years in AI-adjacent data roles
• Proficiency in Python, Pandas, and text processing tools
• Familiar with tokenization libraries (HuggingFace Tokenizers, SentencePiece)
• Experience managing datasets and object storage (MinIO, NFS)
• Understanding of LLM data constraints (context windows, formatting, prompt injection)

Responsibilities

• Build ingestion pipelines for structured/unstructured data using Python
• Clean, normalize, and prepare data formats for LLM fine-tuning (e.g. JSONL, CSV)
• Create high-quality task-specific datasets for training and evaluation
• Apply versioning to datasets using DVC or LakeFS for reproducibility
• Generate embeddings using HuggingFace or Sentence Transformers
• Manage vector indexes (FAISS, Weaviate) and optimize retrieval workflows
• Tokenize and chunk longform data for context window optimization

Skills

Python

Pandas

Text processing

Tools

HuggingFace Tokenizers

SentencePiece

MinIO

NFS

FAISS

Weaviate

DVC

LakeFS

Job description

Business Umbrella seeks a senior Data Engineer in Abu Dhabi to build robust ingestion pipelines for structured and unstructured data using Python. You will prepare data for LLM fine-tuning and create high-quality datasets for training and evaluation.

You will implement dataset versioning with DVC or LakeFS, generate embeddings with HuggingFace or Sentence Transformers, and manage vector indexes with FAISS or Weaviate to optimize retrieval.