Generative AI Engineer (LLM)

EPAM Systems; United States; Remote; USD 150,000 - 210,000; Full time

Job summary

EPAM Systems seeks a Generative AI Engineer to own improving the platform's language models—from fine-tuning to deployment. You will partner with Backend teams handling serving infra and collaborate with Data Analysts to define what a good recommendation looks like.

Responsibilities include data curation, synthetic data generation, and distillation for efficiency, with exposure to Vertex AI and GCS. Strong Python and English skills required.

Qualifications

• 5+ years of hands-on experience with Large Language Models (LLM) fine-tuning (SFT/PEFT/LoRA).
• Minimum 1 year of leadership experience.
• Proficiency with Transformers/PEFT and related tooling; experience with Vertex AI or equivalent.
• Solid understanding of training mechanics (tokenisation, packing, learning rate).
• Knowledge of catastrophic forgetting and GPU/TPU memory constraints.
• Expertise in LLM evaluation design and metrics.
• Data curation skills with high-quality dataset practices.
• Sustained LLMOps experience including versioning and drift monitoring.
• Strong Python skills for notebooks, training scripts, and eval pipelines.
• Excellent English communication; RLHF and distillation/quantisation knowledge are a plus.

Responsibilities

• Own model fine-tuning, evaluation, and deployment for retail next-best-action tasks.
• Manage dataset lifecycle, annotation workflows, deduplication, and eval set decontamination.
• Generate and curate synthetic data; version datasets used in training.
• Optimize cost/latency via distillation, quantisation, batching, and caching.
• Package and deploy tuned models on Vertex AI or equivalent; manage versions.
• Monitor drift and quality decay; define retraining triggers and rollback paths.
• Implement safety, PII controls, and output validation for enterprise data.
• Collaborate with commercial teams to translate expectations into training signals.

Skills

LLM fine-tuning

Leadership

Transformers

PEFT

Vertex AI

GCS

Python

English (written/spoken)

RLHF

Distillation

Quantisation

LLMOps

Model versioning

Drift monitoring

GPU memory optimization

Inference stacks

Security & PII controls

Tools

Hugging Face Transformers

Axolotl

Unsloth

Vertex AI endpoints

Job description

We are seeking a Generative AI Engineer to own how the platform's language models are improved — from fine-tuning and evaluation to deployment. You will partner with Back-End engineers who handle the serving infrastructure, and collaborate with Data Analysts and business stakeholders who define what a "good" recommendation actually looks like.

Responsibilities

• Run supervised fine-tuning and parameter-efficient adaptation (LoRA/QLoRA, adapters) and preference-based alignment (DPO, ORPO, RLHF-style methods) to shape tone, output structure, and domain reasoning for retail next-best-action and content-compliance tasks
• Own the dataset lifecycle — sourcing and curating examples from platform data and expert feedback, designing annotation and review workflows, deduplication, and decontamination against eval sets
• Generate and filter synthetic data, and version every dataset used in training
• Reduce cost and latency through distillation into smaller models, quantization, batching and caching strategy, and model routing — while holding quality above an agreed threshold
• Package and deploy tuned models on Vertex AI (or equivalent), and manage model and prompt versioning
• Monitor drift and quality decay in production, and define the retraining trigger and rollback path
• Implement and tune safety, scope, and PII controls appropriate to enterprise commercial data, including refusal behavior and output validation
• Partner with commercial teams to convert qualitative expectations ("this recommendation reads wrong") into training signal and measurable eval criteria

Requirements

• 5+ years of hands‑on experience with Large Language Models (LLM) fine‑tuning — SFT and PEFT/LoRA at minimum, ideally preference optimisation (DPO or similar)
• At least 1 year of relevant leadership experience
• Proficiency in Hugging Face Transformers/PEFT/TRL, Axolotl, or Unsloth, or managed tuning on Vertex AI / Bedrock / OpenAI
• Understanding of training mechanics, including tokenisation, context length and packing, and learning‑rate and hyper‑parameter choices
• Knowledge of catastrophic forgetting, over‑fitting on small domain sets, and GPU/TPU memory constraints
• Expertise in LLM evaluation — designing eval sets and metrics for generative tasks, with healthy scepticism about benchmark scores that don’t reflect user outcomes
• Skills in data curation with strong instinct for dataset quality over quantity — labeling guidelines, inter‑annotator agreement, and leakage control
• Familiarity with synthetic data pitfalls and filtering strategies
• Background in LLMOps, including model and prompt versioning, drift monitoring, and retraining lifecycle management
• Competency in Google Cloud Platform (Vertex AI, GCS, Cloud Run) and Git‑based workflows
• Sufficient Python skills to work in notebooks, training scripts, and eval pipelines, and to adapt existing framework code
• Excellent command of written and spoken English (B2+ level)
• Nice to have Experience with Reinforcement Learning from Human Feedback (RLHF) and preference-based alignment methods
• Familiarity with distillation and quantisation techniques (GPTQ, AWQ, GGUF, bitsandbytes) and their quality cost
• Understanding of inference stacks such as vLLM, TGI, and Vertex AI endpoints, including streaming and latency/throughput implications
• Knowledge of LLM security, including safety guardrails, scope controls, and PII handling for enterprise commercial data