Senior ML Engineer - Inference and Model Serving

YAL; Hyderabad; Presencial; INR 3.000.000 - 6.000.000; Jornada completa

Descripción de la vacante

YAL is seeking a Senior ML Engineer to own how our language and speech models run in production. You will ensure fast, reliable inference on GPUs, applying optimisations and secure metering across the deployment pipeline.

You’ll work closely with architects on model-serving interfaces and AI engineers building agents on top, driving performance and scalability. The role requires 4–8 years of experience, strong Python and C++ skills, and hands-on production experience with language models and

Formación

• Proficient Python and C++ for production ML systems.
• Experience serving language models in production with engines such as vLLM, TensorRT-LLM, SGLang or Triton Inference Server.
• Strong understanding of GPU architecture and memory optimization for inference performance.

Responsabilidades

• Serve open-weight language models with low latency on GPUs using suitable inference engines.
• Implement continuous batching, caching, and multi-LoRA strategies to reduce cost per call.
• Manage GPU memory, quantisation, and parallelism to maximise throughput and reduce memory usage.
• Collaborate with AI engineers to optimize end-to-end latency from audio input to streaming output.

Conocimientos

Python

C++

Production ML

GPU optimization

Kubernetes

TensorRT

OpenTelemetry

Educación

Bachelor's/Master's in CS/EE or related

Herramientas

Nsight Systems

Nsight Compute

TensorRT

Kubernetes

MLflow

Hugging Face Hub

CUDA

Descripción del empleo

Senior ML Engineer - Inference and Model Serving

4 to 8 years

Hyderabad office

Full-Time

Role overview

We are hiring a senior ML engineer to own how our language and speech models run in production. Our

agents depend on fast, reliable inference: every spoken reply passes through speech recognition, a

language model and speech synthesis within a tight latency budget. You will serve these models on

GPUs at the lowest cost per call without losing quality, build the security and metering layers around

them, and automate how models move from registry to production. You will work with the architect on

model-serving interfaces and with the AI engineers who build agents on top.

Key responsibilities

• Serve open-weight language models with engines such as vLLM, TensorRT-LLM or SGLang, tuned for both time to first token and throughput.
• Apply continuous batching, prefix caching, speculative decoding and multi-LoRA serving to cut latency and cost per request.
• Manage GPU memory with PagedAttention, FlashAttention, KV-cache quantisation and tensor or pipeline parallelism, to maximise batch size and context length without out-of-memory.
• Quantise models (INT8, INT4, FP8) and choose precision per model based on measured quality and speed.

Speech model serving

• Serve streaming speech recognition and speech synthesis models on GPUs, with low time to first partial transcript and time to first audio.
• Batch and schedule many concurrent audio streams on shared GPUs without letting one stream delay another.
• Work with AI engineers to optimise end-to-end latency of the speech pipeline, from audio chunking to model execution to streaming output.
• Place speech and language models on GPUs to balance latency, utilisation and cost when they run side by side.

Performance and quality

• Profile models and serving stacks with tools such as NVIDIA Nsight Systems and Nsight Compute, then find and fix bottlenecks.
• Use compilers and runtimes such as TensorRT, TVM or PyTorch Inductor to fuse operations and reduce memory traffic.
• Check every optimisation against quality evals — task accuracy, word error rate and voice quality — and block any change that makes a model faster but worse.
• Benchmark candidate models and hardware on latency, throughput, quality and cost, and recommend what to serve.

Scale, cost and deployment

• Autoscale GPU capacity with traffic, plan capacity for peaks, and track cost per request, per call and per tenant.
• Deploy to cloud, on-premises and restricted-network environments, including air-gapped installs for enterprise clients.
• Build fallback paths across model sizes, replicas and providers, so traffic keeps flowing when a model or node fails.

Access, metering and MLOps

• Design secure model endpoints with OAuth2, JWT and role-based access control, and enforce strict multi-tenant isolation.
• Build high-throughput metering for input and output tokens, audio seconds and GPU time, feeding rate limits, billing and cost allocation.
• Build CI/CD pipelines that profile, regression-test and deploy models from registries such as MLflow or Hugging Face Hub into Kubernetes.
• Monitor latency, throughput, GPU utilisation and errors with OpenTelemetry and Prometheus, and respond to production incidents.

Required qualifications and skills

• Strong Python and working C++ skills.
• Hands-on experience serving language models in production with engines such as vLLM, TensorRT-LLM, SGLang or Triton Inference Server.
• Experience serving or optimising speech or other streaming models under strict real-time.
• Solid understanding of GPU architecture — streaming multiprocessors, memory hierarchy, asynchronous copies — and how it drives inference performance.
• Practical experience with quantisation, batching strategies, KV-cache management and model parallelism.
• Experience profiling and fixing performance problems with tools such as Nsight Systems or PyTorch Profiler.
• Experience running GPU workloads on Kubernetes, with CI/CD, monitoring and autoscaling.
• Ability to measure quality alongside speed, and to explain trade-offs between latency, throughput, cost and accuracy.
• A degree in computer science, computer or electrical engineering, or equivalent practical experience, with clear examples of delivered work.

Preferred experience

• Writing or modifying custom GPU kernels in CUDA or Triton, including kernel fusion.
• Deep familiarity with recent NVIDIA architectures such as Ampere, Hopper and Blackwell.
• Serving large models (70B+ parameters) across multiple GPUs or nodes under strict latency budgets.
• API gateways and rate limiting (Envoy, Kong), and streaming systems (Kafka, Redpanda).
• Air-gapped or on-premises GPU deployments for enterprise clients.
• Serving vision or multimodal models.
• A masters degree or PhD in a relevant field.

Success in the role

Success means our language and speech models respond fast enough for natural conversation, at a cost per call that scales with the business. In your first six months, that looks like:

• Latency: speech and language models meet their per-hop budgets at p95 under peak traffic.
• Cost: cost per call and GPU utilisation are tracked, and both improve measurably.
• Quality: no optimisation ships without passing quality evals, and quality never drops unnoticed.
• Reliability: model endpoints stay available through node and model failures, with fallbacks that work.
• Deployment: new models reach production through an automated, tested pipeline, in the cloud
• Metering: every tenants usage is measured accurately for rate limits and billing.