Real-Time ML Inference Platform Architect

Arkhya Tech Inc.; Charlotte (NC); On-site; USD 150,000 - 210,000; Full time

Job summary

Arkhya Tech Inc. seeks a Cortex Real-Time Inference Engineering Lead to design and industrialize a low-latency model serving platform for real-time predictive workloads.

You will bridge data science and production engineering, creating autoscaling deployment patterns and robust operational frameworks. The ideal candidate will have deep expertise in online inference architectures, container orchestration, and performance optimization to ensure sub-second latency and high availability across cloud

Qualifications

• Experience with real-time model evaluation lifecycles.
• Production-ready model serving and MLOps practices.
• Strong API design for high-performance services.

Responsibilities

• Design, build, and industrialize low-latency model-serving services for real-time workflows.
• Establish canary, blue/green, and shadow deployments for safe model updates.
• Define autoscaling and capacity controls to handle fluctuating traffic while minimising cloud spend.
• Profile and optimize performance to meet strict SLAs for latency and availability.
• Design monitoring, alerting, and logging for SLOs/SLIs and drift detection.
• Build CI/CD pipelines for testing, validation, packaging, and deployment of models.

Skills

Online Inference Architecture

Model Serving Frameworks

API Development

Programming languages (Python, Go, C/C

Education

Bachelor’s or Master’s degree in CS/SE or related field

Tools

Kubernetes

Triton Inference Server

TorchServe

TF Serving

vLLM

Seldon Core

KServe

Job description

Arkhya Tech Inc. seeks a Cortex Real-Time Inference Engineering Lead to design and industrialize a low-latency model serving platform for real-time predictive workloads.

You will bridge data science and production engineering, creating autoscaling deployment patterns and robust operational frameworks. The ideal candidate will have deep expertise in online inference architectures, container orchestration, and performance optimization to ensure sub-second latency and high availability across cloud