MLOps Architect

Arkhya Tech Inc.; Charlotte (NC); On-site; USD 150,000 - 210,000; Full time

Job summary

Arkhya Tech Inc. seeks a Cortex Real-Time Inference Engineering Lead to design and industrialize a low-latency model serving platform for real-time predictive workloads.

You will bridge data science and production engineering, creating autoscaling deployment patterns and robust operational frameworks. The ideal candidate will have deep expertise in online inference architectures, container orchestration, and performance optimization to ensure sub-second latency and high availability across cloud

Qualifications

• Experience with real-time model evaluation lifecycles.
• Production-ready model serving and MLOps practices.
• Strong API design for high-performance services.

Responsibilities

• Design, build, and industrialize low-latency model-serving services for real-time workflows.
• Establish canary, blue/green, and shadow deployments for safe model updates.
• Define autoscaling and capacity controls to handle fluctuating traffic while minimising cloud spend.
• Profile and optimize performance to meet strict SLAs for latency and availability.
• Design monitoring, alerting, and logging for SLOs/SLIs and drift detection.
• Build CI/CD pipelines for testing, validation, packaging, and deployment of models.

Skills

Online Inference Architecture

Model Serving Frameworks

API Development

Programming languages (Python, Go, C/C

Education

Bachelor’s or Master’s degree in CS/SE or related field

Tools

Kubernetes

Triton Inference Server

TorchServe

TF Serving

vLLM

Seldon Core

KServe

Job description

We are seeking a highly skilled Cortex Real-Time Inference Engineering Lead to design, build, and industrialize our next-generation, low-latency model serving platform. In this role, you will bridge the gap between data science and production engineering. You will be responsible for creating resilient, autoscaling deployment patterns and robust operational frameworks that support real-time predictive use cases at scale.

The ideal candidate has deep expertise in online inference architectures, container orchestration, and performance optimization, ensuring our predictive models deliver high availability and sub-second latency.

Key Responsibilities

• Inference Platform Engineering: Design, build, and industrialize low-latency, highly available model-serving services and architecture for real-time predictive workflows.
• Deployment Patterns: Establish standardized deployment patterns (e.g., canary, blue/green, shadow deployments) to safely deploy and update machine learning models in production without downtime.
• Capacity & Autoscaling Control: Define capacity controls and implement advanced autoscaling strategies to handle highly fluctuating traffic patterns efficiently while minimizing cloud spend.
• Performance & Latency Optimization: Conduct continuous performance profiling, load testing, and optimization to meet strict service-level agreements (SLAs) for model execution and API response times.
• Monitoring & SLOs: Design and implement comprehensive monitoring, alerting, and logging systems to track model drift, data quality, system health, and Service Level Objectives (SLOs).
• CI/CD Automation: Build robust CI/CD pipelines to automate the testing, validation, packaging, and deployment of models and inference code.
• Resilience & Fault Tolerance: Build self-healing systems and implement fallback mechanisms to ensure high operational resilience and disaster recovery across both cloud and on-premises environments.

Required Skills & Qualifications

Core MLOps & Architecture

• Online Inference Architecture: Deep understanding of real-time model architectures, feature stores, and the lifecycle of online model evaluation.
• Model Serving Frameworks: Hands-on experience with production model servers such as Triton Inference Server, TorchServe, TF Serving, vLLM, Seldon Core, or KServe.
• API Development: High proficiency in designing and consuming high-performance APIs using gRPC, REST, or GraphQL.

Infrastructure & Operations

• Kubernetes Mastery: Strong experience deploying, managing, and scaling containerized workloads on Kubernetes (including microservices architecture).
• Autoscaling & Orchestration: Expertise in configuring horizontal pod autoscaling (HPA), cluster autoscaling, and custom metrics-driven scaling.
• Hybrid Operations: Proven track record managing deployments across both cloud providers (AWS, GCP, or Azure) and on-premises infrastructure.

Performance & Reliability

• Load Testing & Profiling: Experience using benchmarking tools like Locust, JMeter, or K6 to run performance, stress, and load testing.
• Latency Optimization: Knowledge of techniques to reduce inference latency, including model quantization, pruning, hardware acceleration (GPUs/TPUs), and optimized serialization.
• Observability: Experience building dashboards and alerts using Prometheus, Grafana, ELK stack, or Datadog to enforce tight SLOs/SLIs.
• CI/CD & Automation: Experience with GitOps and automation tools such as GitHub Actions, GitLab CI, ArgoCD, or Jenkins.

Preferred Qualifications

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related technical field.
• 5+ years of experience in Software Engineering or DevOps, with at least 3 years dedicated to MLOps and production ML pipelines.
• Strong programming skills in Python, Go, C++, or Java.
• Familiarity with machine learning frameworks like PyTorch, TensorFlow, or Scikit-Learn.