Inference Lead

Compunnel, Inc.; Charlotte, Northern (NC, KY); Hybrid; USD 170,000 - 210,000; Full time

Job summary

Compunnel, Inc. is seeking a Real-Time Inference Engineering Lead to design and industrialize low-latency model-serving services for predictive AI use cases.

You will define deployment patterns, capacity controls, monitoring, and high-availability practices across cloud and on-premises environments. The role requires strong expertise in real-time inference architecture, distributed services, Kubernetes, performance engineering, CI/CD, and technical leadership to mentor inference and platform

Qualifications

• 8–12 years of professional experience in software engineering, inference engineering, platform engineering, ML engineering, SRE, or related technical disciplines.
• Strong experience with online inference and real-time model-serving architecture.
• Hands-on experience designing REST and/or gRPC APIs and distributed microservices.
• Strong experience with Kubernetes, containers, autoscaling, and traffic management.
• Experience with performance engineering, latency optimization, benchmarking, and load testing.
• Experience with monitoring, SLOs, capacity planning, and production operations.
• Strong experience building CI/CD pipelines and implementing progressive-deployment approaches.
• Experience designing resilient, highly available systems, including failover, rollback, disaster recovery, and graceful degradation.
• Experience deploying and operating solutions across both cloud and on-premises environments.
• Strong technical leadership skills with experience conducting technical reviews and mentoring engineering teams.

Responsibilities

• Architect low-latency online inference and real-time model-serving solutions.
• Develop scalable APIs, microservices, and deployment patterns for predictive models.
• Implement Kubernetes-based deployment, autoscaling, load balancing, and traffic-management strategies.
• Conduct benchmarking, performance tuning, capacity planning, and load testing.
• Optimize latency, throughput, resource consumption, availability, and operational cost.
• Define monitoring, alerting, SLOs, runbooks, and incident-response practices.
• Build CI/CD pipelines to support repeatable model and service releases.
• Design resilience, failover, disaster recovery, and graceful-degradation patterns.
• Lead technical reviews and provide mentorship to inference and platform engineers.
• Establish operational and performance standards for real-time model-serving services across cloud and on-premises environments.

Skills

Real-time inference

REST/gRPC APIs

Performance engineering

SRE/production ops

Technical leadership

Education

Bachelor's degree in Computer Science or related

Tools

Kubernetes

Containers (Docker)

Autoscaling

Load balancing

CI/CD pipelines

Job description

The Real-Time Inference Engineering Lead will design and industrialize low-latency, resilient model-serving services for predictive AI use cases. The role will define deployment patterns, capacity controls, monitoring, performance standards, and operational practices across cloud and on-premises environments. This position requires strong expertise in real-time inference architecture, distributed services, Kubernetes, performance engineering, production operations, CI/CD, and high-availability engineering, along with the ability to provide technical leadership and mentorship to inference and platform engineering teams.

Key Responsibilities

• Architect low-latency online inference and real-time model-serving solutions.
• Develop scalable APIs, microservices, and deployment patterns for predictive models.
• Implement Kubernetes-based deployment, autoscaling, load balancing, and traffic-management strategies.
• Conduct benchmarking, performance tuning, capacity planning, and load testing.
• Optimize latency, throughput, resource consumption, availability, and operational cost.
• Define monitoring, alerting, Service Level Objectives (SLOs), runbooks, and incident-response practices.
• Build CI/CD pipelines to support repeatable model and service releases.
• Design resilience, failover, rollback, disaster recovery, and graceful-degradation patterns.
• Lead technical reviews and provide mentorship to inference and platform engineers.
• Establish operational and performance standards for real-time model-serving services across cloud and on-premises environments.

Required Qualifications

• 8–12 years of professional experience in software engineering, inference engineering, platform engineering, ML engineering, SRE, or related technical disciplines.
• Strong experience with online inference and real-time model-serving architecture.
• Hands-on experience designing REST and/or gRPC APIs and distributed microservices.
• Strong experience with Kubernetes, containers, autoscaling, and traffic management.
• Experience with performance engineering, latency optimization, benchmarking, and load testing.
• Experience with monitoring, SLOs, capacity planning, and production operations.
• Strong experience building CI/CD pipelines and implementing progressive-deployment approaches.
• Experience designing resilient, highly available systems, including failover, rollback, disaster recovery, and graceful degradation.
• Experience deploying and operating solutions across both cloud and on-premises environments.
• Strong technical leadership skills with experience conducting technical reviews and mentoring engineering teams.

Preferred Qualifications

• Degree in Computer Science, Engineering, or a related discipline.
• Experience with enterprise model-serving platforms and inference runtimes.
• Cloud, Kubernetes, SRE, or ML engineering certification.