Infrastructure System Engineer

Berkley Hunt; New York (NY); On-site; USD 150,000 - 210,000; Full time

Job summary

Berkley Hunt is partnering with a well-funded startup building next-gen infrastructure for high-performance, low-latency LLM inference at scale. This founding-level role focuses on low-level systems, distributed environments, and mission-critical performance challenges.

The engineer will own global routing, backend infra, and security controls, shaping edge-based inference and scalable production systems across global locations.

Qualifications

• Experience building distributed, concurrent systems focused on speed and reliability.
• Deep expertise in systems programming (Rust, C++, or Zig).
• Strong Python skills for automation and control-plane development.
• Experience designing infrastructure for LLM inference or high-scale CDN environments.
• Proficiency with AWS, Postgres, Redis, and Kafka.
• Familiarity with Linux internals, performance tuning, and kernel-level debugging.
• Observability tools such as Zipkin or Jaeger.
• Clear, concise communicator with a strong documentation and operational mindset.

Responsibilities

• Architect and implement global routing engines with microsecond decision times.
• Build and manage high-performance backend infrastructure from scratch, focusing on concurrency, throughput, and resilience.
• Own the control plane, monitoring systems, and production documentation for all deployments.
• Lead efforts in internal security and access management systems.
• Design and optimize infrastructure for edge-based inference with maximum hardware utilization and minimal latency.
• Collaborate cross-functionally to drive infrastructure decisions from prototype to production.

Skills

Distributed systems

Systems programming

Python scripting

Cloud services

Observability

Tools

Kafka

PostgreSQL

Redis

AWS

Zipkin/Jaeger

Job description

Berkley Hunt has partnered with a well-funded, cutting-edge startup building next-gen infrastructure to power high-performance, low-latency LLM inference at scale. This team is rethinking how to optimize global server utilization and request routing for AI workloads, enabling real-time execution across edge locations and novel hardware.

This is a founding-level opportunity for an infrastructure engineer who thrives in low-level systems, distributed environments, and mission-critical performance challenges.

Role Responsibilities

• Architect and implement global routing engines that intelligently distribute LLM inference requests with microsecond decision times.
• Build and manage high-performance backend infrastructure from scratch, focusing on concurrency, throughput, and resilience.
• Own the control plane, monitoring systems, and production documentation for all deployments.
• Lead efforts in internal security and access management systems.
• Design and optimize infrastructure for edge-based inference with maximum hardware utilization and minimal latency.
• Collaborate cross-functionally to drive infrastructure decisions from prototype to production.

Requirements

• Extensive experience building distributed, concurrent systems optimized for speed and reliability.
• Deep expertise in systems programming ("Rust, C++, or Zig preferred").
• Strong Python skills for automation and control plane development.
• Proven experience designing infrastructure for LLM inference or high-scale CDN environments.
• Proficiency with AWS, Postgres, Redis, and event streaming tools like Kafka.
• Comfort with Linux internals, performance tuning, and kernel-level debugging.
• Familiarity with observability tools such as Zipkin or Jaeger.
• Clear, concise communicator with a strong documentation and operational mindset.