Director Production Engineering

Vista Applied Solutions Group Inc; Durham (NC); On-site; USD 130,000 - 160,000; Full time

Job summary

A top-tier software engineering firm seeks an experienced individual to lead production engineering efforts in ensuring system reliability and performance. The role includes setting non-functional criteria, overseeing incident analysis, and driving resilience testing for complex systems. Candidates should possess a Bachelor's in a relevant field, extensive experience in production engineering, and strong leadership skills. This position is critical in embedding reliability in engineering practices and leading incident prevention initiatives.

Qualifications

• 10–15+ years building production engineering capabilities for distributed software platforms.
• Experience defining production readiness standards.
• Strong knowledge of failure modes and degradation behavior.

Responsibilities

• Own non-functional release criteria for reliability and performance.
• Lead blameless incident reviews and ensure corrective actions.
• Drive chaos and load testing for real failure modes.

Skills

Production engineering capabilities

Distributed systems fundamentals

Incident analysis

Education

Bachelor’s degree in Computer Science, Engineering, or equivalent

Tools

Kubernetes/AKS

Modern observability platforms

Job description

Responsibilities

• Own non-functional release criteria and automated release gates for reliability, resilience, performance, and correctness across complex release trains.
• Define and enforce Production Readiness Reviews (PRRs) and platform-wide engineering standards.
• Establish objective, measurable “safe-to-release” signals consumed by CI/CD and release tooling.
• Partner with Architects and Principal Engineers to define failure modes, degradation behavior, and system guardrails for distributed and eventually consistent workflows.
• Ensure systems behave correctly during retries, partial outages, intermittent connectivity, degraded modes, and recovery.
• Lead engineering initiatives that reduce risk of data loss, duplication, corruption, or inconsistent state across POS, middleware, and cloud services.

Incident Learning That Prevents Recurrence

• Lead blameless incident reviews using formal analysis methods.
• Ensure corrective actions are engineered into systems, validated, tracked, and audited.
• Institutionalize learning so failures do not reappear under new conditions or scale.

Resilience & Performance Engineering

• Own platform-level strategies for resilience, performance, and scalability validation.
• Drive chaos, failover, load, stress, and soak testing focused on real failure modes, not synthetic demos.
• Validate store-mode behavior, payment workflows, edge-device dependencies, and multi-service interactions.

Observability & Reliability Signals

• Ensure high-fidelity telemetry (logs, metrics, traces, and business signals) that supports release gating, correctness verification, and diagnosis.
• Drive instrumentation standards that allow teams to prove reliability outcomes with data.

Cross-Org Technical Leadership

• Partner with Software Engineering, Architecture, Quality Engineering, Cloud Operations, and TPM/TPO teams.
• Build and lead senior technical managers and staff-level engineers.
• Set expectations for technical depth, ownership, and execution quality.

Required Experience

• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.
• 10–15+ years building production engineering capabilities for distributed software platforms, with direct accountability for production outcomes.
• Demonstrated experience defining and enforcing production readiness standards and non-functional release gates that prevent unsafe changes from shipping.
• Proven ability to lead formal root-cause / reliability analysis and ensure systemic fixes reduce recurrence.
• Strong distributed systems fundamentals, including the ability to reason about:
• Failure modes and degradation behavior
• Dependency risk, retries, and backpressure
• Consistency tradeoffs and correctness under failure
• Experience partnering deeply with Architecture and Software Engineering to embed reliability guardrails into design reviews, CI/CD pipelines, and system standards.
• Senior leadership experience building teams and influencing across large engineering organizations.

Preferred Experience

• Designing reliability automation (release scoring, regression detection, incident pattern analysis).
• Hybrid cloud + edge architectures; Kubernetes/AKS; modern observability platforms.
• Leading reliability transformations in large, complex engineering organizations.