Principal Engineer, Core Infrastructure

Ll Oefentherapie; Nashville (TN); On-site; USD 120,000 - 160,000; Full time

Job summary

TSO/AOCI is building Kilt, a platform that enables OCI services to be deployed and operated at scale. You will join the Observability & Supervision team to shape how the platform sees itself, collect metrics and logs, and act on live state to maintain SLOs.

We value engineers who enjoy solving hard problems, designing resilient systems, and shipping reliable automation. This role offers ownership, collaboration with a motivated team, and opportunities to drive real impact in production

Qualifications

• Designing distributed control loops and reasoning about performance and correctness at scale.
• Owning a component from design through production operation.
• Value simplicity and direct technical discussion.

Responsibilities

• Lead development and architect components of scalable, elastic distributed systems.
• Define scalability requirements and optimize data paths for high throughput.
• Establish telemetry, dashboards, and alerts for proactive operations.

Skills

Distributed systems

Observability

SRE principles

Distributed tracing

Job description

As part of the TSO/AOCI organization, our mission is to build Kilt, the platform on which OCI services are built, deployed, and operated across every region. We are investing in core platform capabilities that let service teams focus on their product rather than on the undifferentiated work of running it at scale, and we are doing it with a small group of engineers concentrated on high-leverage, low-level systems.

Within Kilt, the Observability & Supervision team owns how the platform sees itself and reacts. We build the systems that collect metrics and logs from every host and service, evaluate that signal against what healthy is supposed to look like, upscale to the right owner with the right context, and increasingly act on their own, whether that means scaling capacity, triggering recovery, or adjusting behavior based on the live state of upstream dependencies. This is real-time distributed systems work with strict latency budgets, where a decision has to be both fast and correct under partial information, because a wrong action taken across a fleet is worse than no action at all.

For this role, we are looking for engineers with a passion for understanding how systems fail, not only how they work. You should be comfortable designing distributed control loops, reasoning about performance and correctness at scale, and owning a component from design through production operation. You should value simplicity, be direct in technical discussion, and be excited by problems where the answer is not yet known.

We are a small, growing team with broad ownership and real autonomy. You will work with motivated, diverse people who care about doing things well, and you will have the support to do your best work in a workplace where you belong.Leads development and begins architecting components of scalable, elastic distributed systems. Defines and enforces scalability requirements for owned components; optimizes code and data paths for high‑throughput, hyper‑scale workloads; and leverages data plane platforms for large‑scale retrieval, storage, and processing. Designs fault‑tolerant, in‑service‑upgradable systems using redundancy, replication, failover, and policies for partitions, applying load‑shedding, throttling, and rate‑limiting to handle network unreliability while meeting SLOs. Establishes KPIs and telemetry; builds proactive dashboards and alerts; and designs complex validation (fault injection, brownouts), replication, and synchronization for correctness and durability. Proactively diagnoses and resolves production issues, mentors peers, and ensures operational readiness. Implements robust security controls, executes remediation, maintains compliance documentation, and develops IaC and automation that enable safe patching, updates, and rollbacks within change‑management plans.