Production Engineer (IC4)

Ontrac Solutions LLC.; Northern (KY); Hybrid; USD 90,000 - 140,000; Full time

Job summary

Ontrac Solutions LLC. seeks a Production Engineer (IC2) to support enterprise OS modernization, packaging migrations, and CI/CD hardening.

The role requires deep Python software foundations and hands-on experience across RHEL7–EL9 upgrades, packaging, and automation. The candidate will own runbooks, contribute to monitoring/logging migrations, and work with SRE teams to ensure reliable deployments and rapid incident response.

Qualifications

• 3+ years of professional software engineering experience with hands-on Python and code ownership
• Experience with enterprise Linux fleet-scale and OS upgrade programs (RHEL7 to EL8/EL9)
• Experience building RPM packages to replace legacy configurations and with large-scale packaging migrations (Chef to CINC)
• Independent bug ownership: triage, reproduce, fix, test, and ship without hand-holding
• Ability to harden CI/CD pipelines and rollout/rollback mechanisms for migrations

Responsibilities

• Transition monitoring infrastructure to modern stacks (Chronosphere, Prometheus, Grafana) and manage Splunk integrations
• Onboard services to new monitoring/logging stacks, including metrics, dashboards, and alerts
• Harden observability frameworks and ensure regressions are caught before users are affected
• Harden CI/CD pipelines for legacy-to-modern transitions, including build and test stages
• Design rollout and rollback mechanisms to make large-fleet changes reversible
• Provide Tier-2 operational support and incident response with the client’s SRE team
• Document runbooks, migration procedures, and packaging/pipeline ownership notes

Skills

Python development

Linux administration

CI/CD pipelines

Incident response

Automation

Education

Bachelor's degree in CS or related

Tools

rpmbuild

Koji

Mock

Chef

CINC

Job description

Overview

Ontrac Solutions is seeking a high-aptitude Production Engineer (IC2) to support a large-scale enterprise OS modernization and infrastructure hardening program for one of our enterprise clients. This role is built for an engineer with real software engineering foundations - specifically Python - who has since moved into infrastructure and is comfortable working across OS modernizations (RHEL7 - EL8/EL9), packaging migrations (Chef - CINC), and CI/CD hardening, while simultaneously executing hands‑on runbooks, automation, and service onboarding.

What your application must clearly show

We screen against the requirements below exactly as written - your resume should make these easy to find. Specifics matter more than vocabulary: a resume that restates this posting's terminology without the detail below will not advance.

• Python you actually wrote, described as software, not as a skills keyword. Name the project, what it did, who depended on it, and how it was tested and shipped. "Python (scripting)" in a skills list will not clear this bar. A GitHub, GitLab, or public repo link is strongly preferred - we look at code.
• RPM packaging you personally did. Name the .spec files you authored or maintained, how you handled dependencies and versioning, your build tooling (rpmbuild, mock, Koji, or an internal builder), roughly how many packages you owned, and where they were published.
• A real OS migration you worked on - with version numbers and what actually broke. System Python 2→3, OpenSSL and crypto-policy changes, systemd unit differences, deprecated or renamed packages. We are more interested in the failure modes you hit than in the name of the program.
• Configuration management you have run in production - Chef (cookbooks, recipes, Ohai, Test Kitchen/InSpec), CINC, Puppet, Ansible, or Salt. Say which resources you wrote and how you tested convergence.
• Monitoring and logging work you executed. Name the stack, what you onboarded to it, and what you actually instrumented - metrics, dashboards, alert rules, log pipelines - not just the product name.
• Tier-2 or on-call experience: the rotation you carried, the scale of the fleet behind it, and one incident you personally drove to resolution.
• CI/CD pipelines you built or hardened, including how rollout and rollback were handled when a change went wrong.
• Your certifications, named, with dates and credential IDs or verification links - we verify certifications.

Required Qualifications

• Software engineering foundation: 3+ years of professional software engineering experience, with strong hands-on Python. You have written and maintained code other engineers depended on - modules, packaging, tests, code review, and version control, not just single-file scripts.
• Enterprise OS modernization: Hands-on experience with enterprise Linux at fleet scale and with large-scale OS upgrade programs - RHEL7 - EL8/EL9 or an equivalent major-version migration you executed rather than observed.
• Packaging migrations: Experience building RPM packages to replace legacy configuration, and with large-scale packaging or configuration-management migrations such as Chef - CINC.
• Independent bug ownership: Able to triage, own, and resolve bugs end to end without hand-holding - reproduce, isolate, fix, test, and ship.
• CI/CD and release safety: Ability to harden CI/CD pipelines, observability frameworks, and rollout/rollback mechanisms specifically tailored for legacy-to-modern infrastructure transitions.
• Tier-2 operational support: Willingness and experience to partner closely with an SRE team providing "follow-the-sun" tier-2 support, including hands-on incident response and break/fix operations on existing platforms.
• Service onboarding: Experience onboarding services to newly established monitoring and logging stacks.
• Automation and documentation: A demonstrated habit of automating repetitive operations and documenting technical procedures for others to run.
• Location and work authorization: Must be located in the United States and authorized to work in the US.

Preferred Qualifications

• Proven experience planning and executing logging and monitoring tool rollouts end to end - not only operating a stack someone else stood up.
• Experience supporting a team through cloud cutovers and component migrations to cloud environments.
• Perl scripting experience - legacy tooling in this environment is Perl, and the ability to read and safely modify it is a real advantage.
• Hands-on exposure to modern observability tooling - Chronosphere, Prometheus, or Grafana - and to Splunk integrations.
• Provisioning and image work: Kickstart/PXE, golden images, or repository and mirror management.
• Drive the technical transition of legacy systems to modern enterprise Linux environments, including RHEL7 - EL8/EL9 upgrade paths.
• Build and maintain RPM packages to replace legacy configuration, and carry the packaging migration from Chef to CINC.
• Develop and execute automated runbooks that make the migration repeatable rather than manual.
• Triage, own, and resolve migration bugs independently, from first report through verified fix.

Key Responsibilities - Observability & Monitoring Transition

• Transition monitoring infrastructure to a modern stack - Chronosphere, Prometheus, and Grafana - and manage Splunk integrations.
• Onboard services to the newly established monitoring and logging stacks, including metrics, dashboards, and alert rules.
• Harden observability frameworks alongside the pipelines they instrument, so regressions surface before users find them.

Key Responsibilities - CI/CD, Rollout & Rollback

• Harden CI/CD pipelines for legacy-to-modern infrastructure transitions, including build, test, and package promotion stages.
• Design and maintain rollout and rollback mechanisms that make large-fleet changes reversible.
• Automate repetitive operational work and replace manual runbook steps with tested, reviewed code.

Key Responsibilities - Tier-2 Operations & Migration Support

• Provide Tier-2 operational support and incident response under a follow-the-sun model, in close partnership with the client's SRE team.
• Perform hands-on break/fix operations on existing platforms while the modernization proceeds in parallel.
• Assist application developers with architectural support and troubleshooting during cloud migration phases.
• Author and maintain technical documentation - runbooks, migration procedures, and package and pipeline ownership notes.

__________________________________