Remote AI Software Engineer: LLM Evaluation & Automation

Stage 4 Solutions Inc; United States; Remote; USD 99,000 - 108,000; Full time

Benefits offered by this job

Health benefits

401K

Job summary

Stage 4 Solutions Inc. seeks an AI Software Engineer to build evaluation harnesses and automation for AI software use cases. The role focuses on turning real engineering artifacts into repeatable benchmark tasks, with reproducible runs and controlled environments.

The position is remote in the United States, 6 months with potential extension, 40 hours per week, W2 employee benefits including health and 401K. Must align to Pacific Time hours.

Qualifications

• Strong software engineering background with automation/test/validation experience.
• Proficient in Python and at least one of Java or JavaScript.
• Familiar with containers and Git.
• Experience with APIs, CI/CD, and engineering workflows.
• Understanding of AI, LLM, or agent evaluation concepts.
• Able to troubleshoot and analyze results carefully.

Responsibilities

• Build and integrate evaluation harnesses and automation for software development use cases.
• Create versioned, repeatable evaluation processes with reproducible environments.
• Validate evaluation approaches against human judgment for consistency.
• Support benchmarking across quality, productivity, and efficiency, including cost and latency.
• Analyze results across repeated runs to improve workflow reliability and automation.
• Collaborate with engineering and data teams and document methods and findings.

Skills

Python

Java

JavaScript

Git

Docker

APIs

CI/CD

AI evaluation

Troubleshooting

Automation

Tools

Docker

Git

Job description

Stage 4 Solutions Inc. seeks an AI Software Engineer to build evaluation harnesses and automation for AI software use cases. The role focuses on turning real engineering artifacts into repeatable benchmark tasks, with reproducible runs and controlled environments.

The position is remote in the United States, 6 months with potential extension, 40 hours per week, W2 employee benefits including health and 401K. Must align to Pacific Time hours.