Software Development Engineer in Test - Agentic Automation

Jobgether; United States; Remote; USD 85.000 - 120.000; Vollzeit

Benefits dieser Stelle

Remote work across Canada

Professional development

Wellness programs

Equity

Zusammenfassung

Jobgether is seeking a Software Quality Engineer (SDET) to design, develop, and maintain automated test suites for distributed, cloud-based systems. The role emphasizes AI-enhanced testing, CI/CD integration, and robust evaluation frameworks to ensure reliability across services.

You will build AI-powered workflows, extend test frameworks, and collaborate across teams to improve tooling, coverage, and speed of releases. Remote work options are available for Canadian locations.

Qualifikationen

• Degree in Computer Science or equivalent practical experience.
• 5+ years in software quality with 3+ years SDET/test automation experience.
• Strong programming in Java, Node.js, or Python (Python preferred).
• Experience designing and maintaining test frameworks integrated into CI/CD (Jenkins, GitLab).

Aufgaben

• Design, develop, and maintain automated test suites for distributed, cloud-based systems.
• Extend agentic test generation and analysis in core testing frameworks.
• Build AI-powered workflows and agent skills to reduce manual testing; includes data setup and failure triage.
• Maintain evaluation suites, datasets, and scoring pipelines for AI agents across multiple metrics.
• Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration.
• Identify and mitigate unreliable AI outputs, including hallucinations and flaky tests.
• Design and execute performance, load, and stress tests for scalability of distributed services.
• Test integrations across platform systems, services, data, and customer-facing features.
• Review designs to assess testability and coverage; share knowledge to raise standards.
• Share technical knowledge and contribute to team-wide quality standards.

Kenntnisse

Java

Python

Node.js

SDET

Test automation

CI/CD

REST testing

gRPC testing

Locust

k6

AI evaluation

Data analysis

Ausbildung

CS degree

Tools

Jenkins

GitLab CI/CD

REST APIs

Jobbeschreibung

Accountabilities

• Design, develop, and maintain automated test suites and frameworks for distributed, cloud-based systems.
• Maintain and extend agentic test generation and analysis capabilities within core testing frameworks.
• Build AI-powered workflows and agent skills that reduce manual testing effort, including test data setup, failure triage, and release validation.
• Maintain evaluation suites, golden datasets, and scoring pipelines to assess AI agents across correctness, tool usage, execution trajectories, faithfulness, safety, latency, and cost.
• Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration before release, incorporating production failures into future evaluations.
• Identify and mitigate unreliable AI outputs, including hallucinated assertions, flaky or redundant tests, inaccurate failure analysis, and evaluation bias.
• Design and execute performance, load, and stress tests to identify bottlenecks and validate the scalability of distributed services.
• Test integrations across platform systems, services, data, applications, and customer-facing features.
• Review product, system, and agent designs to assess testability, evaluation coverage, and opportunities for improvement.
• Share technical knowledge and contribute to team-wide quality and evaluation standards.

Requirements:

• Degree in Computer Science or equivalent practical experience, with a solid understanding of object-oriented programming, data structures, algorithms, and distributed systems.
• At least 5 years of experience in software quality, including 3+ years of hands‑on SDET or test automation development.
• Strong programming skills in at least one of Java, Node.js, or Python; Python is preferred for AI evaluation and agent-related work.
• Demonstrated experience designing, developing, and maintaining test frameworks integrated into CI/CD pipelines such as Jenkins or GitLab CI/CD.
• Experience testing web services and APIs, including REST, gRPC, and JSON.
• Hands‑on performance and load testing experience with tools such as Locust or k6, including workload design and interpretation of latency percentiles, throughput, and error rates.
• A skeptical, verification-first approach to AI-generated output, with the ability to distinguish when deterministic assertions are appropriate and when probabilistic evaluation is needed.
• Experience applying AI technologies to improve decision‑making, streamline workflows, increase efficiency, and deliver business outcomes.
• Strong analytical, problem-solving, and collaboration skills, with the ability to communicate technical findings and contribute to shared quality standards.

Nice to have:

• End‑to‑end eval‑driven development experience, including release gating, trajectory and tool‑use evaluations, and calibrating LLM‑as‑judge systems against human review.
• Experience with prompt engineering and designing agent context or tools.
• Experience integrating performance tests into CI/CD pipelines and using observability tools such as Grafana, Prometheus, or CloudWatch.
• Background in cybersecurity, threat intelligence, malware analysis, or sandboxing and detonation environments.
• Familiarity with technologies such as MongoDB, MinIO, RabbitMQ, Kafka, AWS services (EC2, S3, DynamoDB), Docker, Nomad, Terraform, Unix/Linux, Git, Jenkins, GitLab CI/CD, Jira, and Bitbucket.
• Experience with LLM APIs, agent frameworks, and LLM evaluation tooling.

Benefits:

• Annual base salary ranging from CA$120,000 to CA$170,000 , plus variable or incentive compensation, equity, and benefits.
• Market-leading compensation and equity award opportunities.
• Comprehensive physical and mental wellness programs.
• Competitive vacation allowance and holidays.
• Paid parental and adoption leave.
• Professional development opportunities available to employees at every level.
• Employee networks, local community groups, and volunteer opportunities.
• Remote work opportunities across eligible Canadian location