Senior Test Automation Engineer

Metrosion Inc.; Burlingame (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Metrosion Inc. is hiring a senior test automation engineer to own automation for AI-enabled features across interfaces, APIs, and workflows.

You will lead test strategy, mentor teams, and ensure reliable release checks while collaborating with software engineers, data scientists, product managers, and designers. You will build and maintain robust regression coverage, harness AI evaluation datasets, and drive testability improvements, with a focus on reducing flaky tests and providing actionable

Qualifications

• Requires at least 6 years in test automation or software quality engineering.
• Experience at an AI-forward company is required.
• Strong programming skills in TypeScript and Go with production test automation.
• Solid understanding of test isolation, fixtures, selectors, and flaky/test reliability issues.

Responsibilities

• Develop a risk-based test strategy for critical workflows and release acceptance.
• Integrate checks into CI/CD with reproducible environments and meaningful failure artifacts.
• Test authentication, authorization, tenant isolation, approval flows, retries, and recovery.
• Collaborate with data scientists and engineers on AI evaluation datasets and thresholds.
• Exercise failure conditions like unavailable dependencies, timeouts, and interrupted workflows.
• Plan and automate performance and resilience checks against latency and load expectations.
• Investigate failures using logs, traces, screenshots, and reproducible examples; distinguish defects from env issues.
• Reduce flaky tests by addressing root causes and improving execution time.
• Communicate coverage, defects, and remaining risks for release readiness.
• Review AI-generated tests for correct assertions and secure data handling.

Skills

TypeScript

Go

Test automation

AI experience

Tools

Docker

CI/CD tooling

Job description

Own reliable test automation and release-quality evidence for AI software used in regulated industries and critical operations.

The opportunity

Build the automation that helps us ship with confidence. Turn important customer workflows, security boundaries, and operational failure modes into repeatable tests that find consequential defects and give teams useful feedback.

Work alongside software engineers, data scientists, product managers, and designers to make quality part of development from the start. Own automation across interfaces, APIs, integrations, and AI-assisted workflows, with clear evidence of what is covered, what failed, and what remains uncertain.

Independently lead the test strategy and automation for substantial product capabilities. Choose effective test boundaries, improve testability with engineers, and connect automated checks to meaningful release decisions. Remain hands‑on with framework development, test implementation, and failure investigation while mentoring colleagues in reliable testing practices.

Success requires sound risk assessment and dependable engineering, including knowing which behavior belongs in a deterministic test, which needs an AI evaluation, and where exploratory testing or human review adds value.

What you will do

• Develop a risk-based test strategy covering critical workflows, integration boundaries, negative scenarios, and release acceptance criteria.
• Integrate fast, reliable checks into continuous integration and delivery; maintain reproducible environments, isolated test data, and useful failure artifacts.
• Test authentication, authorization, tenant isolation, approval flows, cancellation, retries, and recovery with explicit expected behavior.
• Partner with data scientists and engineers on representative AI evaluation datasets, repeatable regression runs, and quality thresholds for model-assisted features.
• Exercise failure conditions such as unavailable dependencies, malformed responses, timeouts, partial completion, and interrupted workflows.
• Plan and automate performance and resilience checks against agreed latency, load, and recovery expectations.
• Investigate failures using logs, traces, screenshots, network evidence, and reproducible examples; distinguish product defects from test or environment problems.
• Reduce flaky tests by addressing root causes, improve execution time and maintainability, and keep release gates trustworthy.
• Communicate coverage, defects, unresolved risks, and release readiness clearly; help turn production incidents and customer issues into regression coverage.
• Use AI coding tools effectively while reviewing generated tests for correct assertions, realistic scenarios, secure data handling, and actual defect detection.

What you bring

• Senior level: at least 6 years of relevant professional experience in test automation, software development in test, or software quality engineering.
• Prior professional experience working at an AI-forward company is required.
• Strong programming skills in TypeScript and Go are required, with a record of building and maintaining production test automation.
• Strong understanding of test isolation, fixtures, selectors, asynchronous behavior, test data, and the causes of flaky or misleading test results.
• Experience building continuous integration pipelines, working with containerized environments, and maintaining reliable automated release checks.
• Practical knowledge of HTTP, APIs, SQL, authentication, authorization, and the failure modes of distributed applications.
• Experience testing AI-enabled features and understanding why model behavior, evaluation datasets, and variability require care beyond conventional pass-or-fail assertions.
• Ability to prioritize coverage by user impact and risk, investigate failures systematically, and explain the limits of test evidence.
• Clear technical communication and examples of improving testability, reducing escaped defects, or making automation more reliable and useful.

Success in this role

• Critical workflows and permission boundaries have meaningful, repeatable coverage that catches regressions before release.
• Automated checks provide timely and trustworthy feedback, with low flakiness and failures that engineers can diagnose.
• AI evaluations and conventional software tests provide complementary evidence, with documented datasets, thresholds, and limitations.
• Release decisions reflect visible quality evidence and remaining risks, and lessons from incidents improve future coverage.

Helpful experience

• Testing software in regulated industries, security-sensitive settings, or critical operational workflows.
• Accessibility testing, performance and load testing, fault injection, or event-driven and long-running workflows.
• Adversarial testing of AI-assisted workflows, including prompt injection, sensitive-data exposure, and unsafe or unauthorized tool actions.
• Building quality engineering practices and shared automation libraries for a growing product team.

Location & working arrangement

This is a full-time position based in our Burlingame, California office and requires working on-site five days a week.

AI & ML knowledge

Strong applied understanding of AI and ML behavior, including model variability, retrieval, evaluation datasets, leakage, failure analysis, and the limits of model-based scoring. You should be able to design appropriate regression evidence for AI-assisted workflows and collaborate with engineering and data science on evaluation quality, security boundaries, and responsible handling of test data.

Every team member should be able to discuss the capabilities and limitations of AI relevant to their work, verify AI-generated output, protect sensitive information, and keep their knowledge current as the technology evolves.

Different perspectives. Shared purpose.

We welcome applicants from all backgrounds and value an inclusive, respectful workplace.