Lead Software Development Engineer in Test

Alteryx, Inc.; Irvine (CA); On-site; USD 165,000 - 185,000; Full time

Benefits offered by this job

Connectivity Plus stipend (monthly)

Home office reimbursement (annual)

Medical, dental, and vision coverage

401(k) with company match

Paid parental leave

Caregiver leave

Flexible time off

Mental health support

Wellness reimbursement

Career development and education

Job summary

Alteryx, Inc. is seeking a Lead SDET to shape testing strategy for AI-driven product workflows. You will design automated tests, build scalable frameworks, and collaborate with engineers, product managers, and UX to ensure quality. The role emphasizes evaluation tooling, CI/CD integration, and proactive design feedback.

The position focuses on advanced testing for AI-enabled features, with a strong emphasis on observability, performance, and robust test reporting in a fast-moving environment.

Qualifications

• Experience evaluating AI agents, including multi-step task completion, tool selection, tool-call accuracy, and failure recovery.
• Working knowledge of agent frameworks, preferably LangChain, with ability to understand and debug context handling, tool execution, and agent state.
• Hands-on experience with LangSmith or similar platforms for evaluation datasets, custom evaluators, experiments, and comparisons across agent changes.
• Experience with agent observability, using traces and telemetry to investigate quality issues, latency, token usage, and cost.
• Strong Python software engineering skills to build and extend evaluation tooling, metrics pipelines, and reporting systems.
• JavaScript or TypeScript experience is a nice-to-have.
• Applied statistics and data analysis skills to assess variability, compare experiments, and distinguish meaningful improvements from noise; modeling experience is a plus.
• Experience developing evaluation datasets and scoring methods, combining deterministic checks, human feedback, and LLM-based judges.
• Ability to contribute at a technical lead level, guiding evaluation practices, mentoring engineers, integrating evaluations into CI/CD, and translating production failures into regression coverage.

Responsibilities

• Analyze requirements to produce a comprehensive test strategy and detailed test cases.
• Design and implement manual and automated tests at all levels in the test pyramid, including unit, component, integration, and end-to-end levels.
• Design, build, and enhance scalable automated test frameworks to support feature testing and AI-driven product workflows.
• Develop and maintain automation for UI and API tests using tools such as Selenium, pytest, and Playwright.
• Conduct performance and load testing (using tools like locust) to ensure product scalability.
• Build test reports using reporting tools such as Allure.
• Report regressions to a corresponding dev team.
• Influence design decisions by providing timely feedback on design documents and testability.
• Collaborate closely with other members of an agile team, including engineers, product management, and UX, to become a valued member of an autonomous, cross-functional team.

Skills

AI agent evaluation

LangChain

LangSmith

Agent observability

Python

JavaScript/TypeScript

Statistics & data analysis

CI/CD integration

Testing frameworks

Tools

Selenium

pytest

Playwright

Locust

Job description

Meet the Moment with Alteryx

We're living through a once-in-a-generation shift in how work gets done. Data, automation, and AI are quickly becoming the center of every business decision - and Alteryx is leading the transformation.

You'll be working on the challenges that sit at the heart of modern business. No matter your role, the work you do will help organizations move faster, see more clearly, and tackle questions that used to feel impossible. If you're ready to meet the moment with innovation, curiosity, and excellence, there's a place for you here.

Lead SDET Responsibilities

• Analyze requirements to produce a comprehensive test strategy and detailed test cases.
• Design and implement manual and automated tests at all levels in the test pyramid, including unit, component, integration, and end-to-end levels.
• Design, build, and enhance scalable automated test frameworks to support feature testing and AI-driven product workflows.
• Develop and maintain automation for UI and API tests using tools such as Selenium, pytest, and Playwright.
• Conduct performance and load testing (using tools like locust) to ensure product scalability.
• Build test reports using reporting tools such as Allure.
• Report regressions to a corresponding dev team.
• Influence design decisions by providing timely feedback on design documents and testability.
• Collaborate closely with other members of an agile team, including engineers, product management, and UX, to become a valued member of an autonomous, cross-functional team.

Requirements

• Experience evaluating AI agents, including multi-step task completion, tool selection, tool-call accuracy, and failure recovery.
• Working knowledge of agent frameworks, preferably LangChain, with the ability to understand and debug context handling, tool execution, and agent state.
• Hands-on experience with LangSmith or similar platforms for evaluation datasets, custom evaluators, experiments, and comparisons across agent changes.
• Experience with agent observability, using traces and telemetry to investigate quality issues, latency, token usage, and cost.
• Strong Python software engineering skills to build and extend evaluation tooling, metrics pipelines, and reporting systems.
• JavaScript or TypeScript experience is a nice-to-have.
• Applied statistics and data analysis skills to assess variability, compare experiments, and distinguish meaningful improvements from noise; modeling experience is a plus.
• Experience developing evaluation datasets and scoring methods, combining deterministic checks, human feedback, and LLM-based judges.
• Ability to contribute at a technical lead level, guiding evaluation practices, mentoring engineers, integrating evaluations into CI/CD, and translating production failures into regression coverage.

Compensation

Alteryx is committed to fair, equitable, and transparent compensation. Final compensation is determined by several factors, including but not limited to relevant work experience, education, certifications, skills, and geographic location.

The base salary range for this role in the United States is $165,000-$185,000 .

This role is also eligible for a target annual bonus of 15% of base salary, based on individual and company performance.

• A monthly Connectivity Plus stipend of $150 to support remote work-related expenses
• An annual $200 home office reimbursement
• Medical, dental, and vision coverage
• 401(k) with company match
• Paid parental leave, caregiver leave, and flexible time off
• Mental health support and wellness reimbursement
• Career development and education assistance

Benefits & Perks

Alteryx offers a comprehensive benefits package designed to support your health, financial security, and overall well-being, including:

• Medical, dental, and vision coverage
• 401(k) with company match
• Paid parental leave, caregiver leave, and flexible time off
• Mental health support and wellness reimbursement
• Career development and education assistance

As part of our ongoing commitment to a diverse, equitable, and inclusive workplace, we’re invested in building teams with a wide variety of backgrounds, identities, and experiences.

Legal and Compliance

For roles in San Francisco and Los Angeles: Pursuant to the San Francisco Fair Chance Ordinance and the Los Angeles Fair Chance Initiative for Hiring, Alteryx will consider for employment qualified applicants with arrest and conviction records.

This position involves access to software/technology that is subject to U.S. export controls. Any job offer made will be contingent upon the applicant’s capacity to serve in compliance with U.S. export controls.

Who We Are

Alteryx is a leading AI-ready data and analytics company that powers actionable insights to help organizations drive smarter, faster decisions with AI-ready data. More than 8,000 customers around the world rely on Alteryx to automate analytics, improve revenue performance, manage costs, and mitigate risk across their businesses.

Disclaimers

If you require reasonable accommodation as part of the recruiting process, please contact [e-mail].

Alteryx, Inc. is an Equal Employment Opportunity Employer.

Unsolicited third party/agency profiles and resumes will not be considered.