AI Automation Quality Engineer

Harvard University; Cambridge (MA); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Generous paid time off

Medical, dental, and vision insurance

Retirement plan with university match

Wellbeing resources

Tuition assistance/reimbursement

Commuter benefits

Job summary

Harvard University’s IT Academy is hiring an AI Automation Quality Engineer to lead next‑generation quality engineering for the my.harvard portal. You will design intelligent, agent‑based testing that reads Jira stories and architectural docs to generate comprehensive test scenarios.

You will build agents that analyze Django/FastAPI and frontend code, develop RAG pipelines, and implement human‑in‑the‑loop validation before producing automation code.

Qualifications

• Minimum 3+ years building automated test frameworks for enterprise web applications.
• Strong Python development experience.
• Experience testing Django and/or FastAPI applications.
• Experience with Playwright, Selenium, Cypress, pytest, or similar frameworks.
• Experience using Jira and Agile development processes.
• Understanding of Git workflows and CI/CD pipelines.
• Experience working with LLMs (OpenAI, Anthropic, Gemini, etc.), prompt engineering, and AI-assisted software development.
• Understanding of Retrieval-Augmented Generation (RAG), embeddings, vector databases, MCP, or agent frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar.
• Strong analytical thinking and ability to translate business requirements into automated quality strategies.
• Knowledge of SDLC and STLC.

Responsibilities

• Lead AI automation QA for the my.harvard portal, enabling intelligent, agent-based testing.
• Design AI agents to read Jira stories and documentation to generate test scenarios.
• Build code-aware tests for Django, FastAPI, and frontend codebases.
• Develop RAG pipelines to improve test quality and reduce hallucinations.
• Implement human-in-the-loop validation with BAs and POs before automation code is produced.
• Generate maintainable automated tests using Playwright, Selenium, pytest, etc., after approval.
• Maintain regression suites, API tests, UI automation, and CI/CD integration.
• Collaborate with developers, architects, BAs, and DevOps to set QA standards.

Skills

Python development

Automated testing

QA design

AI/ML concepts

Education

Bachelor's degree in a related field

Tools

Playwright

Selenium

Cypress

pytest

Job description

By working at Harvard University, you join a vibrant community that advances Harvard's world-changing mission in meaningful ways, inspires innovation and collaboration, and builds skills and expertise. We are dedicated to creating a diverse and welcoming environment where everyone can thrive.

Why join Harvard University Central Administration?

Harvard University's Central Administration (CADM) is a 5,000+ employee organization that supports the university's overall excellence by understanding and serving the needs of its schools, students, faculty, staff, alumni, and surrounding communities. Through dynamic and collaborative partnerships, CADM provides high-quality and efficient services to the schools to help them achieve their goals.

Harvard University Information Technology (HUIT) is a community of Information Technology professionals committed to understanding our users and devoted to making it easier for faculty, students, and staff to teach, research, learn, and work through the effective use of information technology. We are recruiting an IT workforce that has both breadth in their ability to collaborate and innovate across disciplines – and depth in specific areas of expertise. HUIT offers opportunities for IT professionals to learn and work in a unique technology landscape and service-focused environment. If you are a technically proficient, nimble, user-focusedand accountable IT professional who also connects with the importance of collaborating well in a teamenvironmentwe are looking for you!

More About HUIT:

Our Mission:

We empower the Harvard community with essential and transformative technologies to advance education, knowledge, and discovery.

HUIT’s core values are:

• Human-centered
• University-focused
• Innovation-driven
• Team-oriented

IT Academy (designed for IT Staff):

HUIT’s IT Academy aims to enable each IT staff person to grow professionally and become a trusted partner to her or his team. The IT Academy is built on the belief that every IT staff member across the University (including technology employees at each school and campus) can grow in her or his area of expertise as well as building strong people and project management skills. Learn more here: https://itacademy.harvard.edu/.

Job-Specific Responsibilities:

The AI Automation Quality Engineer leads the next generation of quality engineering for the my.harvard portal, moving beyond traditional QA to design intelligent, agent-based testing capabilities that automatically analyze requirements, understand application code, generate reusable test assets, and continuously improve HUIT’s testing strategy.

• This role combines strong software testing experience with modern AI engineering concepts, including Retrieval-Augmented Generation (RAG), large language models (LLMs), agentic workflows, and test automation frameworks. Working closely with Business Analysts, developers, and product owners, the AI Automation Quality Engineer helps build an autonomous QA platform that accelerates delivery while improving software quality.
• AI-Powered Test Generation: Design and implement AI agents that read Jira user stories, acceptance criteria, architectural documentation, and business rules to automatically generate comprehensive functional, integration, regression, accessibility, and API test scenarios
• Code-Aware Testing: Build agents capable of analyzing Django, FastAPI, and frontend codebases to understand application behavior, identify impacted components, and generate reusable test cases with high coverage
• RAG & Knowledge Engineering: Develop Retrieval-Augmented Generation (RAG) pipelines that leverage Jira, Confluence, Git repositories, existing test suites, architecture documentation, API specifications, and historical defects to improve test quality and reduce hallucinations
• Human-in-the-Loop Validation: Implement workflows where generated test scenarios are reviewed and validated by Business Analysts and Product Owners before automation code is produced
• Autonomous Test Automation: After approval, generate maintainable automated tests using Playwright, Selenium, pytest, or equivalent frameworks, validate results, and prepare commits or pull requests for inclusion in the shared test repository
• Continuous Learning: Capture review feedback, production defects, and failed test patterns to improve prompts, retrieval strategies, reusable testing patterns, and agent performance over time
• Quality Engineering: Maintain regression suites, API tests, UI automation, performance validation, and CI/CD integration while ensuring generated tests remain reliable and maintainable
• Collaboration: Partner with developers, architects, BAs, and DevOps engineers to define AI-driven quality engineering standards and governance

Basic Qualifications:

Minimum of two years’ post-secondary education or relevant work experience

Required:

• 3+ years building automated test frameworks for enterprise web applications
• Strong Python development experience
• Experience testing Django and/or FastAPI applications
• Experience with Playwright, Selenium, Cypress, pytest, or similar frameworks
• Experience using Jira and Agile development processes
• Understanding of Git workflows and CI/CD pipelines
• Experience working with LLMs (OpenAI, Anthropic, Gemini, etc.), prompt engineering, and AI-assisted software development
• Understanding of Retrieval-Augmented Generation (RAG), embeddings, vector databases, MCP, or agent frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar
• Strong analytical thinking and ability to translate business requirements into automated quality strategies
• Knowledge of advanced IT project management principles (e.g. Agile) and software
• Knowledge of SDLC and STLC

The following qualifications are strongly preferred. If you meet some, but not all, you are still encouraged to apply; we value employees with a willingness to learn.

• Experience developing AI agents that generate code or software tests
• Experience integrating AI into SDLC workflows
• Knowledge of GitHub Copilot, Cursor, Claude Code, Windsurf, or similar AI development environments
• Experience with vector databases (Pinecone, pgvector, Chroma, Weaviate)
• Experience with OpenTelemetry, observability, and evaluation frameworks for AI systems
• Experience in higher education or enterprise ERP/student information systems
• Knowledge of Oracle, REST APIs, GraphQL, Docker, Kubernetes, and cloud-native development
• Experience measuring test coverage, mutation testing, and AI quality metrics
• AI/ML engineering and agentic system design
• Test automation and quality engineering best practices
• Cross-functional collaboration with Business Analysts, developers, and product owners
• Strong analytical thinking and problem-solving
• Demonstrated team performance skills, service mindset approach, and the ability to act as a trusted advisor
• Continuous improvement and adaptability to emerging AI and testing technologies

Certificates and Licenses:

• Completion of Harvard IT Academy specified foundational courses (or external equivalent) preferred

Working Conditions:

Onsite work is performed in an office setting.

Appointment End Date:

This is a fully benefited, full-time Harvard University position that has been funded through 6/30/27. There is the possibility of renewal, contingent upon your work performance meeting the needs of the department and the continued availability of funding designed to support the position.

Hybrid - Details to be discussed during the interview process

Visa Sponsorship Information:

Harvard University is unable to provide visa sponsorship for this position

Pre-Employment Screening:

Identity

Other Information:

This position has a 180-day orientation and review period.

Work Format Details

This position has been determined by school or unit leaders that some of the duties and responsibilities can be effectively performed at a non-Harvard location. The work schedule and location will be set by the department at its discretion and based upon operational needs. When not working at a Harvard or Harvard-designated location, employees in hybrid positions must work in a Harvard registered state in compliance with the University’s Policy on Employment Outside of Massachusetts. Additional details will be discussed during the interview process. Certain visa types and funding sources may limit work location. Individuals must meet work location sponsorship requirements prior to employment.

Salary Grade and Ranges

This position is salary grade level 056. Please visit Harvard's Salary Ranges to view the corresponding salary range and related information.

Benefits

Harvard offers a comprehensive benefits package that is designed to support a healthy work-life balance and your physical, mental and financial wellbeing. Because here, you are what matters. Our benefits include, but are not limited to:

• Generous paid time off including parental leave
• Medical, dental, and vision health insurance coverage starting on day one
• Retirement plans with university contributions
• Wellbeing and mental health resources
• Support for families and caregivers
• Professional development opportunities including tuition assistance and reimbursement
• Commuter benefits, discounts and campus perks

Learn more about these and additional benefits on our Benefits & Wellbeing Page.

EEO/Non-Discrimination Commitment Statement

Harvard University is committed to equal opportunity and non-discrimination. We seek talent from all parts of society and the world, and we strive to ensure everyone at Harvard thrives. Our differences help our community advance Harvard's academic purposes.

Harvard has an equal employment opportunity policy that outlines our commitment to prohibiting discrimination on the basis of race, ethnicity, color, national origin, sex, sexual orientation, gender identity, veteran status, religion, disability, or any other characteristic protected by law or identified in the university’s non-discrimination policy. Harvard's equal employment opportunity policy and non-discrimination policy help all community members participate fully in work and campus life free from harassment and discrimination.