Software Development Engineer in Test - AI

Jobgether; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Fully remote

International teams

Learning opportunities

Mentorship

Job summary

Jobgether is seeking a Quality Engineer focused on AI/LLM testing to design and optimize validation pipelines for chatbots, RAG systems, and autonomous agents.

You will build automation in Python using Robot Framework, Playwright, and CI/CD integration, collaborating with AI/ML, product, and DevOps teams to raise testing standards.

This fully remote US role offers full-time engagement, competitive benefits, and exposure to cutting-edge AI technologies across cloud platforms.

Qualifications

• Bachelor's degree in Computer Science or related field; equivalent practical experience valued.
• Strong Python skills and ability to build automation utilities.
• Solid understanding of software testing principles and modern automation practices.
• Hands-on experience with AI/LLM testing, prompt engineering, and evaluation frameworks.
• Experience designing automated output evaluation, hallucination detection, and safety validation.
• Experience API testing, microservices, UI automation, and SQL data validation.
• Familiarity with AI agent frameworks (LangChain, LangGraph, CrewAI, Semantic Kernel).
• Experience integrating automated testing into CI/CD pipelines and cloud platforms (AWS, Azure, GCP).
• Strong debugging, analytical, and problem-solving abilities in Agile contexts.

Responsibilities

• Design, implement, and optimize testing strategies for AI-powered applications and RAG pipelines.
• Develop AI evaluation frameworks to assess hallucinations, bias, safety, and response quality.
• Build automated validation pipelines for LLM outputs using structured assertions and metrics.
• Test agentic systems including multi-agent orchestration, memory, and tool calls.
• Contribute to prompt and context engineering to improve AI reliability.
• Evaluate performance, latency, and cost of AI workloads across cloud infra.
• Create scalable UI/API automation frameworks using Python and Robot Framework.
• Automate testing across REST/GraphQL, events, streaming, and microservices.
• Validate RAG architectures and vector databases for retrieval accuracy.

Skills

Python

Automated testing

AI/LLM testing

CI/CD

Debugging

Education

Bachelor's degree in CS or related

Tools

Robot Framework

Playwright

Selenium

LangChain

Job description

Accountabilities:

- AI/LLM Quality Engineering: Design, implement, and optimize testing strategies for AI-powered applications, chatbots, copilots, Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents.

- AI Evaluation Frameworks: Develop and maintain AI evaluation (Evals) frameworks to assess hallucinations, groundedness, toxicity, bias, response quality, contextual relevance, prompt regressions, output consistency, and safety guardrails.

- Automated AI Validation: Build automated validation pipelines for LLM-generated outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics. Create synthetic and benchmark datasets to validate different AI scenarios.

- Agentic Systems Testing: Validate multi-agent orchestration workflows, agent planning and execution, memory retention, context management, tool calling, function calling, and structured outputs. Test MCP (Model Context Protocol) integrations and interoperability.

- Prompt and Context Engineering: Contribute to prompt engineering, context engineering, and prompt optimization strategies to improve AI response quality, reliability, and consistency.

- Performance and Cost Optimization: Evaluate AI application latency, token consumption, throughput, scalability, and model usage costs to ensure efficient system performance.

- Automation Engineering: Develop and maintain scalable UI and API automation frameworks using Python, with tools such as Robot Framework, Playwright, Selenium, and Cypress. Build reusable testing libraries, utilities, and AI-assisted testing workflows.

- API and Application Testing: Automate testing across REST APIs, GraphQL APIs, event-driven systems, streaming services, webhooks, microservices, and micro-frontend architectures.

- RAG and AI Platform Validation: Validate RAG architectures, including document chunking strategies, embedding quality, retrieval accuracy, reranking, and vector search relevance. Work with vector databases such as Pinecone, Weaviate, ChromaDB, Qdrant, and FAISS.

- AI Agent Frameworks: Build, test, and validate AI agents using frameworks such as LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, and OpenAI SDKs.

- DevOps and CI/CD Integration: Integrate automated testing and quality gates into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI/CD, or Azure DevOps.

- Cloud and Infrastructure Testing: Work with Docker, Kubernetes, and cloud environments across AWS, Azure, OCI, and GCP to validate application reliability, scalability, and deployment quality.

- Observability and Reliability: Monitor AI application behavior using logging, tracing, and observability platforms. Validate distributed systems reliability, identify performance bottlenecks, and contribute to shift-left quality engineering practices.

- Cross-Functional Collaboration: Partner with engineering, AI/ML, product, platform, and DevOps teams to embed quality throughout the development lifecycle and continuously improve testing standards.

Requirements:

- Bachelor's or Master's degree in Computer Science, Software Engineering, Artificial Intelligence, Machine Learning, or a related field. Equivalent practical experience and demonstrable technical projects are also highly valued.

- Strong programming skills in Python, with the ability to develop automation frameworks, reusable testing utilities, and complex validation workflows.

- Solid understanding of software testing principles, quality engineering methodologies, and modern test automation practices.

- Hands-on experience with AI/LLM testing, prompt engineering, context engineering, AI evaluation frameworks, RAG architectures, and AI agents.

- Practical experience designing and validating AI-powered workflows, including automated output evaluation, hallucination detection, response quality assessment, and safety validation.

- Experience with API testing and automation, microservices testing, UI automation, and data validation using SQL.

- Familiarity with AI agent frameworks and technologies such as LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, and MCP (Model Context Protocol).

- Experience integrating automated testing into CI/CD pipelines and working with Docker, Kubernetes, and cloud platforms such as AWS, Azure, OCI, or GCP.

- Strong debugging, analytical, troubleshooting, and problem-solving abilities, with the capacity to investigate complex technical issues independently.

- Experience working in Agile or Scrum environments and collaborating effectively with multidisciplinary engineering teams.

- Demonstrated curiosity about emerging AI technologies, with the ability to rapidly learn and experiment with new frameworks, tools, and architectures.

- Strong communication and collaboration skills, combined with a proactive mindset and passion for innovation, automation, and developer productivity.

Good to have:

- Experience with Robot Framework and Playwright for scalable test automation.

- Hands-on familiarity with AI development tools such as Cursor, GitHub Copilot, Claude, Gemini, and OpenAI APIs.

- Experience working with vector databases, embeddings, and semantic search technologies.

- Exposure to MLOps workflows and machine learning lifecycle testing.

- Knowledge of performance testing, security testing, and AI governance or compliance requirements.

- Experience with contract testing, chaos testing, resilience engineering, and distributed systems validation.

- Familiarity with accessibility testing, cross-browser and cross-device testing, synthetic monitoring, and production observability.

Bonus Skills – High Impact:

- AI red teaming and LLM security testing, including prompt injection, jailbreaks, and data leakage detection.

- Agent reliability, safety, and autonomous workflow testing.

- AI model and infrastructure cost optimization.

- Distributed systems testing and test data engineering.

- Feature flag testing and AI observability or telemetry.

- Experience with reinforcement learning feedback workflows and human-in-the-loop evaluation systems.

- Knowledge graph integrations and multimodal AI testing involving vision, audio, and text.

Additional Qualities That Stand Out:

- Personal or professional AI projects, prototypes, proof-of-concepts, or open-source contributions.

- GitHub repositories demonstrating AI automation, intelligent testing frameworks, or agent development.

- Experience building AI agents, copilots, or autonomous testing systems.

- Technical blogs, presentations, workshops, or other contributions related to AI and software testing.

- Experience integrating AI capabilities into software development lifecycle (SDLC) or quality engineering workflows.

- Strong interest in combining traditional quality engineering practices with modern AI-native development approaches.

Benefits:

- Fully remote working arrangement from India.

- Full-time employment with opportunities to collaborate with international teams.

- Competitive healthcare coverage, including an HMO benefit of 175K MBL and one free dependent after one year of service.

- Generous vacation policy to support work-life balance and personal well-being.

- Punctuality bonus and additional employee benefits.

- Access to learning and professional development opportunities to support long-term career growth.

- Opportunities for mentorship, cross-functional collaboration, and exposure to different areas of the business.

- Inclusive and diverse workplace culture that encourages employees to bring their authentic selves to work.

- Access to more than 20 employee affinity groups, providing opportunities to connect, network, and engage with colleagues globally.

- Opportunity to work with emerging AI technologies, intelligent automation solutions, and next-generation quality engineering practices.