Senior Developer, Artificial Engineering

EisnerAmper LLP; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

EisnerAmper LLP is seeking a QA Engineer for AI Initiatives to ensure quality, reliability, fairness, and performance of AI/ML-powered products and systems. You will design AI-specific test strategies, validate data pipelines, and collaborate with data scientists to set acceptance criteria and thresholds.

The role focuses on non-deterministic model behavior, hallucinations, drift monitoring, and ethics/compliance in AI systems, with Bangalore as a preferred location.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 3–6 years of QA experience, with at least 1–2 years in AI/ML quality assurance
• Strong proficiency in Python for test automation and data analysis
• Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith)
• Hands-on experience with testing tools: Pytest, Selenium, Postman, or similar
• Understanding of ML lifecycle — training, validation, deployment, and monitoring
• Knowledge of data quality tools and pipeline testing (Great Expectations, dbt tests)

Responsibilities

• Design and execute test strategies specifically for AI/ML models, LLM-based applications, and data pipelines
• Develop automated test frameworks for model validation, regression testing, and performance benchmarking
• Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs
• Test RAG pipelines, chatbots, recommendation systems, and other AI-driven features
• Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds
• Build and maintain evaluation datasets, ground truth sets, and adversarial test cases
• Monitor models in production for drift, degradation, and anomalous behavior
• Validate data quality, data pipelines, and feature stores that feed AI systems
• Document defects, edge cases, and failure patterns specific to AI behavior
• Ensure AI systems meet ethical, fairness, and compliance standards (bias audits, explainability checks)

Skills

Python

QA for AI

LLM evaluation

Pytest

Selenium

Postman

Great Expectations

dbt tests

Education

Bachelor's or Master's in CS/Engineering

Tools

Selenium

Postman

RAGAS

DeepEval

Promptfoo

LangSmith

Job description

Job Description A QA Engineer for AI Initiatives is responsible for ensuring the quality, reliability, fairness, and performance of AI/ML-powered products and systems. Unlike traditional QA, this role requires deep understanding of non-deterministic model behavior, data quality, and AI-specific failure modes such as hallucinations, bias, and model drift.

Key Responsibilities

• Design and execute test strategies specifically for AI/ML models, LLM-based applications, and data pipelines
• Develop automated test frameworks for model validation, regression testing, and performance benchmarking
• Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs
• Test RAG (Retrieval-Augmented Generation) pipelines, chatbots, recommendation systems, and other AI-driven features
• Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds
• Build and maintain evaluation datasets, ground truth sets, and adversarial test cases
• Monitor models in production for drift, degradation, and anomalous behavior
• Validate data quality, data pipelines, and feature stores that feed AI systems
• Document defects, edge cases, and failure patterns specific to AI behavior
• Ensure AI systems meet ethical, fairness, and compliance standards (bias audits, explainability checks)

Preferred Location: Bangalore

Required Skills & Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• 3–6 years of QA experience, with at least 1–2 years in AI/ML quality assurance
• Strong proficiency in Python for test automation and data analysis
• Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith)
• Hands-on experience with testing tools: Pytest, Selenium, Postman, or similar
• Understanding of ML lifecycle — training, validation, deployment, and monitoring
• Knowledge of data quality tools and pipeline testing (Great Expectations, dbt tests)

Nice to Have

• Experience with prompt engineering and red-teaming LLMs
• Familiarity with MLOps platforms (MLflow, SageMaker, Vertex AI)
• Knowledge of vector databases and embedding quality evaluation
• Understanding of AI safety, responsible AI principles, and fairness frameworks
• Experience with A/B testing and shadow deployment strategies

Soft Skills

• Analytical and inquisitive mindset — comfortable challenging model outputs
• Ability to think like both a user and an adversary (red-team thinking)
• Strong documentation and communication skills
• Collaborative approach with data science, engineering, and product teams
• High attention to detail with a quality-first attitude

EisnerAmper is one of the largest accounting, tax and business advisory firms in the U.S., with more than 4,500 employees and over 450 partners across the country. We combine responsiveness with a long-range perspective; to help clients meet the pressing issues they face today, and position them for success tomorrow. Our clients are enterprises as diverse as sophisticated financial institutions and start-ups, global public firms and middle-market companies, as well as high net worth individuals, family offices, not-for-profit organizations, and entrepreneurial ventures across a variety of industries. We are also engaged by the attorneys, financial professionals, bankers and investors who serve these clients. Our reach extends globally, with offices in the United Kingdom, Israel, India, the Cayman Islands, Singapore, and Ireland. EisnerAmper is a merit-based employer. Qualified applicants will receive consideration for employment regardless of background. If you require assistance completing an employment application due to a disability, please contact [e-mail] so that we can provide you with an accommodation or an alternative application process.