Software Development Senior Analyst

NTT DATA, Inc.; Indianapolis (IN); On-site; USD 85,000 - 125,000; Full time

Job summary

NTT DATA, Inc. is seeking an AI Agent Solution Specialist to design and optimize no-code AI agent journeys in a contact center.

Youll monitor performance, analyze interaction data, and drive continuous improvement while ensuring governance and responsible AI practices across the lifecycle. You will collaborate with product owners, engineers, and CX stakeholders to translate requirements into scalable experiences, build advanced analytics capabilities, and support data-driven decision making.

Qualifications

• English language proficiency required (written and spoken).
• Strong ownership mindset with ability to work autonomously and drive outcomes.
• Excellent analytical and problem-solving skills to translate data into action.
• Clear verbal and written communication for technical and non-technical stakeholders.
• Ability to manage multiple priorities in a deadline-driven environment.

Responsibilities

• Design, configure, and evolve no-code AI agent journeys within a contact center.
• Monitor AI agent performance and optimize journeys using data insights.
• Create and maintain QA scorecards, rubrics, and governance across agent deployments.
• Collaborate with product, engineering, and CX teams to drive quality improvements.
• Lead calibration sessions and provide actionable feedback to AI engineers.

Skills

Analytical thinking

Problem solving

Communication skills

Time management

Autonomy

Cross-functional collaboration

Quality mindset

Job description

Sensation Label: General

AI Agent Solution Specialist

Job Description

This role supports both AI-enabled and human-assisted customer interactions by configuring no-code AI workflows, monitoring performance, and analyzing customer interaction data to drive continuous improvement, operational efficiency, and positive customer outcomes within a contact center environment.

What You’ll Do

AI Agent Journey Design & Configuration

• Design, build, and continuously evolve no-code AI agent journeys, including conversation flows, decision logic, and end-to-end user experiences.
• Configure intents, prompts, business rules, and escalation paths using intuitive tools, enabling scalable content management without direct coding.
• Collaborate with product owners, engineers, and business stakeholders to translate requirements into scalable, no-code agent experiences.
• Ensure all agent journeys align with governance, security, and responsible AI practices across the agent development lifecycle.

Performance Monitoring & Analytics

• Monitor and benchmark AI agent performance across journeys (accuracy, containment, resolution rate, user satisfaction), applying simulation-driven thinking to real-world scenarios.
• Analyze interaction logs and journey analytics to identify drop-offs, failure patterns, and optimization opportunities, feeding insights into continuous improvement loops.
• Design and maintain advanced search and query frameworks (speech and text, pattern-based logic) to enable automated analysis and topic identification in customer interactions.
• Build and maintain advanced speech and text queries (including pattern-based logic) to monitor both human and AI-assisted interactions.
• Generate actionable insights from interaction data and communicate findings to CX stakeholders, supporting data-driven decision making.

AI Agent Quality Management

• Own the end-to-end quality management framework for AI agents — defining quality standards, evaluation criteria, scoring rubrics, and pass/fail thresholds across all deployed agent journeys.
• Conduct systematic conversation reviews and audits of AI agent interactions, scoring responses for accuracy, tone, compliance, escalation appropriateness, and resolution quality.

Sensation Label: General

• Develop and maintain a structured QA scorecard tailored to AI agent interactions, incorporating dimensions such as intent recognition accuracy, hallucination detection, knowledge retrieval relevance, and conversation coherence.
• Identify recurring quality defects, failure modes, and edge cases through interaction sampling and trend analysis — distinguishing between prompt-level, knowledge-level, and integration-level root causes.
• Establish and run calibration sessions with cross-functional stakeholders (product, engineering, CX) to ensure consistent quality evaluation standards across agent deployments.
• Track quality metrics over time (QA pass rate, critical defect rate, regression frequency) and report trends to leadership with clear improvement recommendations.

Training Feedback & AI Engineer Collaboration

• Translate quality findings into structured, actionable feedback for AI Agent Engineers — providing specific examples, annotated conversation logs, and clear descriptions of expected vs. actual agent behavior.
• Maintain a prioritized defect and improvement backlog informed by QA findings, categorized by severity, frequency, and customer impact — collaborating with engineers to drive resolution.
• Participate in regular feedback loops with engineering, reviewing prompt refinements, knowledge base updates, and guardrail adjustments to validate that quality issues are resolved without introducing regressions.
• Develop and curate a library of gold‑standard conversation examples and failure‑case annotations that serve as training references for prompt tuning, knowledge curation, and agent behavior calibration.
• Contribute to the design of automated evaluation pipelines by defining test scenarios, expected outputs, and quality assertions that engineers can integrate into CI/CD workflows.
• Support the creation of regression test suites by documenting resolved defects as repeatable test cases, ensuring fixed issues do not resurface across agent updates.
• Partner with engineers during post‑deployment reviews to assess whether agent updates have improved quality metrics, using before‑and‑after analysis of QA scores and interaction outcomes.

Testing, Experimentation & Continuous Improvement

• Test and validate AI agent behavior through structured experimentation (A/B testing, edge case validation), ensuring quality, compliance, and responsible AI standards.
• Investigate incidents and unexpected agent behavior, conducting root‑cause analysis in non‑deterministic AI systems.
• Contribute to the evolution of self‑improving, generative agent systems by leveraging real‑world interactions and feedback loops.

What You’ll Bring

• Passion for working at the frontier of AI products, especially in generative AI and agent-based systems.
• Language proficiency in English, French, Spanish (written and spoken).
• High ownership mindset with the ability to operate autonomously, navigate ambiguity, and drive meaningful outcomes.
• Strong analytical and problem‑solving skills, with the ability to interpret complex interaction data and translate insights into action.
• Excellent verbal and written communication skills, with the ability to clearly convey findings to both technical and non‑technical stakeholders.
• Ability to manage multiple priorities independently in a deadline‑driven environment, while collaborating effectively across cross‑functional teams.
• Strong planning, organizational, and time‑management skills.
• A quality‑first mindset — methodical attention to detail in reviewing AI agent outputs, with the discipline to maintain consistent evaluation standards across high volumes of interactions.
• Comfort operating in the feedback loop between quality evaluation and engineering execution — able to articulate what’s wrong, why it matters, and what good looks like.

Nice to Have

• Experience with quality assurance and training in a contact center or BPO environment.
• Experience supporting contact center technologies, particularly speech analytics and AI‑driven interaction platforms.
• Experience building AI‑powered products, particularly with LLMs, conversational AI, or autonomous agents.
• Hands‑on experience with AI agent evaluation frameworks, including conversation scoring, automated testing, and regression analysis.
• Familiarity with prompt engineering and knowledge base curation as levers for improving agent quality.
• Experience creating QA rubrics or scorecards for conversational AI or chatbot deployments.