Principal Engineer - MCP or ACP Layer (API & AI Orchestration)

ABC Financial Services; Hyderabad; Hybrid; INR 4,500,000 - 6,500,000; Full time

Benefits offered by this job

Time off and holidays

Mediclaim insurance

Life Insurance

Wellbeing & mental health support

Calm app subscription

Crèche facility support

Job summary

ABC Fitness is seeking a senior individual contributor to independently lead delivery of an LLM feature area, including eval strategy and production readiness. You will architect, build, and evolve the MCP/ACP ecosystem, exposing capabilities as standardized tools for AI agents across ABC Fitness.

You will design schemas, orchestration patterns, and security controls, partnering with API modernization, platform, and security teams to ensure scalable, observable, and auditable AI-native systems.

Qualifications

• 10+ years of software engineering experience, with backend/distributed systems focus.
• Experience designing and operating high-scale backend systems and APIs.
• Strong understanding of LLM interactions with tools, schemas, and orchestration.
• Expertise in abstraction design wrapping complex systems with reusable interfaces.
• Proven ability to define standards and influence engineering direction.
• Strong production mindset with observability, resilience, and performance focus.
• Excellent cross-functional collaboration with multiple teams.
• Growth mindset and fast learning in AI/platform landscapes.
• One Team orientation to enable other teams through reusable platform capabilities.

Responsibilities

• Architect and evolve MCP/ACP ecosystem exposing capabilities as tools.
• Design schemas, tool definitions, and abstraction layers for robust interfaces.
• Implement reusable orchestration patterns for multi-step AI workflows.
• Ensure deterministic, observable, auditable tool interactions.
• Define company-wide AI interaction standards and governance.
• Design security, permissions, and access control for non-human actors.
• Build instrumentation and observability to improve orchestration success.
• Collaborate with API, platform, infrastructure, and security teams for scalable production.
• Lead technical design reviews and influence architecture decisions.
• Document reusable patterns, SDKs, and paved paths for safe AI adoption.

Skills

Distributed backend systems

APIs

Orchestration layers

Platform infrastructure

LLMs & tools

Schema design

Workflow orchestration

Observability

Cross-functional collaboration

Growth mindset

AI governance patterns

Tools

LangChain

LangGraph

APIs gateways

Service mesh

SDKs

Internal developer platforms

Job description

Join ABC Fitness and become part of a culture that’s as ambitious as it is authentic. Let’s transform the future of fitness—together!

Our Values

Best Life

We believe great work begins with great people. That’s why our culture is built on respect, trust, and belonging. We create an inclusive environment where every team member can bring their authentic self to work—because diverse perspectives drive innovation and meaningful impact.

Growth Mindset

We are doers, thinkers, and dreamers. At ABC Fitness, your growth is our investment. Through continuous learning, mentorship, and professional development opportunities, we empower you to reach new heights—personally and professionally.

One Team

From day one, you’ll be part of a team that collaborates, celebrates, and cares. We move fast, support one another, and have fun along the way. Because when you thrive, we all thrive.

We’re looking for a senior individual contributor who can independently lead delivery of an LLM feature area, including eval strategy and production readiness.

WHAT YOU’LL DO

• Architect, build, and evolve the MCP / ACP ecosystem that exposes ABC Fitness capabilities as standardized, agent-consumable tools.

• Design schemas, tool definitions, and abstraction layers that shield AI agents from the complexity and inconsistency of underlying systems while providing clean, structured, hallucination-resistant interfaces.

• Define and implement reusable orchestration patterns for multi-step AI workflows, including tool use, validation, retries, state management, fallback handling, and long-running execution flows.

• Build for deterministic execution by ensuring AI-driven tool interactions are predictable, safe, observable, auditable, and reversible where necessary.

• Establish company-wide standards for AI interaction patterns, structured outputs, input/output validation, schema governance, and agent orchestration workflows.

• Design and implement security, permissions, and access control patterns specifically for non-human actors and autonomous systems, including auditability, rate controls, and operational boundaries.

• Build instrumentation and observability capabilities that trace agent interactions, identify schema friction, measure execution quality, and improve orchestration success rates over time.

• Partner closely with API modernization, platform engineering, infrastructure, security, and SRE teams to define AI-ready integration patterns and ensure scalable production operations.

• Lead technical design reviews and influence architecture decisions across teams building AI-native systems on the platform.

• Define and document reusable patterns, reference implementations, SDKs, and paved paths that accelerate safe AI adoption across ABC Fitness.

WHAT YOU’LL NEED:

• Typically 10+ years of software engineering experience, including deep expertise building distributed backend systems, APIs, orchestration layers, or platform infrastructure; experience in the broader 8–14 year principal range is most aligned for this level.

• Proven experience designing and operating high-scale backend systems with strong understanding of state management, concurrency, reliability, and distributed execution patterns.

• Strong understanding of how LLMs interact with tools and workflows, including schema design, structured outputs, predictable execution patterns, and orchestration reliability.

• Deep expertise in abstraction design and wrapping legacy or complex systems with clean, scalable, and reusable interfaces.

• Experience building workflow orchestration systems handling retries, failure states, long-running processes, and multi-step execution logic.

• Demonstrated ability to define technical standards and influence engineering direction across multiple teams through architecture, design review, and strong engineering judgment.

• Strong production mindset, including experience designing for observability, operational resilience, auditability, scalability, and performance optimization.

• Excellent cross-functional collaboration skills and the ability to work effectively with Engineering, Product, Platform, Infrastructure, Security, and SRE teams.

• A growth mindset and strong learning agility in a rapidly evolving AI and platform engineering landscape.

• A One Team orientation: able to enable other engineering teams, establish scalable paved paths, and create reusable platform capabilities rather than isolated solutions.

AND IT’S GREAT TO HAVE

• Experience with Model Context Protocol (MCP), ACP patterns, LangChain, LangGraph, or similar AI orchestration frameworks.

• Experience with middleware platforms, API gateways, service mesh architectures, or distributed control planes.

• Experience building internal developer platforms, SDKs, or shared engineering tooling.

• Experience implementing AI observability, tracing, prompt tracking, model evaluation, or orchestration telemetry systems.

• Familiarity with policy enforcement, safe model execution, permissioning systems, and AI governance patterns in production environments.

• Experience helping engineering organizations adopt new technical standards through documentation, mentoring, and reference implementations.

What Success Looks Like

• The MCP / ACP layer becomes the standardized paved path for how AI agents interact with ABC business systems.

• Product and platform teams can integrate new AI agents and workflows with core systems quickly, safely, and consistently.

• AI workflows are reliable, observable, auditable, and secure in production environments.

• Agent actions are fully permissioned, rate-limited, and operationally controlled to prevent unsafe or unintended behavior.

• Platform standards reduce integration complexity and accelerate adoption of AI-native development patterns across engineering teams.

• The connectivity layer directly enables new AI-powered product capabilities, intelligent services, and operational automation initiatives.

WHAT’S IN IT FOR YOU:

• Purpose led company with a Values focused culture – Best Life, One Team, Growth Mindset

• Time Off – competitive PTO plans with 15 Earned accrued leave, 12 days Sick leave, and 12 days Casual leave per year

• 11 Holidays plus 4 Days of Disconnect – once a quarter, we take a collective breather and enjoy a day off together around the globe. #oneteam

• Group Mediclaim insurance coverage of INR 500,000 for employee + spouse, 2 kids, and parents or parent-in-laws, and including EAP counseling

• Life Insurance and Personal Accident Insurance

• Best Life Perk – we are committed to meeting you wherever you are in your fitness journey with a quarterly reimbursement

• Premium Calm App – enjoy tranquility with a Calm App subscription for you and up to 4 dependents over the age of 16

• Support for working women with financial aid towards crèche facility, ensuring a safe and nurturing environment for their little ones while they focus on their careers

At ABC Fitness, we don’t just build technology—we build communities and transform lives. Your work matters here, your growth is supported, and your voice is heard.

ABC’S COMMITMENT TO DIVERSITY, EQUALITY, BELONGING AND INCLUSION

ABC is an equal opportunity employer committed to diversity, equity, and inclusion. We strive to create a workplace where every employee, client, and partner feels valued, inspired, and empowered to reach their full potential. For us, inclusion isn’t just the right thing to do—it’s a business imperative. Learn more at abcfitness.com.

About ABC Fitness

ABC Fitness is the #1 software provider for fitness businesses of any size, all around the world. Supporting 30K clubs, 650K coaches, 40M members, and processing more than $12B in payments every year, ABC Fitness provides scalable, data-driven solutions to simplify club management and engage members and clients.

From personal trainers, boutique studios, and gyms, to international franchise health clubs, ABC Fitness will move your business forward through technology and industry insights. Innovation is at the center of everything we do, across ABC Glofox, ABC Ignite, ABC Trainerize, and ABC Evo. Let’s make your fitness vision a seamless reality, together. Learn more at abcfitness.com

#LI-HYBRID