Principal AI Engineer – Agentic AI / LLM

ApTask; New York (NY); Hybrid; USD 180,000 - 300,000; Full time

Job summary

ApTask in New York City is seeking a Principal AI Engineer to design, build, and productionize enterprise-scale Agentic AI, Generative AI, and LLM-powered systems. This is a hands-on leadership role spanning architecture, data pipelines, deployment, and observability.

You will own AI systems end-to-end—from experimentation to production—ensuring secure tool interactions, retrieval, and reasoning over enterprise context.

Qualifications

• 8–14 years of professional software engineering experience; 10+ years preferred.
• Strong understanding of data structures and algorithms.
• Experience with APIs, microservices, system design, and distributed systems.
• Experience building and operating production-grade data pipelines.
• Strong understanding of scalable software architecture.
• Experience with at least one backend language such as Go, Rust, Java, C, or C++ is highly desirable.
• 2+ years of hands‑on LLM/Generative AI engineering experience.
• Experience building and deploying production LLM applications.
• Demonstrated experience designing multiple Agentic AI systems in production.
• Strong understanding of LLM architecture, prompting, context management.

Responsibilities

• Architect and build production-grade Agentic AI systems using LLMs, RAG, tool calling, and agent orchestration.
• Design autonomous AI agents capable of reasoning, retrieval, planning, and secure execution of enterprise actions.
• Backend services
• Agent orchestration
• Evaluation frameworks
• Observability
• Cloud infrastructure
• Design and implement production RAG pipelines (chunking, embeddings, retrieval, hybrid search, reranking, context construction).
• Build and integrate MCP servers and secure tool-calling interfaces.
• Develop agent workflows using LangGraph, Claude Agent SDK, CrewAI, or equivalents.
• Implement AI reliability mechanisms including guardrails, validators, deterministic fallbacks, and rollback strategies.
• Build evaluation frameworks for LLM and agentic systems, including offline and continuous evaluation.
• Implement AI observability and tracing with LangSmith, Langfuse, monitoring cost/throughput/scalability.
• Design and operate scalable backend services, APIs, microservices, and distributed systems.
• Develop high-quality Python and SQL code.
• Deploy AI apps across AWS, Azure, or GCP; work with Kubernetes/Lambda.

Skills

Software engineering

LLM / Generative AI

Agentic AI

RAG & Vector Search

Cloud computing

Python

Distributed systems

LangSmith / Langfuse

Kubernetes

Tool integration

Tools

AWS

Azure

GCP

LangGraph

Claude Agent SDK

CrewAI

Google ADK

Terraform

Kubernetes

Bedrock/Vertex AI

Job description

Job Title: Principal AI Engineer – Agentic AI / LLM

Location: New York City, NY

Work Arrangement: Hybrid – 3 days onsite per week

Employment Type: Full-Time

Work Hours: EST overlap required

Experience: 8–14 years of software engineering experience; 10+ years preferred

Visa Sponsorship: H-1B transfer candidates can be considered

About the Role

We are seeking a highly experienced Principal AI Engineer to design, build, and productionize enterprise-scale Agentic AI, Generative AI, and LLM-powered systems.

This is a highly hands-on engineering role for an experienced technical leader who can operate across AI architecture, software engineering, RAG, LLM applications, agent orchestration, data pipelines, evaluation, observability, and cloud infrastructure.

You will own AI systems end-to-end—from architecture and experimentation through production deployment, optimization, reliability, and continuous improvement.

The ideal candidate has demonstrated experience building production-grade AI agents that can reason over enterprise context, retrieve information, invoke tools and APIs, and securely interact with real-world business systems.

Key Responsibilities

• Architect and build production-grade Agentic AI systems using LLMs, RAG, tool calling, and agent orchestration.
• Design autonomous AI agents capable of reasoning, retrieval, planning, and secure execution of enterprise actions.
• Backend services
• Agent orchestration
• Evaluation frameworks
• Observability
• Cloud infrastructure
• Design and implement production RAG pipelines covering:
  • Chunking
  • Embeddings
  • Retrieval
  • Hybrid search
  • Reranking
  • Context construction
  • Response validation
• Build and integrate MCP (Model Context Protocol) servers and secure tool-calling interfaces.
• Develop reliable agent workflows using frameworks such as LangGraph, Google ADK, CrewAI, Claude Agent SDK, or equivalent technologies.
• Implement AI reliability mechanisms including:
  • Guardrails
  • Validators
  • Deterministic fallbacks
  • Rollback strategies
• Build evaluation frameworks for LLM and agentic systems, including offline evaluation, regression testing, and continuous evaluation.
• Implement AI observability and tracing using tools such as LangSmith, Langfuse, or equivalent platforms.
  • Cost
  • Throughput
  • Scalability
• Design and operate scalable backend services, APIs, microservices, and distributed systems.
• Develop and maintain high-quality Python and SQL code.
• Deploy AI applications across AWS, Azure, or GCP.
• Work with containerized and cloud infrastructure such as EKS/ECS, Kubernetes, Lambda, and related services.
• Implement CI/CD and infrastructure-as-code using Terraform, CloudFormation, or equivalent technologies.
• Partner with product, engineering, data science, and business stakeholders to translate ambiguous business requirements into scalable technical solutions.
• Provide technical leadership through architecture/design reviews, engineering standards, code reviews, and mentoring.

Required Qualifications

• Software Engineering
  • 8–14 years of professional software engineering experience; 10+ years preferred.
  • Strong understanding of data structures and algorithms.
  • Experience with APIs, microservices, system design, and distributed systems.
  • Experience building and operating production-grade data pipelines.
  • Strong understanding of scalable software architecture.
  • Experience with at least one additional backend/systems language such as Go, Rust, Java, C, or C++ is highly desirable.
• LLM / Generative AI
  • 2+ years of hands‑on LLM/Generative AI engineering experience.
  • Experience building and deploying production LLM applications.
  • Demonstrated experience designing and taking multiple Agentic AI systems into production.
  • Strong understanding of LLM architecture, prompting, context management, and model behavior.
• Agentic AI
  • Hands‑on experience with one or more:
  • Experience with:
  • Planning/reasoning workflows
  • Multi‑agent workflows
  • Agent reliability and failure handling
• MCP & Tool Calling
  • Strong hands‑on experience with Model Context Protocol (MCP).
  • Strong understanding of function/tool calling and API integrations.
• RAG & Vector Search
  • Strong understanding of RAG from first principles, including:
  • Experience with vector databases such as:
  • Other vector database/cloud equivalents
• Evaluation & Observability
  • Offline evaluation datasets and automated evaluation.
  • LLM observability and tracing.
  • Experience with LangSmith, Langfuse, or equivalent.
  • Experience implementing AI guardrails and reliability mechanisms.
• Strong AWS experience preferred, including:
• Also valued:
• AI Performance Optimization
  • Experience optimizing LLM systems for:
  • Token consumption
  • Context pruning and compaction
  • Throughput

Preferred Qualifications

• Experience with Amazon Bedrock, Azure AI, or Google Vertex AI.
• Experience working with enterprise or regulated environments.
• Experience building AI platforms used at scale.
• Ability to translate ambiguous business problems into technical architecture.
• Demonstrated technical leadership and mentoring experience.

Ideal Candidate Profile

Staff/Principal AI Engineer | Agentic AI | LLM | Python | RAG | MCP | AWS

We are particularly interested in candidates who have:

• Built and deployed multiple production AI agents.
• Built MCP servers and secure tool interfaces.
• Hands‑on experience with LangGraph, Claude Agent SDK, Google ADK, CrewAI, or equivalent.
• Experience with tool orchestration and state management.
• Experience with LangSmith/Langfuse or equivalent observability.
• Implemented AI guardrails and reliability mechanisms.
• Experience optimizing token usage, context, latency, and cost.
• Strong AWS/Kubernetes experience.
• Strong distributed systems and Python engineering experience.
• Enterprise/client-facing technical leadership experience.

Important: We are looking for candidates with genuine production ownership, rather than experience limited to prompt engineering, basic chatbot development, research, or LLM API integrations.