Staff AI Architect

Robots and Pencils; United States; Remote; USD 146,000 - 202,000; Full time

Job summary

Robots & Pencils is seeking a senior AI architecture lead to design end-to-end LLM and agentic systems for diverse client engagements. You’ll own architecture decisions across retrieval, RAG, and model serving on AWS, balancing accuracy, latency, and cost.

You’ll mentor engineers, define LLMOps standards, implement guardrails and observability, and collaborate with product and data teams to translate business goals into scalable AI solutions.

Qualifications

• 7+ years in software engineering with AI/ML leadership
• Deep expertise in LLM and agentic architecture
• Production RAG and retrieval systems design
• Model serving and inference optimization on AWS
• LLMOps and MLOps including eval pipelines & CI/CD
• Experience with AI evaluation, guardrails, and tracing tooling
• Proficiency in Python and at least one other language
• Strong cloud-native architecture knowledge (microservices, serverless)
• Understanding of responsible AI governance and data isolation

Responsibilities

• Define AI system architecture and lead design across LLM applications and agentic systems
• Drive RAG and context-engineering strategy (hybrid retrieval, embedding models)
• Lead multi-agent orchestration design using LangGraph, Bedrock Agents, or AgentCore
• Own model serving and inference architecture on AWS, optimizing latency and cost
• Define LLMOps and MLOps standards across engagements (versioning, CI/CD)
• Drive evaluation, guardrails, and observability strategies with offline/online evals
• Lead governance for responsible AI and data isolation measures
• Guide build choices (fine-tuning vs prompting vs RAG) with cost/maintainability in mind
• Collaborate with leadership and clients on AI technical direction
• Communicate complex AI concepts clearly to engineers and non-engineers
• Mentor engineers and align AI architecture with broader business goals
• Develop and maintain AI architecture docs and reusable patterns

Skills

LLM architecture

Multi-agent orchestration

RAG and retrieval design

AWS model serving

LLMOps & MLOps

Observability & eval pipelines

Python & other languages

Cloud-native architecture

Responsible AI governance

Leadership & mentoring

Tools

Claude

Cursor

Job description

At Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencilsmeansengineering paired with creativity, because every agent we shiphas towork for real people in real workflows. That balance is baked into how we operate.

What You’ll Do

Craft & Delivery

• Define AI system architecture and lead design across LLM applications, agentic systems, retrieval, and ML model serving for end-to-end engagements
• Drive RAG and context-engineering strategy, including hybrid retrieval, reranking, chunking, and embedding model selection across vector stores
• Lead multi-agent orchestration designusing frameworks such asLangGraph, Amazon Bedrock Agents, orAgentCore
• Own model serving and inference architecture on AWS,optimizinglatency, throughput, and cost with streaming, caching, batching, and quantization
• DefineLLMOpsandMLOpsstandards across engagements, including prompt versioning, eval pipelines, CI/CD, and feature stores
• Drive evaluation, guardrails, and observability strategy,establishingoffline and online eval, LLM-as-judge scoring, and tracing
• Own the responsible AI posture for AI systems, including governance, PII and data isolation, prompt-injection defense, and grounding controls that reduce hallucination
• Lead decisions on build approach, including fine-tuning, prompting, RAG, and tool use, weighing accuracy, cost, and maintainability
• Bring an AI-forward mindset to your daily work, using tools like Claude, Cursor, and other modern AI assistants to ship higher-quality work at pace
• Partner with leadership and clients on AI technical direction, translating business goals into architectural decisions
• Communicate complex AI concepts and tradeoffs clearly to engineering and non-engineering stakeholders alike
• Engage closely with engineering, data, ML, and product teams to align AI architecture with broader business priorities
• Develop andmaintainAI architecture documentation, evaluation standards, and reusable patterns

Leadership & Influence

• Establish architectural standards and best practices that lift quality and consistency across the team
• Mentor junior and mid-level engineers, helping them grow their architectural thinking and craft
• Act as a technical escalation point on complex architectural and integration challenges
• Evaluate emerging technologies and recommend tools, frameworks, and patterns that improve architecture over time

What You’ll Bring

• 7+ years of professional software engineering experience, with at least 3 years in AI/ML architecture or technical leadership roles
• Deep expertise in LLM and agentic application architecture, including multi-agent orchestration and tool-calling patterns
• Strong experience designing production RAG and retrieval systems, including hybrid retrieval, reranking, and vector store selection
• Strong experience with model serving and inference optimization on AWS, including latency, throughput, and cost tuning
• Deep understanding of LLMOps and MLOps, including eval pipelines, prompt versioning, CI/CD, and observability for AI workloads
• Demonstrated experience with AI evaluation, guardrails, and tracing tooling
• Strong judgment on fine-tuning versus prompting versus RAG tradeoffs for accuracy, cost, and maintainability
• Expert software engineering background, with proficiency in Python and at least one other language (e.g., TypeScript, Java, Go)
• Strong background in cloud-native architecture, including microservices, serverless, containerization, and event-driven systems
• Strong understanding of responsible AI and governance, including data isolation, prompt-injection defense, and compliance
• Demonstrated leadership and technical mentoring experience across a team
• Demonstrable, day-to-day usage and expert knowledge of AI-forward tools such as Claude and Cursor
• Excellent problem-solving skills and the ability to navigate highly ambiguous technical and business challenges with sound judgment
• AWS AI certifications, fine-tuning or model customization experience, or responsible AI experience is a plus

Helpful Extras and Unique Skills

You’ll Do Well Here if You Are

• A doer. You see something broken and fix it. You'd rather move on clarity than wait for certainty.
• A fast learner who knows you don't know everything. The AI landscape changes weekly. You're senior enough to know better and curious enough to keep learning anyway.
• Direct in a way that makes the work better. You give honest feedback. You'd rather have the hard conversation than blow smoke.
• Obsessed with craft. You know genius is in the details. You ship exceptional, not perfect, and you don't put your name on work you wouldn't stand behind.
• Built for ownership. You honor commitments, admit mistakes fast, and back your teammates when a decision costs something. No handoffs, no finger-pointing.
• All in. You treat clients' businesses like your own. You take the work seriously without taking yourself seriously.
• Resourceful when the budget, timeline, or team is tight. Constraints don't slow you down. They sharpen you.
• Glad to be in the room with people who care as much as you do. Our teams average fifteen-plus years of experience. We hire people who push each other to do better work.

The below range reflects the range of possible compensation for this role at the time of this posting. We may ultimately pay more or less than the posted range. This range may be modified in the future. An employee's position within the salary range will be based on several factors including, but not limited to, relevant education, qualifications, certifications, experience, skills, seniority, geographic location, performance, shift, travel requirements, sales or revenue-based metrics, any collective bargaining agreements, and business or organizational needs. The salary range for this role is $146,071 USD - $201,541 USD.

We offer a comprehensive package of benefits including paid time off, medical/dental/vision insurance and 401(k) to eligible employees.

An offer of employment may be conditional upon successful completion of a background check in accordance with local legislation and our candidate privacy notice. Your current employer will not be contacted without your permission. We are committed to ensuring equal employment opportunities for all job applicants and employees. Employment decisions are based upon job-related reasons regardless of an applicant's race, color, religion, sex, sexual orientation, gender identity, age, national origin, disability, marital status, genetic information, protected veteran status, or any other status protected by law.