Principal AI Engineer

Robots and Pencils; United States; Remote; USD 180,000 - 280,000; Full time

Benefits offered by this job

20 days paid vacation

15 days unpaid vacation

All official public holidays off

5 paid sick days without a doctor’s证?

Medical insurance

Military draft deferment (reservation)

Job summary

Robots & Pencils is seeking a Principal AI Engineer to define and drive the technical direction of our AI/ML systems, leading architecture across the organization. You will shape the AI roadmap, solve complex integration challenges, and collaborate with leadership to scale AI work for clients.

You will mentor engineers, ensure production-grade reliability and security, and partner with product and data teams to deliver scalable AI solutions using AWS GenAI, Docker, Kubernetes, and modern tooling.

Qualifications

• 9+ years of professional software engineering experience with 5+ years focused on AI/ML systems in production.
• Strong software engineering background (Python or similar) with design sensibilities for distributed, scalable systems.
• Deep experience designing distributed systems and event-driven architectures for production AI workloads.
• Extensive cloud experience, especially AWS GenAI offerings.
• Proven track record shipping complex AI systems across multiple teams.
• Expertise with AI frameworks, orchestration tools, and observability for LLM apps.

Responsibilities

• Define AI architecture and technical strategy, leading end-to-end implementation from design to production.
• Build scalable ML platforms, pipelines, and workflow orchestration for model development at scale.
• Architect and build LLM-powered systems, including prompt engineering, function/tool calling, multi-agent orchestration, and retrieval techniques.
• Design and develop APIs and backend services that integrate AI into enterprise systems.
• Lead model development from research to production, ensuring production-ready results.
• Ensure AI reliability, security, and scalability with robust logging, monitoring and debugging.
• Mentor engineers, champion AI engineering standards, and drive technical vision across the organization.

Job description

Company Overview

Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into enterprise operations and deliver measurable results for our clients. We’re all in on AWS, combining deep UX capability with senior engineering talent to get AI into production fast and keep it there.
We’ve earned the trust of leaders across Consumer Products and Retail, Education, Energy, Financial Services, Healthcare, and Manufacturing and more, and earned a reputation as the nimble alternative to traditional global systems integrators. Founded in 2009, with delivery centers in Canada, the United States, Eastern Europe, and Latin America, we are smaller, faster, and more senior by design. Our teams average 15+ years of experience. We move fast, sweat the details, and build things that actually ship.

Position Overview

We’re looking for a Principal AI Engineer to define and drive the technical direction of our AI/ML systems. This role is ideal for a deeply experienced engineer who leads from the front — shaping architecture at the system level, solving the hardest problems, and raising the bar for how AI gets built and operated across the organization.

In this role, you will work on a cross-functional team, setting AI architecture standards, leading complex system design and implementation challenges, and solving integration problems that require both deep expertise and broad systems thinking. You’ll partner closely with leadership to define the AI roadmap and make high-stakes technical decisions that shape how our AI work scales across clients and over time.

Why This Role Matters

At Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencils means engineering paired with creativity, because every agent we ship has to work for real people in real workflows. That balance is baked into how we operate.

Every role here contributes directly to that mission. Here, you shape how AI systems integrate into enterprise operations, how teams move at real velocity, and how products create measurable impact for clients and the people they serve. We ship production-ready AI in 30 to 45 days. That pace demands people who take ownership, lead with craft, and care deeply about what they put their name on.

What You’ll Do

Craft & Delivery

• Define AI architecture and technical strategy, and lead implementation across the full lifecycle from design through production
• Build scalable ML platforms, pipelines, and workflow orchestration that support model development and event-driven, asynchronous operations at scale
• Architect and build LLM-powered systems, including prompt engineering, function/tool calling, multi-agent orchestration, RAG patterns, vector databases, embeddings, and streaming responses
• Design and develop APIs and backend services that integrate AI capabilities with enterprise systems and third-party platforms
• Lead model development, optimization, and the path from research to production, ensuring promising approaches translate into reliable, production-ready systems
• Ensure AI reliability, security, and scalability across deployed systems, including logging, monitoring, and debugging in production environments
• Bring an AI-forward mindset to your daily work, using tools like Claude, Cursor, and other modern AI assistants to ship higher-quality work at pace

Collaboration & Communication

• Co-define the AI roadmap with leadership, operating as a peer in strategic technical conversations
• Communicate complex technical concepts clearly to engineering and non-engineering stakeholders alike, translating depth into decisions others can act on
• Engage with product, engineering, and data teams to align AI work with broader business priorities

Leadership & Influence

• Define and champion AI engineering standards that shape how the organization builds and operates AI systems
• Mentor across the organization, shaping engineering culture and developing the next generation of technical leaders
• Own high-stakes architectural decisions that carry significant organizational and cross-engagement weight
• Drive technical vision — defining not just what gets built, but how AI engineering evolves at R&P over time

What You’ll Bring

• 9+ years of professional software engineering experience, with 5+ years focused on AI/ML systems in production and deep, hands-on expertise with generative AI development
• Expert software engineering background (Python or similar) with strong design sensibilities for distributed, scalable, and maintainable systems
• Deep experience designing distributed systems and event-driven architectures that underpin production AI workloads
• Deep expertise with cloud platforms, with mastery of AWS services and AWS GenAI offerings
• Proven track record architecting and shipping complex agentic systems in production across multiple teams or engagements
• Mastery of AI frameworks and orchestration tools, with clear judgment on when to adopt versus build
• Deep experience with evaluation frameworks and observability tools for LLM apps, including establishing these practices where they don’t yet exist
• Deep expertise in AI safety, responsible AI principles, prompt injection defenses, PII handling, and AI governance and compliance in production
• Extensive experience designing RAG architectures: chunking strategies, embedding models, vector databases, and advanced retrieval techniques
• API and systems design experience, including architecting and integrating with internal and third-party services at scale
• Advanced cost optimization expertise: token economics, caching strategies, model routing, quantization
• Strong working knowledge of Docker, Kubernetes, and cloud-native AI architecture patterns
• Expertise in MLOps and AI infrastructure, including model versioning, monitoring, deployment automation, and reproducibility
• Demonstrable, day-to-day usage and expert command of AI-forward coding tools such as Claude Code and Cursor
• Track record of technical leadership and mentoring at an organizational level, with influence that extends beyond the immediate team
• Excellent problem-solving and stakeholder communication skills, with the ability to navigate the most ambiguous technical and business challenges with sound judgment

You’ll Do Well Here if You Are

• A doer. You see something broken and fix it. You’d rather move on clarity than wait for certainty.
• A fast learner who knows you don’t know everything. The AI landscape changes weekly. You’re senior enough to know better and curious enough to keep learning anyway.
• Direct in a way that makes the work better. You give honest feedback. You’d rather have the hard conversation than blow smoke.
• Obsessed with craft. You know genius is in the details. You ship exceptional, not perfect, and you don’t put your name on work you wouldn’t stand behind.
• Built for ownership. You honor commitments, admit mistakes fast, and back your teammates when a decision costs something. No handoffs, no finger-pointing.
• All in. You treat clients’ businesses like your own. You take the work seriously without taking yourself seriously.
• Resourceful when the budget, timeline, or team is tight. Constraints don’t slow you down. They sharpen you.
• Glad to be in the room with people who care as much as you do. Our teams average fifteen-plus years of experience. We hire people who push each other to do better work.

We Offer

• 20 days of paid vacation.
• 15 days of unpaid vacation.
• All official public holidays off.
• 5 paid sick days without a doctor’s certificate.
• Medical insurance.
• Military draft deferment (reservation) support.
• Cooperation under a gig-contract.