Principal AI Engineer — Agentic Systems & AI Infrastructure

Hashmark Labs; Myrtle Point (OR); Remote; USD 110,000 - 248,000; Part time

Job summary

Hashmark Labs is seeking a Principal AI Engineer to independently architect, engineer and deploy sophisticated AI systems, with deep expertise in agentic workflows and AI-native infrastructure. This is not a prototype role; you will personally ship production AI systems and own complex technical delivery from architecture to scale.

You will build agentic AI architectures, RAG and enterprise search, LLM evaluation and observability, multimodal pipelines and scalable AI infrastructure, with

Qualifications

• Strong backend architecture and distributed-systems fundamentals.
• Production experience with LLMs, AI agents, tool calling, structured outputs and agent orchestration.
• Hands-on experience with FastAPI, PostgreSQL/pgvector, vector databases, Redis, Kafka, Docker, Kubernetes, AWS or GCP.

Responsibilities

• Architect, engineer and deploy production AI systems from architecture to scale.
• Own complex technical delivery for client projects with measurable business outcomes.
• Collaborate with international product and engineering teams while retaining hands-on ownership of critical decisions.

Skills

Python engineering

Distributed systems

Backend architecture

LLMs & AI agents

Tools

FastAPI

PostgreSQL/pgvector

Vector databases

Redis

Kafka

Docker

Kubernetes

AWS

GCP

Job description

Principal AI Engineer — Agentic Systems & AI Infrastructure

• ₹40,000 – ₹1.3L No equity
• Remote (Everywhere)
• 4 years of exp
• Contract

About Hashmark Labs

Hashmark Labs is a senior-led engineering consultancy building production-grade systems across AI, Web3 and FinTech for international clients.

We are looking for an exceptional Principal AI Engineer who can independently architect, engineer and deploy sophisticated AI systems, with deep expertise in agentic workflows, retrieval, evaluation and AI-native infrastructure.

This is not a prompt-engineering or prototype-only role. We need an experienced engineer who has personally shipped production AI systems and can own complex technical delivery from architecture to scale.

What You'll Build

• Agentic AI Systems: Multi-agent architectures, graph-based orchestration, planning, routing, tool calling, memory, queues and human-in-the-loop workflows.
• RAG & Enterprise Search: Hybrid retrieval, embeddings, vector databases, metadata-aware reranking, multi-tenant knowledge systems and access control.
• LLM Evaluation & Observability: Evaluation datasets, quality metrics, tracing, model benchmarking, regression testing and reliable production monitoring.
• Multimodal & Document Intelligence: AI pipelines processing PDFs, tables, images, voice and complex enterprise information with traceable outputs.
• AI-Native Infrastructure: Scalable APIs, asynchronous services, distributed workloads, data pipelines, containerized/cloud deployments and cost-efficient inference.
• End-to-End Technical Ownership: Translate complex client requirements into robust architectures, production implementations and measurable business outcomes.

What We're Looking For

• Exceptional Python engineering with strong backend architecture and distributed-systems fundamentals.
• Deep production experience with LLMs, AI agents, tool calling, structured outputs and agent orchestration.
• Strong expertise in RAG, vector search, retrieval optimization and enterprise AI systems.
• Practical experience with LLM evaluation, observability, reliability engineering and performance optimization.
• Hands-on experience with FastAPI, PostgreSQL/pgvector, vector databases, Redis, Kafka, Docker, Kubernetes, AWS or GCP.

Preferred: Experience building enterprise SaaS, financial AI, underwriting or decision systems, voice agents, multimodal platforms, regulated workflows or AI + blockchain applications.

We value demonstrated engineering depth and systems ownership over arbitrary years of experience.

How We Work

Our engineering philosophy is evaluation-first and production-focused.

We prioritize measurable accuracy, grounded outputs, system reliability, data security, latency, cost efficiency and maintainable architectures.

You will collaborate directly with international product and engineering teams while retaining hands-on ownership of critical technical decisions.

Engagement & Compensation

• Type: Contractual engagement
• Location: Fully remote — worldwide
• Commitment: Approximately 5–6 hours/day, 5 days/week
• Schedule: Flexible working hours
• Compensation: Negotiable based on experience, technical expertise and demonstrated production ownership
• Performance-based revisions: Compensation may be reviewed and increased based on delivery and project performance, potentially including increases of 50% or more. Such revisions are not guaranteed.
• International payments: USDC or INR arrangements may be discussed where applicable
• Flexibility: Other professional engagements are permitted, subject to meeting agreed commitments