Principal AI Engineer

BASIL TECHNOLOGIES PTE. LTD.; Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

BASIL TECHNOLOGIES PTE. LTD. in Singapore is seeking a hands-on Principal AI Engineer to design, build and productionize agentic AI systems for cybersecurity use cases.

This role focuses on defining the agentic AI harness, control plane, memory and knowledge architecture, guardrails, observability, and production standards to deploy safe AI agents across cyber functions.

Qualifications

• Strong hands-on experience building production-grade LLM, agentic AI, ML, automation or platform systems.
• Deep understanding of agent architecture, orchestration frameworks, tool calling, memory design, RAG, model routing and multi-agent workflows.
• Experience with frontier models, open-source models or both, including evaluation, benchmarking and model comparison.
• Strong software engineering background, including Python, APIs, backend services, cloud platforms, containers, CI/CD, authentication, logging and production observability.
• Experience integrating AI systems with enterprise APIs, identity systems, data platforms, workflow engines, ticketing systems, coderepos and operational tools.
• Prior experience operating or supporting production systems, including monitoring, alerting, incident response, rollback, release management, access control, cost management and post-incident review.
• Practical understanding of production failure modes such as model drift, prompt regressions, broken tool calls, API failures, retrieval errors, permission issues, latency problems, data quality gaps, cost spikes and unsafe outputs.
• Practical understanding of AI safety risks, including hallucination, prompt injection, insecure tool use, excessive agency, sensitive data leakage, memory poisoning, adversarial manipulation and unsafe autonomous behaviour.
• Experience designing human-in-the-loop workflows for high-risk, regulated or security-sensitive environments.
• Ability to design for operational handover, including runbooks, support models, service ownership, observability, change control and measurable service health.

Responsibilities

• Design, build and productionize agentic AI systems for cybersecurity use cases.
• Define and implement the agentic AI harness, control plane, and memory/knowledge architecture.
• Develop guardrails, observability models and production standards for safe AI deployments.
• Collaborate with SOC, incident response and security SMEs to align AI systems with cyber operations.
• Ensure reliability, auditability, recoverability and supportability of production AI platforms.

Skills

LLM engineering

Agent architecture

Production systems

Python development

CI/CD pipelines

Observability

Security risk

Model evaluation

Job description

Job Description

Role Summary

We are seeking a hands‑on Principal AI Engineer to design, build and help production wise agentic AI systems for cybersecurity use cases.

This is an AI engineering role applied to cybersecurity. The role will define and build the agentic AI harness, control plane, model evaluation framework, AI‑to‑system interface layer, memory and knowledge architecture, guardrails, observability model and production standards needed to deploy AI agents safely across cyber functions.

Cybersecurity knowledge is useful, but not the primary requirement. The core requirement is deep experience building production‑grade LLM, agentic AI, ML, automation or platform systems. Cyber domain expertise will be provided by SOC, incident response, vulnerability management, AppSec, cloud security, IAM, GRC, threat intelligence, red‑team and security engineering SMEs.

The candidate should also have prior experience operating or supporting production systems, so they can design systems that are reliable, observable, auditable, recoverable and supportable. Day‑to‑day operations ma ysit with a separate AI platform, engineering or operations team.

Qualifications

Required Experience

• Strong hands‑on experience building production‑grade LLM, agentic AI, ML, automation or platform systems.
• Deep understanding of agent architecture, orchestration frameworks, tool calling, memory design, RAG, model routing and multi‑agent workflows.
• Experience with frontier models, open‑source models or both, including evaluation, benchmarking and model comparison.
• Strong software engineering background, including Python, APIs, backend services, cloud platforms, containers, CI/CD, authentication, logging and production observability.
• Experience integrating AI systems with enterprise APIs, identity systems, data platforms, workflow engines, ticketing systems, codere positories and operational tools.
• Prior experience operating or supporting production systems, including monitoring, alerting, incident response, rollback, release management, access control, cost management and post‑incident review.
• Practical understanding of production failure modes suchas model drift, prompt regressions, broken tool calls, API failures, retrieval errors, permission issues, latency problems, data quality gaps, cost spikes andunsafe outputs.
• Practical understanding of AI safety risks, includinghallucination, prompt injection, insecure tool use, excessive agency, sensitive data leakage, memory poisoning, adversarial manipulation and unsafe autonomousbehaviour.
• Experience designing human‑in‑the‑loop workflows for high‑risk, regulated or security‑sensitive environments.
• Ability to design for operational handover, including runbooks, support models, service ownership, observability, change control and measurable service health.

Preferred Experience

• Experience building AI agents for software engineering, code review, test generation, vulnerability discovery, workflow automation or enterprise operations.
• Experience with LangGraph, AutoGen, CrewAI, SemanticKernel, AgentSea, OpenAI Agents SDK, MCP, vector databases, graph databases or similar agentic AI tooling.
• Experience with RAG pipelines, knowledge graphs, structured retrieval, event schemas, data contracts and context engineering.
• Experience with secure connector patterns, permission boundaries, service accounts, API gateways, immutable audit logging and too lmediation.
• Experience with AI red teaming, model evaluation, AI governance, secure‑by‑design AI or regulated‑sector AI deployment.
• Experience designing or operating simulation environments, cyber ranges, replay systems, benchmark suites or adversarial test harnesses.
• Exposure to cybersecurity, AppSec, cloud security ,DevSecOps, vulnerability management, SOC operations, incident response, threat intelligence, GRC or offensive security testing.