Senior AI Engineer

Checkmarx; United States; Remote; USD 150,000 - 210,000; Full time

Job summary

Checkmarx is seeking a Senior AI Engineer to join an R&D team building the AI-driven core of our product. You will design, implement, and own LLM-powered pipelines and agent-based systems that find, triage, and remediate security risk at scale in production.

The role emphasizes ground-truthing model behavior, evaluating tools and models, andscaling cloud-native architectures in Go. Strong knowledge of RAG, memory strategies, and observability is required.

Qualifications

• 5+ years of professional software engineering experience.
• Hands-on experience with LLMs and AI agents.
• Experience designing RAG pipelines, retrieval strategies, or context/memory systems for LLM apps.
• Experience evaluating model/agent quality (accuracy, latency, cost) and using evals to guide decisions.
• Experience with cloud AI services (AWS Bedrock, Azure OpenAI, Vertex AI).
• Strong proficiency in Go and/or Python.
• Solid understanding of distributed systems (message queues, gRPC/REST, relational DB) and Docker/Kubernetes.
• Strong testing, CI/CD, and observability fundamentals.
• Excellent English communication skills.

Responsibilities

• Design, build, and own LLM-powered pipelines and agent-based systems at scale.
• Design retrieval and context strategies grounding model behavior in product data.
• Define and run evaluations for model/agent quality and drive iteration.
• Evaluate emerging AI tools and provide build-vs-buy-vs-integrate recommendations.
• Stay ahead of AI landscape and translate developments into engineering proposals.
• Own reliability of AI features in production: guardrails, validation, observability.
• Participate in design discussions and code reviews with AI perspective.

Skills

LLMs & AI agents

Go

Python

Distributed systems

CI/CD & testing

English communication

Tools

Docker

Kubernetes

gRPC

REST APIs

Job description

Who We Are

Checkmarx is the leader in application security, ensuring that enterprises worldwide can secure their application development from code to cloud. Our consolidated platform and services address the needs of enterprises by improving security and reducing TCO, while simultaneously building trust between AppSec, developers, and CISOs. At Checkmarx, we believe it's not just about finding risk, but remediating it across the entire application footprint and software supply chain with one seamless process for all relevant stakeholders. We are honored to serve more than 1,800 customers, including 40 percent of all Fortune 100 companies, such as Siemens, Airbus, Salesforce, Stellantis, Adidas, Walmart, and Sanofi.

About The Role

We're looking for a Senior AI Engineer to join one of our R&D teams building the AI-driven core of our product. Beyond building, we want someone who tracks the AI landscape closely: new models, frameworks, and agent architectures as they emerge; and who can turn that knowledge into real recommendations: evaluating whether a new tool, model, or technique is worth adopting, and helping the team decide what to build vs. buy vs. integrate.

The specific product area may evolve with our roadmap, but the core of the job stays constant: AI-driven features, applied judgment about the AI tooling landscape, and cloud-native architecture in Go.

What You’ll Do

• Design, build, and own LLM-powered pipelines and agent-based systems that find, triage, and remediate security risk in production, at scale
• Design retrieval and context strategies (RAG, tool use, memory) that ground model behavior in our product's data and domain
• Define and run evaluations for model/agent quality and use them to drive iteration and model selection
• Evaluate emerging AI tools, models, and frameworks against our product needs, producing clear build-vs-buy-vs-integrate recommendations for the team and leadership
• Stay ahead of the AI landscape: new LLM releases, agent architectures, orchestration frameworks, and techniques and translate relevant developments into concrete engineering proposals
• Own the reliability of AI features in production: guardrails, structured output validation, fallback behavior, and observability tuned for LLM-based systems
• Participate in design discussions and code reviews, bringing an AI-informed perspective to architecture decisions

What You’ll Need

• 5+ years of professional software engineering experience, with strong hands-on experience building with LLMs and AI agents, not just experimenting with them
• Experience designing RAG pipelines, retrieval strategies, or context/memory systems for LLM applications
• Practical experience evaluating model and agent quality (accuracy, hallucination, latency, cost) and using evals to guide decisions
• A demonstrated habit of evaluating and comparing AI tools, models, or vendors, with informed opinions on tradeoffs
• Experience with cloud AI services (AWS Bedrock, Azure OpenAI, Vertex AI)
• Strong proficiency in Go and/or Python
• Solid understanding of distributed systems (message queues, gRPC/REST APIs, relational databases) and experience with Docker/Kubernetes
• Strong testing, CI/CD, and observability fundamentals
• Excellent English communication skills

Nice to have

• A genuine research habit, following AI research/practitioner communities
• Experience with vector databases and embedding models
• Application security knowledge
• GitOps tooling (ArgoCD, Helm), OpenTelemetry

What we have to offer

Checkmarx offers a great work environment, professional development, challenging careers, competitive compensation, great work-life balance, as well as great benefits and perks throughout the year.

Checkmarx is an Equal Opportunity Employer. Employment decisions are made without regard to race, color, religion, national or ethnic origin, gender, sexual orientation, gender identity or expression, age, disability, or other characteristics protected by law.