Senior Software Engineer, Artificial Intelligence/LLM

We Love AI Jobs; San Carlos, Northern (CA, KY); Hybrid; USD 173,000 - 225,000; Full time

Job summary

Beacon AI in San Carlos is seeking a Senior Software Engineer to build and operate user-facing LLM features for aviation products, from retrieval and tool-calling flows through APIs, evaluations, and production monitoring.

You’ll collaborate with product and infra teams to improve quality, latency, and cost in a safety‑critical domain, owning features end-to-end with 5–8 years of experience in ML/LLM systems.

Qualifications

• 5-8 years of experience in software/ML/LLM systems.
• Production ML/LLM experience with end-to-end feature ownership.
• Strong coding skills in Python and/or TypeScript; ability to ship reliable services.

Responsibilities

• Build production RAG and tool-calling features with validated outputs and resilient service APIs.
• Create evaluations, metrics, and experiments to improve LLM quality, latency, and cost.
• Implement safety, privacy, and auditing controls, then monitor and troubleshoot production services.
• Collaborate with infra to design retrieval/data preparation pipelines and embeddings.
• Own the service layer APIs and workers with clear contracts, streaming, and backoff.

Skills

Python

TypeScript

LangChain

AWS Bedrock

OpenAI

Anthropic

OpenSearch

pgvector

Tools

LangChain

AWS Bedrock

OpenAI

Anthropic

Triton

TensorRT-LLM

OpenSearch

pgvector

Job description

# Senior Software Engineer, Artificial Intelligence/LLMBeacon AISan Carlos - HybridFull-time$173,000 – $225,000 / yearAI tools:LangChainAWS BedrockOpenAIAnthropicTritonTensorRT-LLMUpload your resume once and Scout applies for youApply with ScoutShare this job:LinkedInXSave imageBuild and operate user-facing LLM features for Beacon AI’s aviation products, from retrieval and tool-calling flows through APIs, evaluations, and production monitoring. You’ll work with product and infrastructure teammates to improve quality, latency, and cost in a safety-critical domain.* Full-time* Hybrid* 5-8 years### Skills and expertisePythonTypeScriptLangChainAWS BedrockOpenAIAnthropicOpenSearchpgvector### Key responsibilities* Build production RAG and tool-calling features with validated outputs and resilient service APIs.* Create evaluations, metrics, and experiments to improve LLM quality, latency, and cost.* Implement safety, privacy, and auditing controls, then monitor and troubleshoot production services.## How this job comparesCompared with 410 open senior machine learning jobs on October 10, 2026.Pay: This job pays $173K to $225K a year. The 143 open senior machine learning jobs that publish US pay start at $186K on average, so this job starts about 7% lower.Similar open jobs: We Love AI Jobs lists 410 open senior machine learning jobs at 176 companies.Remote work: This job is hybrid. 90 of the 410 senior machine learning jobs (22%) are remote.AI in this role: High. AI is the core of the work: building, training or deploying AI systems. Shipping and improving LLM-powered product features is the core of the role. Daily work includes retrieval, tool calling, model integrations, evaluations, guardrails, and production monitoring.## Full descriptionAbout Beacon AIWe’re a fast-moving team of aviators, engineers, and operators building an AI platform to make flying safer, more efficient, and more capable. Backed by top investors, we’ve secured a dozen Department of Defense contracts and partnered with major airlines to deliver mission-critical systems. We operate without silos or heavy processes. Small, focused teams own what they build, ship quickly, and learn fast, pushing the boundaries of how humans and AI work together in aviation.You will ship LLM-powered product features end-to-end. That means designing retrieval and tool-calling flows, writing the services that run them, building evals and guardrails, and watching cost, latency, and quality in production. You’ll partner with the ML/infra teammates on embeddings, indexing, and model hosting, and with the product teammates on user experience and outcomes. We move fast, and we care about reliability in a safety-critical domain.This role is for senior engineers who can own a feature or service end-to-end (5-8 years experience, including some in production ML/LLM systems). You'll make independent architecture and eval decisions within your service's scope and partner directly with product and infra.What you’ll doBuild user-facing LLM features• Design and implement retrieval-augmented generation and tool-calling flows using frameworks like LangChain or equivalent primitives, where simpler is better.• Deliver robust JSON and schema-bound outputs with validation, retries, and fallbacks.• Add function calling to integrate with internal tools, search, routing, and data services.Own the service layer• Ship APIs and workers in Python or TypeScript with clear contracts, streaming, and backoff.• Add caching, request shaping, prompt templates, and context packing to control latency and cost.• Integrate with AWS Bedrock, OpenAI, Anthropic, or self-hosted endpoints as needed.Retrieval and data prep• Collaborate with infrastructure teammates to develop chunking, embeddings, and indexing capabilities for documents, time series, and multimedia.• Choose and tune vector backends such as OpenSearch, pgvector, or Pinecone.• Keep knowledge bases fresh with data syncs from S3, Aurora, DynamoDB, and external sources.Evaluation and quality• Create offline evals and golden sets for prompts, retrievers, and tools.• Stand up online metrics for task success, hallucination rate, retrieval precision/recall, p95 latency, and cost per request.• Run A/B tests and prompt/version rollouts with guardrails and canaries.Safety, privacy, and compliance• Implement content and policy checks, PII detection and redaction, access controls, and auditing.• Design human-in-the-loop paths for sensitive actions.• Handle aviation data with care and follow internal security standards.Operate what you build• Add tracing, logs, and dashboards for model calls, token usage, errors, and saturation.• Debug tricky failures across retrieval, prompts, tools, and providers.What will make you successful• Shipped LLM apps: You’ve put LLM features in front of users and improved them with data.• Strong builder: Comfortable writing production code, tests, and docs. You keep things simple and observable.• RAG and tools depth: You understand embeddings, chunking, vector search tradeoffs, and function calling.• Quality mindset: You design evals, define success metrics, and iterate based on evidence.• Cost and latency aware: You track p95, hit SLAs, and reduce cost without hurting quality.• Clear communicator: You explain tradeoffs and align partners across product, infra, and security.• Ownership: You can take a feature from design through production with minimal oversight.