Prompt Engineer

BayOne Solutions; Mountain View (CA); On-site; USD 130,000 - 180,000; Full time

Job summary

BayOne Solutions in Mountain View, CA seeks an innovative Context / Prompt Engineer to bridge LLM capabilities with business applications. You will design, optimize, and maintain system prompts, context pipelines, and RAG architectures to ensure accurate and instruction-aligned outputs.

You will collaborate with Data/ML engineers and Product Managers to integrate prompts into core APIs, and help prepare datasets for fine-tuning. This is a long-term contract role on-site in Mountain View.

Qualifications

• Deep understanding of LLMs, context limits, and model behavior.
• Experience building evals, benchmarks, and testing pipelines.
• Ability to craft precise, instruction-aligned prompts.
• Excellent written English with clear technical exposition.

Responsibilities

• Design, test, and refine system prompts and context windows.
• Optimize context retrieval and injection for RAG workflows.
• Establish evals/benchmarks and monitoring pipelines.
• Implement guardrails and safety measures against prompts risks.
• Collaborate with Data/ML engineers and product managers.
• Assist in model fine-tuning and data preparation.

Skills

LLM Expertise

Prompting Mastery

Python

AI Frameworks

Eval Frameworks

Structured Thinking

Tools

LangChain

LlamaIndex

LiteLLM

Guidance

Job description

Job Location: Mountain View, CA (4 days onsite)

Job Type: Long Term Contract (12+ Months)

Role Overview

We are seeking an innovative and analytical Context / Prompt Engineer to join our AI & Product team. In this role, you will bridge the gap between complex Large Language Model (LLM) capabilities and business applications. You will be responsible for designing, optimizing, and maintaining system prompts, complex context pipelines, and prompt architectures (such as Retrieval-Augmented Generation / RAG) to ensure our AI models deliver accurate, instruction-aligned, and safety-compliant outputs.

Key Responsibilities

• Prompt Engineering & System Instruction: Design, test, and refine system prompts, instruction templates, and context windows across diverse LLM architectures (e.g., OpenAI GPT, Anthropic Claude, open-source models).
• Context Architecture & RAG: Optimize context retrieval and injection strategies, ensuring models receive relevant, dense, and structured data for downstream tasks.
• Evaluations & Benchmarking: Establish evaluation metrics (evals), benchmark suites, and testing pipelines to measure prompt performance, latency, accuracy, hallucination rates, and instruction adherence.
• Safety & Alignment: Implement guardrails, red-teaming protocols, and edge-case handling to mitigate risks such as prompt injections, jailbreaks, data leakage, and toxic outputs.
• Cross-Functional Collaboration: Partner with Data Engineers, Machine Learning Engineers, and Product Managers to integrate optimized prompts into core software APIs and product workflows.
• Model Optimization & Fine-Tuning: Collaborate with ML engineers to format and curate high-quality datasets for fine-tuning or few-shot learning setups.

Qualifications & Skills

Must-Have

• LLM Expertise: Deep operational understanding of LLMs, attention mechanisms, context limits, tokenization, and model behavior variations.
• Prompting Mastery: Proficiency in advanced techniques including Chain-of-Thought (CoT), Few-Shot learning, ReAct, System Message steering, and Structured Output generation (JSON/Pydantic schemas).
• Technical Skills: Proficiency in Python and familiarity with AI frameworks/libraries (e.g., LangChain, LlamaIndex, LiteLLM, Guidance, or instructor).
• Data & Evals Experience: Experience building automated eval frameworks using tooling like Ragas, DeepEval, Braintrust, or custom scoring scripts.
• Structured Thinking: Exceptional written English and natural language clarity; ability to express complex logic precisely in text.

Preferred / Nice-to-Have

• Experience working with Vector Databases (e.g., Pinecone, Weaviate, Qdrant, Milvus) and hybrid search strategies.
• Background in Computational Linguistics, Natural Language Processing (NLP), Computer Science, or Cognitive Science.
• Understanding of API integration, RESTful services, and asynchronous programming in Python.

BayOne is an Equal Opportunity Employer and does not discriminate against any employee or applicant for employment because of race, color, sex, age, religion, sexual orientation, gender identity, status as a veteran, and basis of disability or any federal, state, or local protected class.

This job posting represents the general duties and requirements necessary to perform this position and is not an exhaustive statement of all responsibilities, duties, and skills required. Management reserves the right to revise or alter this job description.