AI Optimization Lead: Scale Evaluation & Impact

Franklin Templeton; San Ramon (CA); Hybrid; USD 156,000 - 183,000; Full time

Benefits offered by this job

Annual discretionary bonus

401(k) match

Health insurance

Stock investment program

LEAP education assistance

Generous PTO and leave options

Job summary

Franklin Templeton is seeking a Lead - AI Optimization to drive technical delivery of AI solutions across business groups. You will translate needs into practical designs, guide testing, deployment, and production readiness while mentoring engineers and sharing reusable components.

The role focuses on building evaluation capabilities for LLMs and AI apps, establishing benchmarks, and guiding model selection with emphasis on quality, reliability, latency, and cost.

Qualifications

• Bachelor's degree or equivalent practical experience in a technical field and 6+ years in applied ML/AI engineering.
• Experience leading through ambiguity with cross-functional partners and mentoring engineers.
• Strong hands-on Python development, design, testing, and production-minded coding.
• Experience evaluating LLMs and AI applications using datasets and automated tests.
• Proven ability to design experiments, analyze results, and drive decisions.

Responsibilities

• Translate business needs into practical AI system designs and lead delivery.
• Develop reusable evaluation tools and benchmarks for LLMs and AI apps.
• Improve quality, latency, and cost through prompting and retrieval optimization.
• Run structured experiments and A/B tests with clear hypotheses and guardrails.
• Communicate trade-offs, decisions, and risks to stakeholders.

Skills

Python development

LLM evaluation

prompt optimization

retrieval optimization

communication

technical leadership

Education

Bachelor's degree in CS/Engineering or quantitative field

Advanced degree preferred

Tools

Python

LLM frameworks

Vector tooling

Model serving systems

Job description

Franklin Templeton is seeking a Lead - AI Optimization to drive technical delivery of AI solutions across business groups. You will translate needs into practical designs, guide testing, deployment, and production readiness while mentoring engineers and sharing reusable components.

The role focuses on building evaluation capabilities for LLMs and AI apps, establishing benchmarks, and guiding model selection with emphasis on quality, reliability, latency, and cost.