AI Optimization Lead: LLM Evaluation & Delivery

Franklin Templeton; New York (NY); Hybrid; USD 156,000 - 183,000; Full time

Benefits offered by this job

Annual discretionary bonus

401(k) with generous match

Healthcare options

Employee stock investment program

Education assistance (LEAP)

Paid time off

Job summary

Franklin Templeton, a leading asset manager, is seeking a Lead - AI Optimization to drive the delivery and improvement of AI solutions across business groups. You translate business needs into technical approaches and guide initiatives from discovery through deployment.

You will build reusable evaluation tools for LLMs, optimize prompts and retrieval, and run structured experiments with clear hypotheses. This role is hands-on, individual contributor, with no direct reports, and requires strong

Qualifications

• Bachelors in CS/Engineering or quantitative field with 6+ years in applied ML/AI including production LLMs.
• Six or more years in applied ML/AI engineering, including production LLM or model-serving systems.
• Advanced degree is preferred.

Responsibilities

• Translate business needs into practical AI designs and guide initiatives from discovery to deployment.
• Develop reusable evaluation tools for LLMs and applications, including benchmarks and datasets.
• Improve quality and performance via prompting, retrieval, and model tuning.
• Run structured experiments and AB tests with guardrails and rollback criteria.
• Communicate progress, risks, and decisions across teams and stakeholders.
• Mentor engineers and share reusable evaluation components.

Skills

Python development

Technical leadership

LLM evaluation

Experimentation & statistics

Excellent communication

Education

Bachelor's degree in CS/Engineering or quantitative field

Advanced degree preferred

Tools

LLM tooling

Job description

Franklin Templeton, a leading asset manager, is seeking a Lead - AI Optimization to drive the delivery and improvement of AI solutions across business groups. You translate business needs into technical approaches and guide initiatives from discovery through deployment.

You will build reusable evaluation tools for LLMs, optimize prompts and retrieval, and run structured experiments with clear hypotheses. This role is hands-on, individual contributor, with no direct reports, and requires strong