Data & Eval Operations Program Manager

Mohamed bin Zayed University of Artificial Intelligence; Sunnyvale (CA); In loco; USD 250.000 - 450.000; Tempo pieno

Vantaggi offerti da questo lavoro

Visa sponsorship

Comprehensive health benefits

Generous paid time off

401K Plan

Parental Leave

Descrizione del lavoro

Mohamed bin Zayed University of Artificial Intelligence in Sunnyvale, CA is seeking a Data & Eval Operations Program Manager to design and run end-to-end data operations and evaluation programs for foundation models. You will partner with researchers, engineers, and external vendors to ensure high-quality data, timely evaluations, and scalable processes.

This role demands autonomous program ownership, strong budget discipline, and cross-functional collaboration with Legal and Procurement.

Competenze

• 5+ years in industry, with at least 3 years in program management with technical teams.
• Budget management experience with multi-million-dollar budgets.
• Vendor management across multiple vendors, with contracts and SOWs.
• Autonomous program ownership including budget, timeline, and quality.
• Technical fluency; understand core technical concepts.

Mansioni

• Establish end-to-end Data Ops program from scratch.
• Define an Eval Ops program and governance.
• Partner with legal and procurement on data partnerships.
• Tailor programs with researchers and engineers to changing needs.
• Monitor data and eval performance to guide strategy.

Conoscenze

Program management

Budget management

Vendor management

Cross-functional collaboration

Technical fluency

Formazione

Bachelor3s degree in CS/Engineering/Business/Economics

MBA or advanced technical degree is a plus

Descrizione del lavoro

About the Institute of Foundation Models

We are a dedicated research lab for building, understanding, using,and risk-managing foundation models. Our mandate is to advanceresearch, nurture the next generation of AI builders, and drivetransformative contributions to a knowledge-driven economy.

As part of our team, you’ll have the opportunity to work on thecore of cutting-edge foundation model training, alongsideworld-class researchers, data scientists, and engineers, tacklingthe most fundamental and impactful challenges in AIdevelopment. You will participate in the development ofgroundbreaking AI solutions that have the potential to reshapeentire industries. Strategic and innovative problem-solving skillswill be instrumental in establishing MBZUAI as a global hub forhigh-performance computing in deep learning, driving impactfuldiscoveries that inspire the next generation of AI pioneers.

The Role

As the Data & Eval Operations Program Manager for our team, youwill structure and implement the processes for data acquisition andeval execution for our foundation models and will develop theseinto comprehensive programs. You will work closely with ourtechincal team, including research scientists, research engineers,and software engineers, as well as with external data vendorpartners as our key interface. These programs will have a massiveimpact on the continuous improvement of the models we develop,meaning that this is a critical role to the ultimate success of theorganization. You will be expected to stand up and manage theseprograms autonomously, and tailor them over time to the changingneeds of the team.

Key Responsibilities

Data Operations

• Establish Data Ops Program: Establish a complete end-to-enddata operations program from scratch, to include program structure,goals, and management. You will be completely responsible forstanding up and owning this program.

• Understand the Landscape: Analyze and develop an understandingof the constantly evolving landscape of data offerings from new andexisting vendors, and assess how this fits the evolving needs ofIFM with technical leads.

• Vendor Management: Work closely with new and existing vendorson a regular basis. You will serve as the key interface between IFMand our data vendors.

• Budget and Performance Management: Perform budget managementand tracking, as well as performance management. You will beresponsible for the full data budget and will devise a rigorousprocess to ensure this spend is highly effective, with all spendtied back to actual data performance.

• Cross-Functional Collaboration: Work closely withcross-functional teams, including Legal and Procurement, to ensurenew data partnerships and acquisitions are completed quickly and inalignment with all organizational procedures.

Eval Operations

• Establish Eval Ops Program: Establish a complete end-to-endeval operations program from scratch, to include program structure,goals, and management. You will be completely responsible forstanding up and owning this program.

• Eval Process: Devise a process for how our team regularlycompletes, documents, organizes, and reviews our evals.

• Eval Process Changes: Work with technical leads to understandtheir evolving eval needs and adjust the eval process to meet theseneeds.

• Streamline Eval Service: Streamline the eval process andinternal eval service with our engineering team and with individualeval owners to improve how evals are completed.

• Tie Evals to Data: Develop a sense of where our models might belagging with respect to key evals and use this to inform strategyfor the data operations program.

• Understand the Landscape: Analyze and develop an understandingof the constantly evolving landscape of benchmarks that may be ofinterest to IFM.

Academic Qualifications

A bachelor's degree in computer science, engineering, business, oreconomics. (MBA or advanced technical degree is a plus).

Professional Experience - Minimum

• 5+ years of relevant industry experience, with at least 3 yearsin program management with technical teams, including workingdirectly with researchers and engineers.

• Budget Management: Budget management experience, managingmulti-million-dollar budgets.

• Vendor Management: Vendor management experience, working acrossmultiple vendors, and experience with contracts and SOWs.

• Complete Program Ownership: Autonomy in fully managing projectswith budget, timeline, and quality components.

• Technical Fluency: You don’t need to write code or understandnuanced research but must understand relevant technicalconcepts.

Professional Experience - Preferred

• Foundation Model Experience: Experience working specificallywith LLMs, Generative AI, or ML Infrastructure.

• Data Operations Experience: Experience working with datavendors for LLM development.

• LLM/Agentic Benchmarks Understanding: Familiarity with keyLLM/agentic benchmarks, including an understanding of how LLMs,aagentic capabilties, and benchmarks are evolving over time.

$250,000 - $450,000 a year

Compensation

$250,000 – $450,000/year + discretionary performance bonus +benefits

The posted salary range represents the company’s good faithestimate of the base compensation for this position upon hire. Theactual base compensation offered may vary within this rangedepending on individual qualifications, including but not limitedto relevant skills, experience, education, certifications, andspecific business needs. In addition to base salary, totalcompensation includes an annual discretionary performance bonus andcomprehensive benefits.

Benefits Include

• Comprehensive medical, dental, and vision benefits
• Bonus
• 401K Plan
• Generous paid time off, sick leave and holidays
• Paid Parental Leave
• Employee Assistance Program
• Life insurance and disability

This position is eligible for visa sponsorship.

We may use artificial intelligence (AI) tools to support parts ofthe hiring process, such as reviewing applications, analyzingresumes, or assessing responses and identifying potentialinconsistencies or verification signals in application materialsbased on available information. These tools assist our recruitmentteam but do not replace human judgment. Final hiring decisions areultimately made by humans. If you would like more information abouthow your data is processed, please contact us.