ML Safety Research Engineer

Apple Inc.; San Francisco (CA); On-site; USD 181,100 - 272,100; Full time

Benefits offered by this job

Comprehensive medical and dental coverage

Retirement benefits

Employee stock purchase plan

Educational expense reimbursement

Job summary

A leading technology company located in San Francisco is seeking a Machine Learning Research Engineer to design and develop safe AI benchmarking methodologies. This role involves collaboration with various teams to implement responsible evaluation techniques. Candidates should possess a strong research background, proficiency in Python, and experience in AI/ML model evaluation. The position offers competitive compensation, stock option opportunities, and comprehensive benefits including medical, dental coverage, and educational reimbursement.

Qualifications

• 1+ years of work experience either as a postdoc or in the industry.
• Experience working with or evaluating AI/ML models, preferably LLMs.
• Ability to interpret unstructured data and derive insights.

Responsibilities

• Design benchmarking methodologies across multiple media and applications.
• Develop automated evaluation pipelines for model outputs.
• Work closely with engineering teams to translate findings into model improvements.

Skills

Empirical evaluation

Experimental design

Benchmarking

Python proficiency

Analytical skills

Communication skills

Education

Advanced degree (MS or PhD) in Computer Science or related field

Tools

Python libraries (pandas, NumPy, PyTorch)

Annotation tools

Job description

San Francisco, California, United States Machine Learning and AI

Apple Services Engineering (ASE) powers many AI features across App Store, Music, Video and more. We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating systemic biases and maintain safe and trustworthy experiences across our AI tools and models.

Description

Our team, part of Apple Services Engineering, is looking for an ML Research Engineer to lead the design and continuous development of automated safety benchmarking methodologies. In this role, you will investigate how media‑related agents behave, develop rigorous evaluation frameworks and techniques, and establish scientific standards for assessing risks they pose and safety performance. This role supports the development of scalable evaluation techniques that ensure our engineers have the right tools to assess candidate models and product features for responsible and safe performance. The capabilities you build will allow for the generation of benchmark datasets and evaluation methodologies for model and application outputs, at scale, to enable engineering teams to translate safety insights into actionable engineering and product improvements. This role blends deep technical expertise with strong analytical judgment to develop tools and capabilities for assessing and improving the behavior of advanced AI/ML models. You will work cross‑functionally with Engineering and Project Managers, Product, and Governance teams to develop a suite of technologies to ensure that AI experiences are reliable, safe, and aligned with human expectations. The successful candidate will take a proactive approach to working independently and collaboratively on a wide range of projects. In this role, you will work alongside a small but impactful team, collaborating with ML and data scientists, software developers, project managers, and other teams at Apple to understand requirements and translate them into scalable, reliable, and efficient evaluation frameworks.

Responsibilities

• Design scientifically‑grounded benchmarking methodologies covering multiple dimensions of responsibility and safety across several media and application marketplace use cases
• Develop automated evaluation pipelines that collect, automatically judge, and analyze model outputs with respect to safety policies, at scale
• Create and curate datasets, tasks, and feature usage scenarios that represent realistic and adversarial use cases across multiple languages, markets, and domains
• Define and validate new metrics for complex phenomena such as multi‑turn agentic interaction patterns
• Apply statistical rigor and reproducibility to above mentioned objectives
• Work closely with engineering and research teams to translate experimental findings into actionable model improvements and safety mitigations
• Publish internal reports and external papers
• Monitor evolving industry practices and academic work to ensure benchmarks remain relevant

Minimum Qualifications

• Advanced degree (MS or PhD) in Computer Science, Software Engineering, or equivalent research/work experience
• 1+ years of work experience either as a postdoc or in the industry
• Strong research background in empirical evaluation, experimental design, or benchmarking
• Strong proficiency in Python (pandas, NumPy, Jupyter, PyTorch, etc.)
• Deep familiarity with software engineering workflows and developer tools
• Experience working with or evaluating AI/ML models, preferably LLMs or program synthesis systems
• Strong analytical and communication skills, including the ability to write clear reports
• Technical Skills:
• Proficiency in Python (pandas, NumPy, Jupyter, PyTorch, etc.).
• Experience working with large datasets, annotation tools, and model evaluation pipelines
• Familiarity with evaluations specific to responsible AI and safety, hallucination detection, and/or model alignment concerns
• Ability to design taxonomies, categorization schemes, and structured labeling frameworks
• Analytical Strength: Ability to interpret unstructured data (text, transcripts, user sessions) and derive meaningful insights
• Communication: Strong ability to stitch together qualitative and quantitative insights into actionable guidance; strong ability to communicate complex architectures and systems to a variety of stakeholders
• Education in Data Science, Linguistics, Cognitive Science, HCI, Psychology, Social Science, or a related field

Preferred Qualifications

• Publications in AI/ML evaluation or related fields
• Experience with automated testing frameworks
• Experience constructing human‑in‑the‑loop or multi‑turn evaluation setups
• Intermediate or Advanced Proficiency in Swift
• Familiarity with RAG systems, reinforcement learning, agentic architectures, and model fine‑tuning
• Expertise in designing annotation guidelines and validation instruments and techniques
• Background in human factors, social science, and/or safety assessment methodologies

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $181,100 and $272,100, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan. You’ll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses — including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits.

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant.

Apple accepts applications to this posting on an ongoing basis.