Member of Technical Staff - Agent DX Research

Rozgoo; New York (NY); On-site; USD 120,000 - 170,000; Full time

Job summary

Modal is seeking an experienced researcher to help make agent-based development more productive by defining quantitative objectives and building evaluation frameworks. You will work with the SDK team and other engineers to measure performance and translate results into product improvements.

Ideal candidates have a PhD in a quantitative field and experience in experimental design, statistical analysis, and coding agents. In-person work from NY or SF is preferred.

Qualifications

• PhD or equivalent training in a quantitative field preferred.
• Experience designing and running scalable agent benchmarking workflows.
• Strong background in experimental design and statistical evaluation.
• Knowledge of coding agents and their current capabilities.

Responsibilities

• Define quantitative objectives for agent productivity evaluation.
• Design systems to measure performance of coding agents.
• Collaborate with SDK team and product engineers to build evaluation framework.
• Translate results into product improvements and roadmap inputs.
• Stay updated on new developments in tools and workflows; engage with customers to understand usage.

Skills

Benchmark design

Statistical evaluation

Experimental design

Coding agents

Developer tooling

Communication skills

In-person collaboration

Education

PhD in CS/HCI/Cognitive Science/OR or related field

Job description

About Us: Modal provides the infrastructure foundation for AI teams. With instant GPU access, sub-second container startups, and native storage, Modal makes it simple to train models, run batch jobs, and serve low-latency inference. We have thousands of customers who rely on us for production AI workloads, including Lovable, Scale AI, Substack, and Suno. We're a fast-growing team based out of NYC, SF, and Stockholm. We've hit 9-figure ARR and recently raised a Series B at a $1.1B valuation. Our investors include Lux Capital , Redpoint Ventures , Amplify Partners , and Elad Gil . Working at Modal means joining one of the fastest-growing AI infrastructure organizations at an early stage, with many opportunities to grow within the company. Our team includes creators of popular open-source projects (e.g. Seaborn , Luigi ), academic researchers, international olympiad medalists, and experienced engineering and product leaders with decades of experience.

The Role:

Modal has always obsessed over developer experience and productivity. With rapid advancements in the capabilities of AI coding agents, the practice of developing software and the meaning of developer experience is changing. We see this as an opportunity. We’re looking for an experienced researcher to join us and help make it even easier and more productive to build on Modal. We believe that our code-first approach to AI infrastructure is uniquely well suited to agent-based development. But we’re looking to do even better by subjecting agent productivity to rigorous evaluation and using those insights to guide the development of our platform. You’ll work in collaboration with Modal’s SDK team and other product engineers to build out a framework and process for agent productivity evaluation. Our goal is to treat developer experience optimization as a scientific problem. You’ll be responsible for defining quantitative objectives, designing systems to measure performance, and translating results into product improvements. You’ll also be expected to stay on top of new developments in tools and workflows and to work with our customers to understand how they’re using coding agents with Modal and where we can be providing more value.

Requirements:

This is a new kind of role, and we don’t have one specific background in mind. Training in quantitative research is preferred: you might have a PhD in Computer Science, Human Computer Interaction, Cognitive Science, Operations Research, or other related field. You also might have prior experience working as a Machine Learning Scientist, Quantitive UX Researcher, or other similar role on a product team. Regardless of your exact background, we’ll be looking for the following:

• Sufficient technical skills to design and implement scalable agent benchmarking workflows
• Experience with experimental design, measurement, and statistical evaluation
• Up-to-date knowledge of the latest advances in coding agents (with a dose of healthy skepticism about their current capabilities)
• Interest in developer tooling and opinions about developer ergonomics
• Familiarity with the use cases that Modal serves (generative AI inference, large-scale batch jobs, multi-node training, etc.)
• Strong communication skills and the ability to convey research insights to decision makers
• The ability to work in person from our New York (preferred) or San Francisco office