Applied Scientist II, Core Shopping Data Science

Socket.dev; Seattle (WA); On-site; USD 120,000 - 180,000; Full time

Job summary

Socket.dev in Seattle is seeking an engineer to own an LLM-based measurement pipeline for CX, extending the framework to the homepage and detail page and building the measurement from scratch. You will turn perception questions into measurable artifacts and create scalable tooling to support evaluation at scale.

The role focuses on reusable labeling pipelines, evaluation frameworks, and inference infrastructure that can be adopted by teams handling model judgments and ground truth at scale,

Job description

Some CX shopping defects might be straightforward to detect and track. The interesting ones are not because they depend on what a customer perceives. For example, a search page may return legitimately different results, yet a shopper has no way to tell apart. We turn these ambiguous perception questions into a measurable artifact using LLMs, and we build the frameworks to know exactly where model judgment can be trusted and where a human must decide and proving it, against ground truth, at Amazon scale.

This is one example of an LLM based measurement pipeline you will own, but that's not all. You will extend that measurement to other parts of the shopping experience like the homepage and the detail page, where the same customer problem looks nothing like it does in search results, and where you will design the measurement from scratch.

The larger goal is what makes this role unusual. Teams across Amazon are each independently figuring out how to label quality with LLMs, hitting the same problems alone: prompts that break on the next model version; golden sets nobody audited, accuracy that collapses in other locales. Through the work above, you will set the standard and build the production tooling behind it. Reusable labeling pipelines, evaluation frameworks, and inference infrastructure that hold up against Amazon-sized data and get adopted by teams who did not have to use them.

Key job responsibilities