Data Scientist, Decision Quality

Socket.dev; Shelton (CT); On-site; USD 90,000 - 130,000; Full time

Job summary

Sperry is building a data science team to retrospectively score analyst decisions, understand where the pipeline errs, and improve training, tooling, and thresholds. You will work closely with risk analytics and production engineering in a hands-on, impact-driven role with real data and tight collaboration with operations.

The ideal candidate thrives on challenging problems, communicates clearly with stakeholders, and can lead workstreams while delivering robust, well-documented code and

Qualifications

• Strong proficiency in Python (NumPy, Pandas, Scikit-learn, or similar) and SQL.
• Applied statistics including classification metrics, sampling, inter-rater agreement, bias, and experimental design.
• Supervised machine learning on imbalanced and imperfectly labeled data.
• Experience evaluating decisions against late, partial, or no ground truth.
• Ability to explain methods and uncertainty to operational audiences.
• Familiarity with Git, CI/CD pipelines, and agile practices.

Responsibilities

• Build retrospective scoring of analyst dispositions using available data.
• Quantify agreement and variation across analysts, shifts, territories, and test conditions.
• Separate analyst-attributable outcomes from detection-threshold and data-quality effects.
• Identify factors behind missed defects and express them as a usable taxonomy.
• Turn findings into feedback loops: training content, guidance, tooling changes, threshold recommendations.
• Define and maintain measures of decision quality over time.

Skills

Python

SQL

Applied statistics

Machine learning

Ground truth evaluation

Communication

Git

CI/CD

Agile development

Problem solving

Self-direction

Leadership (potential)

Education

Bachelor's degree in statistics, CS, or engineering

Tools

Version control

Job description

Role Summary

Sperry's detection pipeline is part machine and part human. A neural network scans ultrasonic and electromagnetic test data collected from track and puts forward candidate defects. Trained analysts then review that output and decide what is real, what is not, and what gets sent to the railroad. Those decisions are the last judgment before a defect either reaches a customer or does not. Your job is to model that decision. Given what the analyst could see at the moment they made the call, was the disposition right, and where the pipeline gets it wrong, what actually caused it. We already capture the decision logs, so the data is there from your first week. This is a quality role rather than an automation role. The point is to make analysts better and to show us where our training, our tooling, and our detection thresholds are letting people down. The first phase is retrospective scoring of decisions already made. Where it goes after that depends in large part on what you find. You are one of the first three seats in a new US data science team, alongside a lead who owns risk analytics and an engineer who puts models into production. The work is internal-facing and it sits close to the operation, so you will spend real time with the people whose decisions you are modeling.

What We Expect From You

We expect an exceptional level of drive and ambition. You think beyond today's work to what the team and organization need next, champion bold ideas, and see them through. Your hunger is infectious - it inspires those around you to aim higher. You should be someone who puts the team first. You share credit openly, admit when you are wrong, and welcome feedback as an opportunity to grow. This role asks you to tell an organization things about its own performance that it may not want to hear, and that only works if people trust how you do it. This role requires a high degree of self-direction. You will manage complex work with minimal oversight, identify problems and solutions proactively, and may lead workstreams. You make well-reasoned technical decisions and escalates when there is genuine business impact. Strong analytical thinking is critical. You will work with imperfect labels, class imbalance, and outcomes that are only partly observable, and you need to be candid about what the data will and will not support. We would rather have a well-qualified answer than a confident one. You should be comfortable explaining a method to people who will not check your math but will act on your conclusion. Analysts, analysis managers, and operations leaders are your audience as much as other data scientists are.

Key Responsibilities

• Build and maintain retrospective scoring of analyst dispositions - given the data available at the time of review, how sound was the decision
• Quantify agreement and variation across analysts, shifts, territories, and test conditions
• Separate analyst-attributable outcomes from detection-threshold, data-quality, and volume effects, so that the organization acts on the right cause
• Identify the contributing factors behind missed and misinterpreted defects, and express them as a taxonomy the analysis organization can use rather than as individual scorecards
• Work with analysis leadership to turn findings into feedback loops: training content, review guidance, tooling changes, and threshold recommendations
• Define and maintain the measures of decision quality that hold up over time, and be clear about their limits
• Design sampling and review studies where the passive data cannot answer the question
• Work with the detection and platform teams so the signals your models need are captured properly at source
• Present findings to analysis leadership and to the wider engineering organization
• Write clean, tested, well-documented code following engineering best practices
• Participate in code reviews, sprint planning, and technical design discussions
• Write and maintain documentation so the methodology is transferable rather than held tacitly

Required Skills & Qualifications

• Strong proficiency in Python (NumPy, Pandas, Scikit-learn, or similar) and SQL
• Applied statistics: classification metrics, sampling, inter-rater agreement, bias, and experimental design
• Supervised machine learning on imbalanced and imperfectly labeled data
• Experience evaluating decisions or predictions against ground truth that arrives late, partially, or not at all
• Ability to explain method and uncertainty to an operational audience and have them act on it
• Familiarity with version control (Git), CI/CD pipelines, and agile development practices
• Strong problem-solving skills and ability to learn new technologies quickly
• Good communication skills – able to explain technical concepts to non-technical stakeholders
• A collaborative, team-first mindset aligned with our values of being Humble, Hungry, and Smart
• Qualifications and years of experience are indicative guidelines, not mandatory requirements. These criteria may be met through demonstrated competency or equivalent experience.

Desirable Skills

• Bachelor's degree in statistics, computer science, engineering, or a related quantitative field; advanced degree welcome
• Human-in-the-loop machine learning, expert review systems, or label quality and annotation quality work
• Model evaluation and monitoring tooling
• Human factors, quality management, or reliability engineering exposure
• Signal or sensor data, particularly ultrasonics, induction, or eddy current
• Experience in rail testing, NDT, or sensor-based inspection industries (ultrasound, eddy current, electromagnetic, etc.)