Work with Microsoft’s Engineering Thrive Insights team to build measurement frameworks and AI data and experimentation systems that improve engineering outcomes at scale.
Responsibilities
- Apply data science and business understanding to review technical projects and assess factors influencing final outcomes.
- Evaluate project plans across resources, risks, contingencies, requirements, assumptions, and constraints.
- Document key business objectives and translate goals into analytical and technical terms.
- Communicate insights to stakeholders consistently.
- Collaborate with immediate team members to define metrics and frameworks for evaluating AI/LLM systems and agentic product performance.
- Experiment with models prior to selection (for example, LLM and prompting approaches) for product or solution tasks.
- Diagnose basic issues and identify improvement opportunities across AI/LLM systems within the domain area.
- Prototype and develop AI models in containerized environments (including AI Azure Foundry and LangChain) to operationalize, build, and validate scalable tools and applications.
- Develop and deploy retrieval-augmented generation (RAG) workflows, including selection of data sources, retrieval systems, and generative models to support output quality.
- Support senior team members in assessing and refining LLM systems and models.
- Determine where to acquire data required to complete the project plan.
- Use querying, visualization, and reporting to describe acquired data (format, quantity, identities, and other surface properties).
- Explore data attributes and contribute to data quality reporting using AI tools (for example, Codex and GitHub Copilot), including task results, initial findings, and project impact.
- Collaborate on data-science experiments using established methodologies, including statistics, optimization, and probability theory for general-purpose software and statistical packages.
- Assess multiple tools and techniques and select appropriate options.
- Partner with Solution Architects, Consultants, and Data Engineers on data preparation efforts.
- Follow Microsoft privacy policy requirements for collecting data, preparing data, and using AI tools (including Codex and GitHub Copilot).
- Identify data integrity problems.
- Use knowledge of modeling and machine learning approaches (for example, state-of-the-art statistical analysis, causal modeling, reinforcement learning) to choose the best approach for objectives.
- Select data preparation, model training and optimization, and evaluation methods based on validation, hypothesis testing, and proper use of data, including statistical and business significance.
- Account for risks such as data leakage, bias-variance tradeoff, and methodological limitations.
- Write required scripts using appropriate languages such as T-SQL, U-SQL, KQL, Python, and R.
- Construct hypotheses, design controlled experiments, analyze results using statistical tests, and communicate findings to business stakeholders.
- Communicate with diverse audiences on data-quality issues and initiatives.
- Understand deployment operational considerations including performance, scalability, monitoring, maintenance, integration into engineering production systems, and stability.
- Develop operational models that run at scale in partnership with data engineering teams.
- Keep up to date on embedding models and vector databases, and understand current embedding/vector database practices.
- Stay current on discipline trends, including tools, techniques, strategies, and processes that can improve efficiency and performance.
- Attend internal research conferences and participate in on-hands training, when appropriate.
- Contribute to thought leadership and IP best practices.
Requirements
- Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field.
- OR Master’s Degree in the same fields/areas AND 1+ year(s) of data-science experience (for example, managing structured and unstructured data, applying statistical techniques, and reporting results) or consulting experience.
- OR Bachelor’s Degree in the same fields/areas AND 2+ years of data-science experience (for example, managing structured and unstructured data, applying statistical techniques, and reporting results).
- OR equivalent experience.
- Ability to meet Microsoft, customer and/or government security screening requirements.
- Microsoft Cloud Background Check: pass upon hire/transfer and every two years thereafter.
Technologies
- AI Azure Foundry, LangChain
- T-SQL, U-SQL, KQL (Kusto Query Language), Python, R
- Codex, GitHub Copilot
- Retrieval-augmented generation, Spark-based big data platform technologies such as Databricks
- Vector databases, embedding models
- DORA, SPACE
- AI coding tools, LLM, reinforcement learning
- Regression modeling, hypothesis testing
Preferred Qualifications
- Doctorate in the listed fields AND 1+ year(s) data-science experience OR Master’s Degree AND 3+ years data-science experience (as described in the application text).
- Experience with observational causal inference methods, design of controlled experiments, regression modeling, and hypothesis testing.
- Experience evaluating LLMs and/or AI agents, including designing offline/online evaluations, defining quality metrics, and measuring model behavior.
- Experience designing surveys and analyzing response data.
- Proficiency in Python.
- Proficiency in analytical query languages such as SQL, KQL (Kusto Query Language), or similar.
- Experience developing analytics on Spark-based big data platform technologies such as Databricks.
- Data storytelling, including crafting and presenting data stories to business decision-makers.
- Familiarity with developer productivity metrics and frameworks (for example DORA, SPACE, or similar).
- Demonstrated effective use of AI coding assistants in the daily data science development workflow.
Location: Redmond, WA (onsite)
Salary: USD 102,100 - 219,200 per year
Minimum experience: 1 year