Data Scientist III

BMC Software; Los Angeles (CA); Hybrid; USD 132,000 - 220,000; Full time

Job summary

BMC Software in Los Angeles, CA is seeking a Data Scientist III to advance enterprise AI capabilities. This hybrid role emphasizes hands-on ML, statistical modeling, and Generative AI with a focus on reproducible workflows and MLOps/LLMOps.

You will design, evaluate, and deploy AI/ML solutions, collaborate with engineering for production readiness, and mentor junior data scientists while applying RAG concepts and advanced model techniques to enterprise use cases.

Qualifications

• 5-9 years of Data Science, ML, or Applied AI experience.
• Strong Python, Pandas, NumPy, scikit-learn, PyTorch or TensorFlow; SQL and data prep skills.
• Solid understanding of ML/AI fundamentals and model evaluation.

Responsibilities

• Design, develop, evaluate, and improve AI/ML solutions for enterprise challenges.
• Apply MLOps/LLMOps practices to improve model quality and cost efficiency.
• Mentor junior Data Scientists and collaborate with engineering for production readiness.
• Analyze latency, token usage, inference costs, and retrieval effectiveness.
• Develop and evaluate document chunking, embedding, and retrieval strategies.

Skills

Python

Pandas

NumPy

scikit-learn

PyTorch/TensorFlow

SQL

MLOps/LLMOps

LLM prompting

RAG concepts

Education

Bachelor’s or Master’s degree in a related field

Tools

OpenAI tools

FastAPI/Flask

OpenSearch/Elasticsearch

Job description

BMC Software’s AMI team is hiring a Data Scientist III to help build and improve enterprise AI capabilities. This hybrid role in Los Angeles, CA focuses on hands‑on machine learning, statistical modeling, and Generative AI, including RAG evaluation and experimentation, with an emphasis on reproducible workflows and MLOps/LLMOps.

The work blends model development with rigorous measurement: designing evaluation datasets and golden test cases, benchmarking models (including open‑weight options), and partnering with engineering to support production readiness and continuous quality.

What you’ll do

• Design, develop, evaluate, and improve AI/ML solutions to address enterprise business and technical challenges.
• Use statistical rigor, reproducible experimentation, and MLOps/LLMOps practices to improve model quality, RAG effectiveness, and overall AI application performance.
• Own defined data science initiatives and lead moderately complex or ambiguous modeling and evaluation tasks.
• Collaborate with engineering teams to support production readiness, packaging, deployment handoffs, monitoring, and lifecycle management.
• Mentor less experienced Data Scientists.
• Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases.
• Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques using traditional ML, embeddings, and LLM-based approaches.
• Analyze execution latency, token usage, inference costs, workload trends, and resource requirements using statistical modeling.
• Use clustering and anomaly detection to identify recurring intents, unusual agent behavior, capability gaps, and execution regressions.
• Perform exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experimentation.
• Evaluate commercial and self-hosted open-weight models for accuracy, instruction adherence, structured outputs, latency, and cost.
• Create and test prompt templates, few‑shot examples, tool‑calling prompts, and context assembly approaches.
• Compare model alternatives and document quality, performance, and cost tradeoffs.
• Collaborate with Senior Data Scientists and AI Engineering on model performance analysis and inference optimization.
• Develop and evaluate document chunking, metadata, embedding, retrieval, reranking, and context selection strategies.
• Compare vector, keyword/BM25, hybrid, and semantic retrieval approaches.
• Measure retrieval effectiveness using Recall@K, Precision@K, nDCG, groundedness, faithfulness, and answer quality.
• Investigate retrieval and generation failures, then recommend improvements based on evaluation results.
• Benchmark embedding and reranking models using representative enterprise datasets.
• Support evaluation of prompting, RAG, supervised instruction fine‑tuning, and parameter‑efficient adaptation approaches.
• Prepare and validate instruction datasets, training examples, and holdout evaluation datasets, including participation in SFT, LoRA, QLoRA, or comparable fine‑tuning experiments under appropriate technical guidance.
• Compare adapted models against baseline models for accuracy, generalization, regressions, and operational cost.
• Collaborate on validation and production handoff of fine‑tuned model artifacts.
• Apply modeling and evaluation techniques to intent classification, agent/skill discovery, tool selection, and routing effectiveness.
• Evaluate task completion, tool‑call correctness, parameter extraction, delegation quality, and recovery behavior.
• Understand agentic AI concepts including planning, tool calling, MCP tools, A2A Agent Cards, skills, and task delegation.
• Contribute model‑based decision signals and evaluation approaches related to agentic workflows, without owning agent runtime, MCP server, or A2A protocol implementation.
• Develop evaluation datasets, golden test cases, holdout sets, and automated evaluation workflows.
• Track accuracy, relevance, groundedness, instruction following, task completion, latency, and cost; investigate evaluation failures using experimental evidence.
• Support automated regression testing and production readiness validation, including experiment tracking and versioning across datasets, code, model configurations, prompts, retrieval settings, metrics, and artifacts.
• Develop and maintain reproducible model training, benchmarking, and evaluation workflows, including model registries and automated evaluation pipelines.
• Contribute to model quality gates, regression testing, and release‑readiness checks, including analysis of performance, quality drift, routing accuracy, latency, and inference cost signals.

What you’ll bring

• 5-9 years of relevant Data Science, Machine Learning, or Applied AI experience.
• Bachelor’s degree in Machine Learning, Statistics, Data Science, Physics, Computer Science, or a related field, or equivalent education and industry experience.
• Strong Python proficiency with Pandas, NumPy, scikit‑learn, and PyTorch or TensorFlow; SQL and practical data preparation skills.
• Solid understanding of statistical analysis, experimental design, hypothesis testing, ML validation, classification, regression, ranking, clustering, forecasting, and anomaly detection.
• Practical experience with LLM prompting, embeddings, model comparison, and RAG concepts, including chunking, retrieval, and reranking.
• Familiarity with instruction for fine‑tuning and parameter‑efficient adaptation or model customization techniques.
• Experience developing evaluation datasets, automated quality metrics, and reproducible ML/LLM experiments.
• Hands‑on experience with MLOps/LLMOps, including experiment tracking, dataset/model/prompt versioning, reproducible workflows, evaluation automation, and monitoring.
• Working knowledge of agentic AI concepts including planning, routing, tool calling, MCP, A2A, and multi‑agent patterns.
• Understanding of model latency, throughput, token usage, inference cost, and model performance tradeoffs.
• Familiarity with AWS and/or Kubernetes or Red Hat OpenShift (OCP), plus collaboration with engineering teams.
• Clear communication, problem‑solving, cross‑functional collaboration, and ability to manage multiple priorities.
• Demonstrated ability to deliver moderately complex AI/ML initiatives, make sound technical decisions, and mentor junior Data Scientists.
• Experience contributing reusable modeling, evaluation, or MLOps/LLMOps practices within project teams.

Tools and technologies

• Python, Pandas, NumPy, scikit‑learn, PyTorch, TensorFlow, SQL
• MLOps, LLMOps, MCP, A2A, AWS, Kubernetes, Red Hat OpenShift (OCP)
• BM25, Recall@K, Precision@K, nDCG, LoRA, QLoRA, SFT
• FastAPI, Flask, OpenTelemetry, OpenSearch, Elasticsearch, FAISS, Milvus, pgvector
• LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, MLflow, Kubeflow
• Ragas, DeepEval, OpenJudge, LangSmith
• vLLM, NVIDIA Triton, quantization, GPU memory

Compensation and benefits

• Annual base salary range $131,700 - $219,500 (actual salaries depend on skill sets; experience and training; licensure/certifications; and other business/organizational needs).
• Other rewards may include a variable plan and country specific benefits.

Additional job details

• Location: Los Angeles, CA, United States (hybrid). Travel may be required occasionally for business.
• Remote: This role can be based remotely in United States, and is also listed as #LI-Hybrid and #LI-Remote.
• Additional locations: San Francisco - Los Angeles - San Diego - Sacramento - Anaheim - San Francisco.
• Job ID: 47815; Published 05-Nov-2026.
• Job name: Data Scientist III - USA (B).

Nice-to-have skills

• Experience with OpenSearch, Elasticsearch, FAISS, Milvus, pgvector, or similar retrieval platforms.
• Exposure to LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, or comparable agentic frameworks.
• Familiarity with MLflow, Kubeflow, Ragas, DeepEval, OpenJudge, LangSmith, or similar MLOps/LLMOps and evaluation tools.
• Exposure to instruction fine‑tuning, SFT, LoRA, QLoRA, or domain‑specific model adaptation.
• Familiarity with vLLM, NVIDIA Triton, quantization, GPU memory considerations, or inference optimization.
• Python backend APIs (FastAPI, Flask, or similar); OpenTelemetry and AI observability.
• AI safety, prompt injection evaluation, privacy, responsible AI, and model governance.
• Time‑series forecasting, causal inference, or anomaly detection in enterprise systems data.
• Agile practices and Atlassian Jira/Confluence.
• Master’s degree in a related field.