Lead MLOps Engineer — Databricks + AI Platform

State Farm; Atlanta (GA); Hybrid; USD 100,000 - 200,000; Full time

Benefits offered by this job

Annual raise

Health benefits

401(k) plan

Education benefits

Flexible time off

Job summary

State Farm’s Applied Innovation Lab seeks a lead contributor to own data foundations and governance for Databricks projects. You will drive medallion architecture, cost-aware execution, and ML lifecycle from experimentation to production, partnering across enterprise teams.

You’ll shape patterns, enable self-healing runbooks, and push forward agentic AI integrations while mentoring engineers in a fast-moving, experimental environment.

Qualifications

• Databricks in production at application layer.
• Production ML experience including RCA and drift detection.
• GenAI and agentic tools are part of daily work.
• MLflow 3.0 lifecycle: registry, experiments, deployment, retraining.
• FinOps and cost tracking at project level.
• Data lineage & observability with automatic ticketing.
• Production handoff: runbooks, governance and SLAs.
• Systems thinking across the full ML/data stack.
• Python, PySpark, SQL; CI/CD for data/ML pipelines.

Responsibilities

• Own data foundations: medallion architecture Bronze→Silver→Gold.
• Improve Spark performance: query bottlenecks and cluster sizing.
• Manage Unity Catalog, access controls, and lineage.
• Lead ML lifecycle: model from experiment to production.
• Ensure cost visibility and FinOps for every project.
• Define handoff criteria: runbooks, self-healing patterns.
• Develop templates and standards for future projects.
• Integrate agentic AI into data validation and CI/CD.
• Provide technical direction and mentor engineers.

Job description

State Farm’s Applied Innovation Lab seeks a lead contributor to own data foundations and governance for Databricks projects. You will drive medallion architecture, cost-aware execution, and ML lifecycle from experimentation to production, partnering across enterprise teams.

You’ll shape patterns, enable self-healing runbooks, and push forward agentic AI integrations while mentoring engineers in a fast-moving, experimental environment.