Lead Data Engineer - AI/Machine Learning

corespecialtyinsurance; Cincinnati (OH); On-site; USD 150,000 - 210,000; Full time

Job summary

corespecialtyinsurance seeks a Lead AI Engineer to shape AI/ML enablement across the organization. You will start hands-on, contributing to our data platform and pipelines while progressively taking a leadership role in governance, deployment, and best practices.

You will partner with Data Governance, Data Engineering, and Product stakeholders to define AI/ML frameworks and MLOps strategy, ensuring scalable, responsible AI adoption and high‑quality engineering outcomes across teams.

Qualifications

• Strong data engineering fundamentals with pipelines and platform design.
• Experience deploying AI/ML solutions with proper governance and security.
• Proficiency in Python and modern data tooling (Spark, Kafka).
• Hands-on in ML/AI workflows and MLOps best practices.

Responsibilities

• Design, build, and optimize data pipelines and platform components.
• Own complex engineering initiatives from design to rollout with minimal oversight.
• Identify and resolve performance, scalability, and reliability issues.
• Define AI/ML frameworks and MLOps strategies with governance alignment.
• Write clean, tested code and infrastructure-as-code.

Skills

Python

Data Engineering

LLM APIs

MLOps

Cloud Platforms

Data Pipelines

Venture Tools

Observability

Tools

Snowflake

Databricks

Azure Synapse

Airflow

Kafka

dbt

Spark

Pinecone / Weaviate

Job description

We are looking for a Lead AI Engineer to help shape and drive AI/ML enablement and readiness across the organization. This role requires strong data engineering fundamentals: you will start hands‑on, contributing directly to our data platform and pipelines, while progressively taking on a leading role in defining how the organization builds, deploys, and governs AI/ML capabilities.

Reporting directly to the VP, Head of Data, you will work autonomously to identify gaps, propose solutions, and bring innovative thinking to how our data and AI/ML ecosystem should evolve. You will partner closely with Data Governance, Data Engineering, and Product stakeholders to define our AI/ML frameworks and MLOps strategy, and to ensure the organization is well‑positioned to adopt AI/ML responsibly and at scale.

Key Accountabilities/Deliverables:

• Design, build, and optimize data pipelines, ingestion frameworks, and platform components that support analytics, reporting, and AI/ML use cases.
• Take direct, autonomous ownership of complex engineering initiatives, from technical design through implementation and rollout, with minimal need for oversight.
• Identify and resolve performance, scalability, and reliability issues across the existing data platform.
• Bring innovative, well‑reasoned solutions to data engineering problems, proactively identifying gaps and proposing improvements rather than waiting for direction.
• Write clean, well‑tested, well‑documented code and infrastructure‑as‑code, maintaining strong engineering hygiene across your work.
• Help define the organization's AI/ML frameworks, evaluating and recommending tools, platforms, and standards for building and deploying AI/ML solutions.
• Build working prototypes that provide immediate value to the engineering teams
• Shape and help implement our MLOps strategy, including approaches to model deployment, monitoring, versioning, and lifecycle management
• Partner in deep, ongoing collaboration with Data Governance to ensure AI/ML frameworks and practices align with data governance, security, and compliance standards.
• Design and advocate for data infrastructure patterns that support AI/ML use cases at scale (e.g., feature stores, curated/governed datasets, streaming access for training and inference).
• Partner with Data Science, Data Engineering, and business stakeholders to assess AI/ML readiness gaps and build a roadmap to close them.
• Act as a subject‑matter expert and thought partner to the VP, Head of Data on emerging AI/ML technologies, practices, and industry trends.
• Document AI/ML standards, frameworks, and decisions to support consistent adoption across the organization as the practice matures.
• Act as a senior technical resource for the team, providing guidance on architecture, design patterns, and best practices AI/ML readiness and ML Ops frameworks
• Partner closely with Enterprise Architecture on establishing architectural blueprints for AI readiness

Other Duties as Assigned.

Technical Knowledge and Understanding:

Data Engineering

Strong data engineering fundamentals: deep expertise in data pipeline design, optimization, and distributed data processing (e.g., Spark, dbt, Airflow, Kafka, or equivalent).

Platforms: hands‑on experience with Snowflake, Databricks, and/or Azure Synapse Analytics, with the ability to architect and optimize workloads on one or more of these platforms.

Strong knowledge of cloud platforms (AWS, Azure, or GCP) and modern data warehouse/lakehouse architectures.

AI/ML Engineering

Programming & software engineering fundamentalsStrong Python (the de facto language for AI/ML tooling); solid software engineering practices (testing, version control, code review) since AI engineers ship production systems, not just notebooks

API design and integration - most AI engineering work today is building systems around models (orchestration, tool‑calling, retrieval), not training them from scratch

LLM & foundation model fluencyPractical experience with LLM APIs (ie. OpenAI) and open‑weight models

Prompt engineering and prompt evaluation as a discipline, not just trial‑and‑error

Understanding of context windows, tokenization, embeddings, and model limitations (hallucination, latency, cost tradeoffs)

RAG (Retrieval‑Augmented Generation) & data retrievalVector databases (Pinecone, Weaviate, pgvector, etc.) and embedding models

Chunking strategies, hybrid search, reranking

Agentic systems & orchestrationFrameworks like LangChain, LangGraph, LlamaIndex, or custom orchestration

Tool‑use / function‑calling design, multi‑step reasoning chains, agent memory and state management

Fine‑tuning & model adaptationWhen to fine‑tune vs. prompt vs. RAG

Familiarity with parameter‑efficient methods (LoRA, etc.) MLOps / LLMOps

Model evaluation frameworks, A/B testing for model outputs, observability (tracing, logging model calls)

Deployment patterns: latency/cost optimization, caching, streaming responses, fallback handling

Versioning prompts a