Senior AI Engineer

Florence Healthcare; United States; On-site; USD 110,000 - 150,000; Full time

Benefits offered by this job

Medical and dental insurance

Competitive compensation package

Office space in the heart of the city

Opportunity to work on health technology

Job summary

Florence Healthcare is seeking a Senior AI Engineer to design, implement, and deploy AI-powered features, focusing on reliability and measurable impact. The role involves managing the entire lifecycle of AI systems, experimenting with AI models, and collaborating with product teams.

The ideal candidate will have 4+ years of experience, strong knowledge of LLMs, and the ability to debug and optimize complex systems. This position offers a competitive compensation package and an opportunity to work on impactful health technology.

Qualifications

• 4+ years of related work experience in AI engineering.
• Strong backend/system design fundamentals.
• Ability to debug complex issues and optimize outputs.

Responsibilities

• Design and implement AI-powered features and workflows.
• Own the full lifecycle of AI systems from prototype to production.
• Build and optimize retrieval systems and pipelines.

Skills

Understanding of LLM capabilities

Experience with prompt engineering

Hands-on experience with embeddings

Familiarity with RAG architectures

Designing agent-based architectures

Debugging complex issues

Education

Bachelor's degree in Computer Science or related field

Tools

AI modeling software

API tools

Job description

What We Do

Florence software advances cures by helping the world’s most important research sites do their best work. Our solutions are now used by over 30,000 research teams in 70 countries around the world—we’re the most widely deployed site workflow tool in the industry. By the end of the decade, we’ll double the pace at which new medicines get to market by doubling the output of trial site teams. To date, we were named a Deloitte Fast 50 business, G2 Category Leader, an Inc. & AJC best place to work, and an Inc. 5000 company five years in a row. At Florence, we are committed to make the world a better place by accelerating research while providing an environment for our employees where they can be happy in their lives, enjoy their jobs, and grow.

What You'll Bring to The Team

We are looking for a Senior AI Engineer to design, build, and deploy high‑quality AI‑powered features. This role focuses on owning end‑to‑end implementation of AI systems within a product area — from prototyping to production — with a strong emphasis on reliability, iteration, and measurable impact.

You Will

• End-to-end AI Feature Ownership
  • Design and implement AI‑powered features (LLM workflows, copilots, and agent‑based systems with tool use and multi‑step reasoning)
  • Own the full lifecycle: prototyping → evaluation → production deployment → iteration
  • Ensure solutions are reliable, performant, and aligned with product needs
• AI System Implementation
  • Build and optimize:
    • Prompt pipelines for specific use cases
    • Retrieval systems (embeddings, chunking, ranking)
    • RAG‑based workflows where needed
  • Iterate on outputs to improve quality, accuracy, and consistency
  • Design scalable and cost‑efficient AI architectures for production workloads
  • Select and evaluate models (hosted vs open‑source) based on use case constraints
  • Agent‑Based Systems (AgentCore)
    • Design and build agentic workflows capable of multi‑step reasoning and decision‑making
    • Integrate agents with tools, APIs, and internal systems to perform real‑world actions
    • Implement planning, execution, and reflection loops for complex tasks
    • Manage context, memory, and state across multi‑step interactions
    • Balance deterministic workflows vs. agent autonomy for reliability and control
• Experimentation & Evaluation
  • Run structured experiments to compare approaches (prompting, retrieval, models)
  • Define and track key metrics for AI performance (quality, latency, cost)
  • Debug and improve non‑deterministic system behavior
  • Build and maintain evaluation datasets and benchmarks
  • Implement automated evaluation pipelines for continuous improvement
• Collaboration & Contribution
  • Drive technical direction and influence AI adoption across teams
  • Partner with product managers and designers to scope AI features
  • Contribute to shared patterns and reusable components
  • Participate in code reviews and design discussions
  • Support and mentor mid‑level engineers where needed
• AI Reliability, Safety & Governance
  • Design guardrails to ensure safe and reliable AI behavior
  • Mitigate hallucinations, prompt injection, and model misuse
  • Ensure compliance with data privacy and enterprise requirements
  • Implement monitoring and observability for AI systems in production
  • Implement guardrails for agent actions (tool access control, execution boundaries)
  • Prevent failure cascades in multi‑step agent workflows

An Ideal Candidate Has

• Core AI Skills
  • Strong understanding of LLM capabilities and limitations
  • Experience with prompt engineering and structured output design
  • Hands‑on experience with embeddings and vector search
  • Familiarity with RAG architectures and when to apply them
  • Experience designing agent‑based architectures (AgentCore concepts)
  • Understanding of tool use, planning strategies, and memory mechanisms in LLM systems
• Engineering Skills
  • 4+ years of related work experience
  • Solid backend/system design fundamentals
  • Evaluate agent performance across multi‑step tasks (task success rate, error propagation)
  • Debug and optimize agent decision‑making and tool selection behavior
  • Experience building and deploying production‑grade systems
  • Ability to debug complex issues, including probabilistic outputs
  • Comfort working with APIs, pipelines, and data flows
• Product Thinking
  • Ability to translate user needs into effective AI solutions
  • Strong intuition for balancing quality, latency, and cost
  • Focus on delivering measurable product impact
• Collaboration
  • Communicates clearly across engineering and product teams
  • Contributes to team knowledge and shared practices

What Success Looks Like

• Ships high‑quality AI features that deliver clear user value
• Demonstrates strong ownership from idea to production
• Improves systems through structured iteration and experimentation
• Contributes to team‑level best practices and reusable solutions

What's in it for you?

• Competitive compensation package, medical and dental insurance, and office space in the heart of the city
• Opportunity to work on health technology, advancing cures and therapies
• High‑output culture to create innovative, modern, and purposeful software solutions

Florence supports workplace diversity and does not discriminate on the basis of race, color, religion, gender identity or expression, national origin, age, military service eligibility, veteran status, sexual orientation, marital status, physical disability, or any other protected class.