Principal AI Systems Engineer

Iovance Biotherapeutics, Inc.; United States; Remote; USD 175,000 - 200,000; Full time

Job summary

Iovance Biotherapeutics is seeking a senior AI/ML engineer to design, build, validate, and deploy production AI systems, including LLM-based apps and RAG pipelines, in a regulated life sciences environment.

You will collaborate with IT, Manufacturing, Quality, Regulatory Affairs, Clinical, and G&A to translate approved use cases into scalable AI capabilities while upholding validation, security, and governance standards.

Qualifications

• 10+ years of progressive software/AI/ML engineering experience, with the most recent 1+ years dedicated to LLM-based development.
• Deep, hands-on production experience with RAG including chunking, embeddings, vector stores, grounding, and MCP or equivalent integration patterns.
• Experience with multiple frontier model families (Claude, GPT, Gemini) and judicious selection based on task fit, cost, latency, safety, and data handling.
• Strong fundamentals: Python and at least one more language; Agile, Git, CI/CD, containerization; AWS (S3, Redshift, Bedrock).
• Hands-on, builder disposition; focus is coding, system design, model evaluation, and production artifacts.
• Knowledge of AWS data/AI services and enterprise search/retrieval (OpenSearch/Elasticsearch).

Responsibilities

• Design, build, and ship AI systems with end-to-end ownership including architecture, retrieval pipelines, prompts, evaluation, and deployment.
• Implement production-grade RAG systems on AWS infrastructure (S3, Redshift, Bedrock) with proper chunking, embeddings, and grounding.
• Develop and run evaluation harnesses with held-out sets, grounding metrics, and regression tests across model changes.
• Create integrations between AI systems and enterprise systems using MCP, APIs, and event-driven patterns with secure access.
• Own LLM security controls, including input/output guardrails, prompt injection defenses, data redaction, and content moderation.
• Develop, deploy, and maintain AI agents for regulated life sciences environments with traceability and oversight.
• Establish modern engineering practices: version control, CI/CD, environment separation, reproducible builds.
• Evaluate AI vendors/models; produce concise, evidence-based recommendations with PoC results when applicable.
• Implement technical controls for production AI: audit logs, access mgmt, model registries, performance monitoring, incident detection.
• Author technical docs: architecture/data flows/runbooks/validation deliverables; contribute to AI Validation Playbook.
• Mentor junior engineers and stay updated on AI tooling and best practices.

Skills

LLM app dev

RAG systems

MCP patterns

Python

AWS Bedrock

CI/CD

Git

Containerization

AI tooling

Architecture

Education

Bachelor’s or Master’s in CS/DS/Engineering

Tools

S3

Redshift

Bedrock

OpenSearch/Elasticsearch

Git

Job description

Iovance Biotherapeutics aims to be the global leader in innovating, developing and delivering tumor infiltrating lymphocyte (TIL) therapy for people with cancer. We are pioneering a transformational approach to treating cancer by harnessing the ability of the human immune system to recognize and attack diverse cancer cells in each patient. The Iovance TIL platform has demonstrated promising clinical data across multiple solid tumors. We are committed to continuous innovation in cell therapy, including gene-edited cell therapy, which may be a promising option for patients with cancer.

Overview

This is a senior role responsible for the hands-on design, build, validation, and deployment of Artificial Intelligence (AI) systems at Iovance Biotherapeutics. This role is focused on execution — turning approved use cases into working, validated, production AI systems that deliver measurable business value. This role directly supports Iovance’s strategy to improve overall operational productivity. The ideal candidate is a deeply technical, hands-on senior engineer with demonstrated production experience designing and shipping Large Language Model (LLM) applications, Retrieval-Augmented Generation (RAG) systems, and modern AI integrations. This position works closely with IT leads, business process owners across Manufacturing, Quality, Regulatory Affairs, Clinical, Commercial, G&A, IT Security, Privacy, and Quality Assurance to deliver AI capabilities that respect Iovance’s regulated environment. This role executes against a roadmap and prioritized backlog, while contributing technical input to refinement, scoping, and sequencing.

Primary Responsibilities

• Design, build, and ship AI systems against the approved Iovance AI roadmap, including end-to-end ownership of architecture, retrieval pipelines, prompts, evaluation, integrations, and deployment for assigned use cases.
• Implement production-grade Retrieval-Augmented Generation (RAG) systems on Iovance’s AWS infrastructure (S3, Redshift, Bedrock), including chunking strategies, embedding selection, vector storage, retrieval and reranking, grounding, and citation handling appropriate to high-accuracy use cases.
• Build, maintain, and run evaluation harnesses for AI systems, including held-out test sets, accuracy and grounding metrics, hallucination detection, adversarial inputs, and regression testing across model and prompt changes; treat evaluation as a first-class engineering deliverable, not an afterthought.
• Design and implement integrations between AI systems and Iovance enterprise systems using Model Context Protocol (MCP), APIs, and event-driven patterns, applying least-privilege access principles and partnering with IT Security on integration approval.
• Own and maintain LLM security controls for production AI systems, including input and output guardrails, prompt injection and jailbreak defenses, sensitive data redaction (PII, PHI, Iovance Confidential Information and Intellectual Property), content moderation, and abuse monitoring, working in partnership with IT Security.
• Design, develop, deploy, and maintain AI agents (multi-step reasoning systems that combine LLMs with tools, retrieval, and planning) appropriate for use in a regulated life sciences environment, including bounded scope, defined human oversight, traceability of agent decisions, and safe handling of write-back actions to systems of record
• Establish and uphold modern engineering practices for AI development including version control for code, prompts, and evaluation sets; CI/CD pipelines; environment separation (dev, test, production); and reproducible builds.
• Conduct hands-on technical evaluation of AI vendors, tools, and Foundation Models when build-vs-buy decisions are under consideration; produce concise, fact-based recommendations to the IT function lead and AI Governance Committee, including proof-of-concept results where appropriate.
• Implement and operate technical controls for production AI systems including audit logging, access management, prompt and model change control, model registry, ongoing performance monitoring, and incident detection, in alignment with Iovance policies.
• Author technical documentation appropriate to the system risk tier, including architecture diagrams, data flow diagrams, evaluation reports, runbooks, and validation deliverables; contribute to the Iovance AI Validation Playbook.
• Mentor junior engineers who collaborate on AI projects; stay current on rapid advances in AI tooling, models, and engineering best practices, and bring technical recommendations forward.

Qualifications

• 10+ years of progressive software and/or AI/ML engineering experience, with the most recent 1+ years dedicated primarily to LLM-based application development. Demonstrated track record of shipping production AI systems that real users depend on.
• Deep, hands-on production experience with Retrieval-Augmented Generation (RAG) including chunking strategies, embedding models, vector stores, retrieval and reranking, and grounding for high-accuracy use cases. Working production experience with Model Context Protocol (MCP) or equivalent agent/tool integration patterns.
• Practical working experience across multiple frontier model families (e.g., Anthropic Claude, OpenAI GPT, Google Gemini, leading open-weight models), with the judgment to select among them based on task fit, accuracy, cost, latency, safety properties, and data-handling commitments. Awareness of model capability changes and how to evaluate new model releases.
• Strong software engineering fundamentals: production Python (and ideally one other language); modern Agile delivery; version control (Git); CI/CD; containerization; cloud platforms (preferably AWS, with working familiarity with S3, Redshift, and Bedrock); MLOps tooling and practices.
• Hands-on, builder disposition; this role spends the majority of its time writing code, designing systems, evaluating models, and producing technical artifacts — not in meetings.
• Working production experience with AWS data and AI services is strongly preferred (S3, Redshift, Bedrock, IAM).
• Working knowledge of enterprise search and retrieval services such as OpenSearch, Elasticsearch, or equivalent, including the design of hybrid retrieval patterns (vector plus lexical/BM25) that improve grounding and relevance in production RAG systems

Preferred/Desirable Knowledge, Skills, and Education

• Relevant cloud and AI/ML certifications (AWS, Azure, etc.);
• Working knowledge of US regulatory requirements applicable to AI in regulated life sciences, including 21 CFR Part 11, GxP, HIPAA, and emerging FDA expectations on AI/ML in pharmaceutical manufacturing and drug development
• Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or related quantitative discipline.

Physical Demands and Activities Required

• Must be able to remain in a stationary position standing or sitting for prolonged periods of time.
• Must be able to move about inside an office and exert up to 10 pounds of force occasionally or a negligible amount of force frequently or constantly to lift, carry, push, pull, or otherwise move objects.
• Must have visual acuity to perform activities such as: preparing and analyzing data and figures, viewing a computer screen, and extensive reading.
• This position requires repetitive motion, substantial movements (motions) of the wrist, hands, and/or fingers.
• Must be able to communicate with others to exchange information.

Mental:

Clear and conceptual thinking ability; excellent judgment, troubleshooting, problem-solving, analysis, and discretion; ability to handle work‑related stress; ability to handle multiple priorities simultaneously; and ability to meet deadlines.

Work Environment

This job operates in a professional workplace or remote office environment and requires standard office equipment and keyboards. Employees who work remotely are expected to maintain their workspace and environment safely and free from safety hazards.

#LI-remote

The annual base salary we reasonably expect to pay is listed. Individual pay decisions depend on various factors, such as primary work location, complexity and responsibility of the role, job duties/requirements, and relevant education, experience and skills.

Pay Transparency

$175,000—$200,000 USD

The statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Management retains the discretion to add or to change the duties of the position at any time.

Iovance is committed to cultivating and offering a diverse and inclusive work environment. As an equal-opportunity employer, our employees and applicants will be considered without regard to an individual’s race, color, religion, sex, pregnancy, national origin, age, physical and mental disability, marital status, sexual orientation, gender identity, gender expression, genetic information, military and veteran status, and any other characteristic protected by applicable law. If you need assistance or accommodation to apply to one of our opportunities, please contact [e-mail].