Principal Quality Engineer & AI Quality Archi

5154a6eb-14d1-4663-b31c-c91a4fc4a51a; Elmwood Park (NJ); Hybrid; USD 110,000 - 115,000; Full time

Benefits offered by this job

Medical, Dental, Vision

Employer‑paid Life and LTD

401k w/ matching

Remote work options

Flexible working hours

Generous FTO

Volunteer program

Lunches onsite

Job summary

ELLKAY is seeking a Principal Quality Engineer and AI Quality Architect to define the technical vision and architecture for quality across software, data, and AI‑enabled products.

You will design scalable automation and evaluation platforms, govern AI‑specific quality controls, and guide delivery teams through high‑risk technical and regulatory decisions.

Qualifications

• Bachelor’s or Master’s in CS/Engineering/Data Science/AI or related field.
• 12+ years in software or quality engineering with enterprise architecture.
• Expert coding skills in Python, Java, TypeScript, or C#, with cloud/CI‑CD experience.
• Deep knowledge of test architecture, reliability, security, and data quality.
• Hands‑on ML/AI evaluation, RAG, embeddings, model monitoring, and responsible‑AI risks.
• Able to define standards while remaining hands‑on with prototypes and reviews.

Responsibilities

• Own enterprise quality engineering and AI assurance architecture with roadmaps.
• Architect modular automation platforms for UI, API, data, and production validation.
• Define AI evaluation frameworks for correctness, grounding, and bias.
• Embed automated quality controls into CI/CD, MLOps and LLMOps pipelines.
• Lead architecture reviews and partner with security, legal, and product teams.

Skills

Senior architect

AI quality

API design

Cloud & CI/CD

Observability

Prototyping & prototyping

Education

Bachelor’s or Master’s in CS/Engineering/Data Science/AI

Tools

Python

Java

TypeScript

C#

Job description

ELLKAY started out providing connectivity solutions to laboratories and within a few years, grew to also provide data management solutions to ambulatory organizations. ELLKAY is now a trusted data management partner in five healthcare segments. ELLKAY’s solutions continue to serve laboratories and ambulatory practices and have expanded to empower hospitals and health systems, healthcare IT vendors, ambulatory practices, health plans, and other healthcare organizations with cutting‑edge technologies and solutions that drive their growth and interoperability strategies. Today, ELLKAY remains true to our core values, building strong partner relationships and offering unparalleled service and support while providing innovative, scalable solutions to the challenges our customers face in today’s data‑rich world. ELLKAY's experience, customer‑focused approach, and reputation for innovation, speed, and accuracy differentiate ELLKAY as a premier partner for your interoperability needs and data management strategy.

Job Description:

The Principal Quality Engineer and AI Quality Architect defines the technical vision, reference architecture, and engineering standards for quality across complex software, data, and AI‑enabled products. This hands‑on principal role architects scalable automation and evaluation platforms, governs AI‑specific quality controls, and guides delivery teams through the highest‑risk technical and regulatory decisions.

Key responsibilities

• Own the enterprise quality engineering and AI assurance architecture, including principles, target‑state capabilities, reference patterns, roadmaps, and measurable adoption standards.
• Architect modular automation platforms spanning UI, API, contract, integration, event, data, performance, resilience, security, accessibility, and production validation.
• Define quality strategies for machine learning, generative AI, agentic workflows, retrieval‑augmented generation, and predictive models across training, evaluation, deployment, and monitoring.
• Establish AI evaluation frameworks for task correctness, grounding, relevance, hallucination, robustness, bias and fairness, explainability, privacy, toxicity, safety, latency, and cost.
• Design representative golden datasets, adversarial suites, synthetic test data, human‑review protocols, statistical thresholds, and regression baselines for probabilistic systems.
• Validate RAG retrieval quality, prompt and model changes, tool use, agent boundaries, fallback behavior, and defenses against prompt injection, data leakage, and unsafe actions.
• Define model and prompt versioning, evaluation gates, lineage, audit evidence, drift detection, observability, rollback criteria, and incident‑response patterns.
• Embed automated quality controls into CI/CD, MLOps and LLMOps pipelines, cloud platforms, and developer workflows with fast feedback and policy‑based release gates.
• Set standards for testability, service virtualization, contract testing, environment strategy, test‑data management, telemetry, and quality metrics across engineering portfolios.
• Lead architecture reviews and proofs of concept; select tools through evidence‑based evaluation of scalability, interoperability, security, maintainability, and total cost.
• Partner with security, privacy, legal, compliance, clinical, product, data science, and platform leaders to translate regulatory and responsible‑AI obligations into verifiable controls.
• Coach senior engineers and architects, resolve cross‑team technical constraints, and influence investment and platform decisions through clear tradeoff analysis.

Required qualifications

• Bachelor's or master's degree in computer science, engineering, data science, AI, or a related discipline, or equivalent experience.
• Typically 12+ years in software engineering or quality engineering, including enterprise architecture responsibility and recent leadership in AI or ML quality.
• Expert‑level coding and framework design skills in Python, Java, TypeScript, C#, or comparable languages, with strong API, distributed‑system, cloud, CI/CD, and observability expertise.
• Deep knowledge of test architecture, reliability engineering, security and privacy testing, data quality, performance engineering, and risk‑based release governance.
• Hands‑on understanding of ML and generative‑AI evaluation, RAG systems, embeddings or vector search, model monitoring, probabilistic behavior, and responsible‑AI risks.
• Ability to define enterprise standards while remaining hands‑on with prototypes, reference implementations, design reviews, and complex technical investigations.

Preferred qualifications

• Experience architecting quality for regulated healthcare platforms and protecting PHI in AI‑enabled workflows.
• Familiarity with NIST AI RMF, model risk management, healthcare interoperability, and emerging AI governance or regulatory expectations.

Benefits:

ELLKAY offers a comprehensive and competitive benefit package that starts day one!

Including:

• Medical, Dental, and Vision benefits
• Employer‑paid Life and LTD
• 401k w/ matching – once eligibility is met
• Work/life balance
• Paid Volunteer Program
• Flexible working hours
• Generous FTO
• Remote work options
• Employee Discounts
• Parental Leave

Our awesome culture includes:

• Working with talented, collaborative, and friendly people who love what they do
• Professional growth within
• Innovation environment
• On site in HQ Free daily lunches

Additional information

At ELLKAY, we are committed to operating as a hybrid workplace, offering employees flexibility in how they structure their time between in‑office and remote work. We recognize the significance of fostering connections, collaboration, and creativity within our office culture and its positive impact on our business. Our philosophy of operating as a hybrid workplace underscores our dedication to enabling employees to tailor work‑life balance to their individual preferences. For those who do not live within 40 miles of one of our offices, we are open to considering remote work for candidates whose skills and experience strongly align with the role. While we prioritize a hybrid work environment for most roles, we understand the importance of flexibility and are open to remote work for specific positions and specialized skill sets.

For more information on our company, visit www.ELLKAY.com.

ELLKAY LLC is a Smoke‑Free Workplace.

ELLKAY, LLC provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

Compensation Range: $110,000 - $115,000