Artificial Intelligence Engineer

Quadrant IQ Solutions LLC; Texas; On-site; USD 120,000 - 190,000; Full time

Job summary

Quadrant IQ Solutions LLC is seeking an AI Engineer to design, test, optimize, and support AI-powered agent experiences across business workflows. This role combines prompt engineering, agent orchestration, QA testing, evaluation framework development, and AI observability.

You will design prompts and systems, conduct experiments, and collaborate with engineering to deploy updates through established SDLC processes. The ideal candidate has strong AI, testing, and collaboration skills.

Qualifications

• Bachelor's degree in CS, engineering, information systems, or related field.
• 5–10 years in software QA, automation testing, or related technical roles.
• 1+ years hands-on with Generative AI, LLMs, or conversational AI.
• Experience with Azure DevOps or similar Agile delivery platforms.
• Experience designing and executing structured test plans.
• Strong analytical and troubleshooting skills.
• Excellent written and verbal communication skills.

Responsibilities

• Design, test, optimize, and support AI-powered agent experiences.
• Lead prompt engineering and agent design across workflows.
• Develop evaluation frameworks and scoring for AI output quality.
• Create test cases for accuracy, safety, tone, instruction following, and consistency.
• Collaborate with engineering to deploy updates via SDLC and Azure DevOps.
• Maintain documentation for prompts, evaluation methodologies, and agent configurations.
• Participate in Agile ceremonies and communicate findings to stakeholders.
• Monitor AI observability platforms and identify improvement opportunities.

Skills

Generative AI

Prompt engineering

API testing

Azure DevOps

QA engineering

Analytical thinking

Education

Bachelor's degree

Tools

Arize AI

OpenAI

Azure OpenAI

Job description

Position Summary

We are seeking an AI Engineer to help design, test, optimize, and support AI-powered agent experiences across customer and internal business workflows. This role combines prompt engineering, agent orchestration, quality assurance, evaluation framework development, and AI observability.

Key Responsibilities

Prompt Engineering & Agent Design

• Design, develop, and iterate on AI agent logic for business use cases.
• Create, test, and optimize prompts, system instructions, tools, and workflows to improve agent performance.
• Conduct prompt experiments and A/B testing to measure effectiveness.
• Analyze agent responses and refine prompting strategies for accuracy, consistency, groundedness, and user satisfaction.
• Document prompt patterns, reasoning frameworks, reusable templates, and agent design decisions.
• Collaborate with engineering teams to deploy prompt and agent updates through established SDLC processes.

AI Quality Assurance & Testing

• Design and execute comprehensive test plans covering AI agents, web applications, APIs, and backend services.
• Develop evaluation frameworks and scoring methodologies for AI output quality.
• Create test cases covering:
• Accuracy
• Safety
• Tone and style adherence
• Instruction following
• Response consistency
• Perform regression testing for prompt and model changes.
• Track, prioritize, validate, and retest defects using Azure DevOps.

AI Evaluation & Observability

• Utilize Arize or similar observability platforms to monitor model and agent performance.
• Analyze user interactions and evaluation results to identify trends and improvement opportunities.
• Develop quality metrics and dashboards for:
• Accuracy
• Relevance
• Investigate production issues and recommend prompt, workflow, or architectural improvements.

Software Engineering & Integration

• Support integration of AI capabilities into enterprise applications and workflows.
• Collaborate with developers to validate API integrations, retrieval pipelines, and agent workflows.
• Assist in CI/CD deployment and release validation activities using Azure DevOps.
• Create and maintain automated testing assets where appropriate.
• Maintain documentation for prompts, evaluation methodologies, testing procedures, and agent configurations.
• Participate in sprint planning, backlog refinement, and Agile ceremonies.
• Communicate findings and recommendations to technical and non-technical stakeholders.
• Share best practices and lessons learned across the team.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
• 5-10 years of experience in software engineering, QA engineering, automation testing, business systems analysis, or related technical roles.
• 1+ years of hands-on experience with Generative AI, LLMs, prompt engineering, or conversational AI solutions.
• Experience working with Azure DevOps or similar Agile delivery platforms.
• Experience designing and executing structured test plans.
• Strong analytical and troubleshooting skills.
• Experience working with APIs and JSON data structures.
• Excellent written and verbal communication skills.

Preferred Qualifications

• Experience with Arize AI or comparable AI observability tools.
• Experience with Azure OpenAI, OpenAI, Anthropic, Gemini, or other LLM platforms.
• Knowledge of Retrieval-Augmented Generation (RAG) architectures.
• Experience evaluating AI systems using human and automated scoring methods.
• Familiarity with vector databases and semantic search.
• Experience creating automated test frameworks.
• Understanding of responsible AI principles and AI governance.
• Knowledge of prompt versioning and experimentation practices.