QA Engineer

Medlytix LLC.; Roswell (GA); On-site; USD 90,000 - 130,000; Full time

Job summary

Medlytix LLC. is seeking an innovative QA Engineer to lead automation and AI-assisted testing for our AI-driven solutions. You will design robust testing strategies, validate model outputs, and ensure data quality across pipelines and endpoints.

You will work with AI Integration engineers and developers to implement CI/CD integrated testing, build reusable test libraries, and explore AI-powered testing tools for data generation, anomaly detection, and log analysis.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 3-5 years of software quality assurance and test automation experience.
• Proven ability to build and maintain automation frameworks from scratch.
• Experience testing cloud-based applications and APIs; ML model testing experience is a plus.

Responsibilities

• Design, develop, and maintain automated test frameworks using Python and modern testing tools.
• Build end-to-end test suites for APIs, data pipelines, and AI model endpoints.
• Implement continuous testing practices integrated with CI/CD pipelines.
• Create reusable test libraries and utilities to accelerate testing across projects.
• Define QA best practices and quality metrics for AI products.

Skills

Python

QA/Testing

Data pipelines

Cloud testing

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

pytest

unittest

Selenium

Postman

REST Assured

GitLab CI

GitHub Actions

Git

SQL

Docker

Kubernetes

Airflow

Temporal

Prefect

n8n

Job description

We're seeking an innovative QA Engineer to join our team. This role goes beyond traditional testing - you'll be at the forefront of using automation frameworks and AI-powered tools to ensure the quality, reliability, and performance of our AI solutions. You'll work closely with AI Integration engineers and software developers to build robust testing strategies that handle the unique challenges of AI systems, including model validation, data quality, and non-deterministic outputs.

Key Responsibilities

Test Automation Development

• Design, develop, and maintain automated test frameworks using Python and modern testing tools
• Build end-to-end test suites for APIs, data pipelines, and AI model/MCP endpoints
• Implement continuous testing practices integrated with CI/CD pipelines
• Create reusable test libraries and utilities to accelerate testing across projects

AI-Assisted Testing

• Leverage AI tools (like GitHub Copilot, ChatGPT, or dedicated test generation tools) to accelerate test case creation and code reviews
• Explore and implement AI-powered testing solutions for test data generation, visual testing, and anomaly detection
• Experiment with automated test scenario generation using LLMs
• Use AI tools for log analysis and defect pattern recognition

AI-Specific Testing

• Develop testing strategies for LLMs including performance, accuracy, and bias detection
• Test data quality, feature engineering pipelines, and model training workflows
• Validate model output and monitor for model drift in production
• Create test datasets that cover edge cases and ensure model robustness

Quality Strategy & Collaboration

• Define and implement QA best practices and quality metrics for AI products
• Collaborate with developers to identify testability requirements early in the development cycle
• Perform code reviews focused on test coverage and quality
• Document test strategies, test cases, and quality reports

Performance & Security Testing

• Conduct load and performance testing for AI inference endpoints
• Identify bottlenecks in data processing and model serving infrastructure
• Participate in security testing activities focusing on data privacy and model security

Required Qualifications

Education & Experience

• Bachelor's degree in Computer Science, Engineering, or related field
• 3-5 years of experience in software quality assurance and test automation
• Proven experience building and maintaining automation frameworks from scratch
• Experience testing cloud-based applications and APIs
• Experience testing machine learning models, agentic data flows and data-intensive applications

Technical Skills

• Strong proficiency in Python for test automation
• Hands-on experience with testing frameworks: pytest, unittest, Selenium, or similar
• Experience with API testing tools: Postman, REST Assured, or Python requests
• Knowledge of CI/CD tools: GitLab CI, GitHub Actions
• Familiarity with version control systems (Git)
• Understanding of SQL and database testing (incl. Data Lake architectures)
• Experience with containerization (Docker) and orchestration tools
• Experience with AI-powered testing tools or test generation platforms
• Experience with cloud platforms: AWS, Azure, or GCP
• Understanding of microservices architecture and distributed systems
• Experience with workflow orchestration tools (Airflow, Temporal, Prefect, n8n)
• Experience mentoring junior QA engineers

What You'll Work With:

Languages: Python, SQL, JavaScript/TypeScript, YAML
Data Tools: Airflow, dbt, Spark, Polars, pyArrow
APIs & Integration: FastAPI, GraphQL, Kafka, Redis, Javascript/Typescript, WASM
AI/ML: OpenAI API, Pydantic AI, Anthropic Claude, LangChain, vector databases, MCP protocol
Cloud: AWS (Lambda, S3, RDS, Bedrock, SageMaker) or equivalent in GCP/Azure
Infrastructure: Docker, Kubernetes, Terraform, GitHub Action

As part of our commitment to quality and excellence, Medlytix will continue to maintain a safe and healthy environment for you by requiring all applicants to submit to a criminal history check and those tentatively selected for a position to submit to screening for illegal drug use prior to appointment for a job. In addition, applicants may be screened for ability to perform essential functions of some positions.