Senior Software QA Engineer

Crimson Education; United States; Hybrid; USD 80,000 - 110,000; Full time

Benefits offered by this job

Collaborative work environment

Focus on AI safety and ethics

Opportunities for professional growth

Job summary

Crimson Education is seeking a Senior Software QA Engineer to design and scale quality automation for AI-powered applications. The role involves developing automation test frameworks, integrating testing into CI/CD pipelines, and ensuring robust performance metrics.

Candidates should have over 4 years of experience in Software QA Automation, proficiency in Python, and familiarity with modern automation tools. This full-time position operates in a hybrid work environment, offering a collaborative atmosphere focused on innovation.

Qualifications

• 4+ years of experience in Software QA Automation, with a focus on AI/ML applications.
• Strong programming proficiency in Python or JavaScript/TypeScript.
• Hands-on experience with automation tools like Playwright, Cypress, and Selenium.

Responsibilities

• Design and build end-to-end automation test frameworks.
• Integrate automated tests into CI/CD pipelines for deployments.
• Conduct API performance and load testing to meet demand.

Skills

Software QA Automation

Python

API testing

Automation tools (Playwright, Cypress, Selenium)

AI evaluation frameworks (Ragas, TruLens, Deepchecks)

Education

Bachelor’s degree in Computer Science

Tools

Postman

RestAssured

PyTest

Job description

Senior Software QA Engineer

Senior Software QA Engineer at Crimson Education: design and scale end-to-end quality automation for AI-powered apps, own test coverage, and ensure safe, reliable admissions tech.

The Role

Responsibilities

• Design & Build: Develop, maintain, and scale robust, end-to-end automation test frameworks from scratch (UI, API, and Integration layers).
• CI/CD Integration: Seamlessly bake automated test suites into our CI/CD pipelines to ensure rapid, high-confidence deployments.
• Coverage Strategy: Define, track, and scale test coverage metrics (targeting high-purity API and critical UI path coverage) to ensure regression-free deployments.
• Performance & Scale: Conduct API performance, load, and stress testing to ensure our infrastructure keeps up with demand.
• AI/LLM Validation: Design testing strategies for non-deterministic systems. You’ll evaluate LLM outputs for accuracy, bias, safety, and hallucination rates.
• Prompt Regression Testing: Establish benchmarks to ensure that tweaking an AI prompt or upgrading a foundational model doesn't secretly break existing features.
• Data & Model Drift Monitoring: Collaborate with Data Scientists and ML Engineers to monitor model performance metrics in production.
• Edge Case Hunting: Actively simulate adversarial inputs (prompt injections, jailbreaks) to ensure our AI features remain secure and guardrailed.

Qualifications

• 4+ years of experience in Software QA Automation, with at least 1–2 years specifically focused on testing AI/ML-powered applications or LLM integrations.
• Strong programming proficiency in Python (highly preferred for AI workflows) or JavaScript/TypeScript.
• Hands-on experience with modern automation tools (e.g., Playwright, Cypress, Selenium).
• Expertise in API testing tools (e.g., Postman, RestAssured, PyTest).
• Familiarity with AI evaluation frameworks (e.g., Ragas, TruLens, Deepchecks) or a proven ability to write custom evaluation scripts for probabilistic outputs.
• Basic understanding of vector databases, embeddings, and how data pipelines feed into ML models.
• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.

Bonus Points

• Familiarity with cloud platforms (AWS, GCP, or Azure) and containerization (Docker, Kubernetes).
• Healthy obsession with AI safety and ethics.

Work Details

• Department: Product & Technology
• Locations: Multiple locations
• Remote status: Hybrid
• Employment type: Full-time