AI Quality Assurance & GenAI Operations Engineer

Social network you want to login/join with:

AI Quality Assurance & GenAI Operations Engineer, milano

col-narrow-left

Client:

altermAInd

Location:

Job Category:

Other

-

EU work permit required:

Yes

col-narrow-right

Job Reference:

830012664429569638433710

Job Views:

2

Posted:

28.04.2025

Expiry Date:

12.06.2025

col-wide

Job Description:

Do you want to be part of a technology company with an AI-first approach, shaping the invisible by blending AI innovation with human experience?

altermAInd is a company that offers solutions enabling organizations to overcome limits, uncover hidden opportunities, and drive transformative growth.

Location: Flexible / Remote

About the Role

We are looking for a hands-on engineer (mid-level or senior, depending on experience) who will own both Quality Assurance for our AI initiatives and the day-to-day running of our cloud-native Generative-AI applications. You will design and automate test strategies for large-language-model (LLM) workflows, keep production GenAI services healthy, and contribute improvements using Python, Docker, and modern open-source frameworks such as LlamaIndex and AutoGen.

What will you do?

• AI Quality Assurance
• Create, automate, and execute test plans for LLM pipelines, data ingestion, and RAG (Retrieval- Augmented Generation) solutions.
• Monitor model accuracy, bias, and drift; investigate anomalies and coordinate fixes with data scientists.
• Integrate automated tests into our CI/CD pipelines for continuous validation.
• GenAI Operations & Maintenance
• Deploy, scale, and troubleshoot GenAI microservices packaged in Docker on our cloud-native platform (Kubernetes or managed container services).
• Maintain prompt libraries, embeddings stores, and vector indexes built with LlamaIndex, AutoGen, or similar frameworks.
• Implement observability (logging, metrics, alerting) and incident-response runbooks to keep SLAs on track.
• Partner with ML engineers, product managers, and DevOps teams to ship reliable AI features.
• Document test cases, operational playbooks, and post-incident analyses to build organizational knowledge.

Who are you?

• 2+ years in Software QA, SDET, DevOps, or a similar engineering role (experience with AI/ML systems is a plus).
• Practical knowledge of Python for scripting tests, data checks, or automation tasks.
• Hands-on experience with Docker and at least one major cloud provider (AWS, Azure, or GCP).
• Familiarity with CI/CD tooling (GitHub Actions, GitLab CI, or similar).
• Strong understanding of software testing principles and comfort working in Linux-based, cloud-native environments.
• Clear communication skills in Italian, both written and spoken.
• Good knowledge of English.

Preferred (Nice-to-Have)

• Direct experience with LLM orchestration frameworks (LlamaIndex, AutoGen, LangChain).
• Kubernetes (EKS, AKS, GKE, or vanilla K8s) for container orchestration.
• Observability stacks such as Prometheus/Grafana, OpenTelemetry, or Datadog.
• Knowledge of vector databases (e.g., Pinecone, Weaviate) and embedding techniques.
• ISTQB, AWS DevOps, or other relevant certifications.

What do we offer?

• Competitive salary with performance-based bonus.
• Remote-first culture and flexible working hours.
• Annual budget for learning, certifications, and conferences in AI/ML and DevOps.
• Modern equipment plus a stipend for your home office.
• The chance to shape production GenAI systems from the ground up in a fast-growing company.