We are looking for a hands-on AI Platform Engineer to design, build, and maintain the Enterprise Agent Development Platform, a production-grade, cloud-native ecosystem that enables engineering teams to define, orchestrate, deploy, and observe AI agents at scale. The role includes standardizing agent development across the organization, implementing advanced agentic evaluation frameworks (using AWS AgentCore), and building robust LLM-as-a-judge and CI/CD deployment gates to reduce development from months to days while enforcing strict security and quality standards. Strong Python skills and deep expertise in LLM evaluation are required, along with experience in cloud infrastructure.ResponsibilitiesDevelop and launch scalable conversational AI systems that operate across multiple channels, including chat, voice, email, and APIsArchitect microservices and asynchronous systems for AI applications, ensuring high availability and performanceUse Python to implement orchestration and decision logic - the control layer that manages tools, workflows, and system interactionsAgentic Evaluation & Quality Gates: Design evaluation frameworks for autonomous agents tracking multi-step workflows, tool-calling accuracy, and session performance, integrating automated gates into CI/CD pipelinesLLM-as-a-Judge & Testing Pipelines: Establish automated evaluation protocols utilizing LLM-as-a-judge patternsOptimize AI models through fine-tuning and LoRA integration, making data-driven decisions between different approaches (FT vs RAG)Work with data specialists to design pipelines for collecting, cleaning, and structuring information for AI agentsDesign CI/CD pipelines, container standards, and cloud strategies for infrastructure migrations, cost optimization, and high availabilityRequirementsStrong background in Python, mainly for hooking up AI models, automating stuff, and handling orchestration logic1–2 years of hands-on experience with Generative AI and large language models (LLMs)Practical experience in machine learning engineering: data preparation, working with vector databases in the cloud, and training/optimizing modelsExpertise in Agentic Evaluation: Hands-on experience designing evaluation frameworks for complex, multi-agent systems and evaluating reasoning traces, tool selection, and agentic workflowsLLMOps & CI/CD Gate Design: Proven experience in building automated testing and evaluation gates within CI/CD pipelines for AI modelsStrong backend and frontend architecture experienceExperience working with cloud platforms such as AWS, GCP, or Azure, including container orchestration and CI/CD pipelinesExperience with agentic frameworks (LangChain, LangGraph) and multi-agent system designComfort with ML frameworks such as TensorFlow, PyTorch, or Scikit-learn is a plusWe offerWe gather like-minded people:Top tech minds driving innovation in AI, cloud and digital platform modernizationSupportive team and agile, startup-like cultureHybrid by design mode and opportunity to work remotely within PolandChance to work abroad for up to 60 days annuallyBusiness-driven relocation opportunitiesWe provide growth opportunities:Career development programsThought leadership, mentoring, soft skills and well-being programsCertification (Anthropic, Gemini, GCP, Azure, AWS)English classesWe cover it all:Stable payParticipation in the Employee Stock Purchase Plan with a 15% discountBenefits package (health insurance, multisport, shopping vouchers)Referral bonuses up to $2,000Offices featuring entertainment and relaxation zones, table tennis and football, free snacks, coffee and moreCorporate, social and well-being eventsPlease, note:Benefits listed above are available to employees onlyWe are open for working with Contractors. Terms of B2B cooperation agreements are agreed individuallyWe will reach out to selected candidates exclusivelyEPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.