Quality Engineer

We're looking for a Quality Engineer to join EPAM in London, in an onsite working mode, contributing to an AI-driven enterprise project for one of our clients. This role focuses on exposing internal enterprise tooling to AI agents via Model Context Protocol (MCP), ensuring robust quality engineering and compliance across innovative intelligent automation solutions.
As a Quality Engineer, you will be responsible for developing automated testing frameworks, evaluation pipelines, and quality controls for MCP Components within our AI platform. You will work closely with engineering and product teams in a Classic Agile environment to guarantee the reliability, accuracy, and performance of agent-driven workflows operating in enterprise-scale systems. This position offers a chance to define standards for testing AI models and tools in production-like environments while being a critical part of one of EPAM’s most transformative projects.
Responsibilities
Develop automated testing frameworks to validate MCP Servers and related AI systems
Design and implement evaluation strategies for LLM accuracy, safety, and reliability
Create automated tests using Python, Pytest, and BDD frameworks
Build quality gates into CI/CD pipelines to maintain continuous assurance
Identify and address agentic AI failure modes such as hallucination, latency, and incorrect tool usage
Collaborate with engineering, QA, and product teams to define quality metrics and acceptance criteria
Contribute to Agile ceremonies, ensuring testing practices align with sprint goals
Prepare detailed reporting on quality outcomes and improvement opportunities
Maintain documentation for test cases, evaluation pipelines, and validation strategies
Requirements
Strong programming experience in Python applied to test automation and evaluation
Expertise in Pytest and familiarity with BDD frameworks such as Behave or Cucumber
Knowledge of LLM evaluation approaches including RAGAS, DeepEval, or custom pipelines
Understanding of common agentic AI issues such as hallucination, tool misuse, and performance bottlenecks
Familiarity with automated testing of AI workflows, distributed systems, or microservices environments
Strong grasp of Agile delivery methodologies and CI/CD integration for quality checks
Excellent communication and problem-solving skills with a focus on accuracy and reliability
Nice to have
Experience with Model Context Protocol (MCP) or other agent orchestration solutions
Exposure to observability, monitoring, or logging tools for AI systems
API and service integration testing background for multi-layered platforms
Knowledge of containerized environments and cloud-native architecture
Background in enterprise AI automation projects or intelligent platform engineering