QA Engineer - Agentic AI & LLM System

Location : Hyderabad/New DelhiMode : HybridShift : UK Shift

About the Role

Are you passionate about ensuring Artificial Intelligence is reliable, safe, and truthful? We are looking for a QA Engineer to join our AI team to focus exclusively on testing the next generation of Agentic AI systems.

In this role, you will act as the gatekeeper of quality, ensuring our LLM applications, AI agents, and RAG systems produce accurate, logical, and reliable outcomes.

• Architect and implement comprehensive evaluation strategies for LLMs and Agentic AI systems to measure accuracy, safety, and reasoning quality.

• Develop test cases specifically designed to detect hallucinations, factual errors, logical inconsistencies, tool misuse, and retrieval failures in RAG pipelines.

• Create and maintain benchmark datasets and automated test suites to validate AI responses against ground truth data.

• Evaluate the performance of AI agents performing complex, multi-step tasks and tool-using operations.

• Partner closely with AI engineers and data scientists to debug model behavior and iterate on prompt engineering based on test findings.

• 3+ years in Quality Assurance or Software Testing, with a strong focus on AI/ML systems in the last 1 - 2 years.

• Deep understanding of LLM behaviors (hallucinations, reasoning chains) and Agentic AI architecture (tool use, RAG).

• Experience using Python for test automation and API testing tools.
• Familiarity with LLM evaluation libraries (e.g., LangChain, DeepEval, or similar).

• A sharp eye for detail and the ability to classify ambiguous outputs (e.g., distinguishing between a 'creative' answer and a 'factually incorrect' one).

• Experience with prompt injection security testing or knowledge of ML pipelines