Software Engineer_AI Tester

About the job

Job Responsibilities

• Test and validate LLM outputs, ensuring accuracy, correctness, completeness, consistency, usability, and hallucination analysis.
• Evaluate RAG systems, including retrieval accuracy, document relevance, context construction, and full response generation flows.
• Test AI agents and autonomous workflows, validating decision-making, task execution, and error handling.
• Design and execute AI-specific test strategies: dataset creation, edge-case testing, adversarial testing, pattern-based testing, and regression validation.
• Develop evaluation frameworks, scoring rubrics, and benchmarking models for AI quality assessment.
• Analyze large volumes of AI-generated responses to identify patterns, root causes, and issue clusters, instead of isolated defects.
• Knowledge of testing conversational AI, workflows, or agent-based systems.
• Exposure to vector search tools and embedding quality analysis.
• Validate fixes using new examples from the same pattern category, ensuring true model improvement.
• Collaborate closely with AI/ML engineers, QA teams, and product managers to improve AI accuracy and performance.
• Contribute to continuous improvement of AI QA practices, automation, tools, and evaluation datasets.

Job Requirement

• B.Tech or equivalent degree in Computer Science (or related field).
• 3-5 years of QA experience, with at least 2 years focused on AI/ML testing (LLMs, RAG, AI agents).
• Proficiency in Artificial Intelligence, Machine Learning, AI Agents, and Large Language Models (LLMs).
• Strong knowledge of QA methodologies, test design, functional/non-functional testing, and defect lifecycle management.
• Solid understanding of LLM evaluation, hallucination types, prompt behavior, response scoring, and quality metrics.
• Good understanding of RAG concepts including vector databases, embeddings, and retrieval relevance.
• Coding proficiency in Python or Java (both preferred).
• Hands-on experience in Web/API automation using frameworks like Playwright, Selenium, or REST Assured.
• Experience in writing automation scripts, maintaining test frameworks, and integrating test suites into CI/CD pipelines.
• Ability to analyze large sets of AI outputs for patterns and systemic issues.
• Excellent analytical reasoning, problem-solving, and communication skills.