Automation Engineer with AI

About the job

Company Profile:

Founded in 1976, CGI is among the largest independent IT and business consulting services firms in the world. With 94,000 consultants and professionals across the globe, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com.

Job Title: Automation Quality Engineer (AI / LLM Focus)

Position: Lead Analyst

Experience: 9+ years

Category: Lead Analyst/Associate Consultant

Shift: 1PM to 10PM EST Time

Main location: Bangalore/Chennai/Pune/Hyderabad

Position ID: J1225-1830

Employment Type: Full Time

Education Qualification: Any graduation or related field or higher with minimum 3 years of Relevant experience.

Position Description:

We are seeking an Automation Quality Engineer with strong hands-on experience in Java and/or Python automation frameworks and practical exposure to AI-driven testing. The ideal candidate will play a key role in validating intelligent applications, ensuring the quality, reliability, and performance of AI-enabled systems. This role involves testing across traditional automation frameworks as well as advanced AI workflows, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Vector Databases, and OpenAI APIs.

Your future duties and responsibilities

• Design, develop, and maintain automation test frameworks using Java and/or Python.
• Implement UI, API, and backend automation using tools such as Selenium, Playwright, REST Assured, PyTest, or similar.
• Validate LLM-based applications, ensuring prompt accuracy, response quality, hallucination detection, and consistency.
• Test and evaluate RAG pipelines, including document ingestion, embeddings, retrieval accuracy, and response relevance.
• Work with Vector Databases (FAISS, Pinecone, Chroma, Weaviate) to validate indexing, retrieval, and performance.
• Integrate and test applications using OpenAI APIs or similar LLM providers.
• Leverage LangChain (or equivalent frameworks) for testing chains, agents, tools, and workflows.
• Create automated test cases for AI workflows, model outputs, prompt versions, and inference pipelines.
• Implement data-driven and AI-assisted testing strategies.
• Collaborate with Data Scientists, Developers, and Product teams to define quality metrics for AI systems.
• Participate in CI/CD pipelines to enable continuous testing and enforce quality gates.
• Ensure compliance with security, performance, and scalability standards.

Required Qualifications To Be Successful In This Role

• Strong programming skills in Python.
• Hands-on experience with automation testing frameworks (Selenium, Playwright, REST Assured, PyTest, etc.).
• Practical experience with LLMs (OpenAI GPT, Azure OpenAI, or similar).
• Solid understanding and exposure to RAG architectures.
• Experience working with Vector Databases for embeddings and semantic search.
• Hands-on usage of OpenAI APIs (chat/completions, embeddings, etc.).
• Experience with LangChain for prompt chaining, memory, and tool usage.
• Strong knowledge of API testing, SQL, and data validation.
• Familiarity with CI/CD tools (Jenkins, GitHub Actions, Azure DevOps).
• Good understanding of SDLC, STLC, and Agile methodologies.

Good-to-Have Skills

• Certification in LLMs and/or Python.
• Exposure to AI-assisted testing strategies and emerging AI QA tools.
• Experience with performance testing of AI-enabled systems.
• Knowledge of cloud-based AI platforms and orchestration frameworks

Together, as owners, let’s turn meaningful insights into action.