AI Automation Tester (Hybrid Setup)

Overview We are seeking an experienced AI Automation Tester to join our AI Center of Excellence (AI COE). This role will be responsible for ensuring the quality, reliability, and scalability of AI-powered solutions across the organization. The ideal candidate brings strong test automation expertise and has experience testing AI-driven applications, chatbots, copilots, or Large Language Model (LLM) based solutions. As part of the AI COE, you will help define testing standards, automation frameworks, and quality assurance practices that support enterprise AI initiatives. Key Responsibilities Design, develop, and maintain automated test frameworks for AI-enabled applications, APIs, and enterprise platforms. Create and execute automated and manual test cases for Generative AI, chatbot, copilot, and intelligent automation solutions. Validate AI-generated responses for accuracy, consistency, relevance, and compliance with business requirements. Develop test scenarios covering prompt validation, response quality, contextual understanding, and conversational flows. Identify and document hallucinations, inaccuracies, bias, safety concerns, and other AI-related quality issues. Validate Retrieval-Augmented Generation (RAG) solutions and ensure responses are aligned with approved knowledge sources. Perform functional, integration, regression, and end-to-end testing across AI-enabled business processes. Test AI workflows that integrate with enterprise systems, APIs, databases, and third-party applications. Contribute to AI testing standards, methodologies, governance, and best practices within the AI COE. Collaborate with AI Engineers, Data Scientists, Product Owners, Developers, and Business Stakeholders to improve solution quality. Support continuous integration and deployment processes by integrating automated tests into CI/CD pipelines. Produce testing documentation, quality metrics, and recommendations for AI solution improvements. Required Qualifications 5+ years of experience in Software Quality Assurance, Test Automation, or Quality Engineering. Strong hands-on experience with automation testing tools such as Selenium, Playwright, Cypress, Robot Framework, or equivalent. Experience developing automated test scripts using Python, Java, JavaScript, or similar languages. Strong experience in API testing and validation of RESTful services. Solid SQL and database testing skills. Experience working with CI/CD tools such as Azure DevOps, Jenkins, GitHub Actions, or GitLab. Strong understanding of software testing methodologies, automation frameworks, and Agile delivery practices. Experience designing and executing end-to-end test strategies for complex applications. Preferred Qualifications Experience testing AI-powered applications, chatbots, copilots, virtual assistants, or Generative AI solutions. Exposure to LLM platforms such as OpenAI, Microsoft Azure AI, Anthropic Claude, or Google Gemini. Understanding of prompt engineering, AI evaluation techniques, and LLM testing methodologies. Experience validating RAG solutions and AI knowledge-based systems. Familiarity with AI testing and evaluation tools such as Promptfoo, DeepEval, or LangSmith. Experience working with cloud platforms such as Azure, AWS, or Google Cloud.