Senior AI Quality Engineer (Hyderabad-based)

We are looking for a Senior AI Quality Engineer (L3+) with strong Python skills and hands-on experience testing AI and ML systems. You will analyze requirements, define evaluation strategies, build automated checks, validate models and outputs, test APIs and data pipelines, verify defects through resolution, and contribute to release readiness for your assigned product area. You will both evaluate AI systems and design, build, and operate QE agents for the software testing life cycle.

AI-assisted work is part of everyday QE practice on this engagement, with Ragas, DeepEval, promptfoo, Cursor, and other AI and ML tools used for evaluation, scripting, test generation, analysis, and maintenance.

You will also build golden datasets, calibrate LLM-as-judge methods, test agentic workflows and RAG systems, implement self-healing and intelligent test approaches, and diagnose quality, safety, performance, and cost failures across distributed AI services.

This position is intended for Hyderabad-based candidates who are available to work from the office.

The client delivers AI agents, retrieval systems, models, and data pipelines across a broad technology landscape. Quality is part of delivery through evaluation gates, automated regression, model and data validation, security and privacy checks, and evidence that a release is ready.

Python is the primary language for AI evaluation, model validation, API testing, data-pipeline quality, and QE agent design. The role combines deterministic automation with probabilistic evaluation to assess AI behavior across models, data, retrieval, tools, and integration surfaces, and designs and operates agents that support the software testing life cycle.

• Analyze requirements and define evaluation strategies, golden datasets, regression benchmarks, baselines, thresholds, and blocking metrics with domain-expert review.
• Design, build, and orchestrate AI agents, persona-based workflows, and reusable skills for the STLC using LangChain, LangGraph, or Semantic Kernel.
• Build automated evaluations and benchmarks for LLMs, agents, RAG, retrieval, tool trajectories, and contracts using Ragas, DeepEval, and promptfoo.
• Calibrate LLM-as-judge against human grading, manage non-determinism through repeated runs, and detect regression and drift.
• Write and maintain Python and pytest automation for AI, ML, API, data, and integration workflows.
• Use Cursor and agentic tools to analyze requirements, generate and review tests, and maintain evaluation suites.
• Validate deep-learning, reinforcement-learning, and predictive models using LangChain.
• Build AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, and defect triage, plus intelligent test selection and defect prediction.
• Integrate LLMs, APIs, embeddings, vector databases, retrieval pipelines, and enterprise data sources into QE frameworks using Python.
• Test REST and streaming APIs, agent tools, and RAG and MCP integrations, including authentication, errors, retries, and recovery.
• Test data and retrieval pipelines for schema integrity, lineage, idempotency, fallback, promotion gates, and rollback.
• Integrate evaluations, benchmarks, and agent workflows into QE frameworks and CI/CD on GitHub Actions, review Allure results, and keep TestRail aligned with automated coverage.
• Test prompt injection, jailbreaks, unsafe tool use, data exfiltration, access control, PII controls, and guardrails.
• Measure accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnose distributed failures.
• Implement tracing, monitoring, and production evaluation using Langfuse, LangSmith, or a comparable AI observability platform.
• Deploy and operate QE agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.
• Deploy and validate AI and ML solutions on AWS or Azure, including secrets, database services, and cloud-native test infrastructure.
• Report, track, and verify defects in Jira, and work with domain experts, product, and engineering on traceable coverage and untested scope.
• Contribute to release readiness for the assigned product area without owning sign-off for the whole team.

• Strong Python proficiency with pytest for test and evaluation automation.
• Hands-on experience building and orchestrating AI agents and agentic workflows with LangChain, LangGraph, Semantic Kernel, or comparable frameworks.
• Hands-on experience building AI evaluations and benchmarks with Ragas, DeepEval, promptfoo, or equivalent tools.
• Hands-on use of Cursor or agentic development tools for test generation and maintenance.
• Experience defining AI evaluation strategies, golden datasets, regression benchmarks, and LLM-as-judge calibration, including non-determinism and drift.
• Experience validating deep-learning, reinforcement-learning, and predictive models with LangChain.
• Experience building AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, defect triage, intelligent test selection, and defect prediction.
• Experience integrating LLMs, APIs, embeddings, vector databases, and retrieval pipelines with automation frameworks in Python.
• Experience testing REST and streaming APIs, agent tools, RAG workflows, and MCP integrations.
• Experience testing data and retrieval pipelines for correctness, lineage, and safe promotion.
• Experience integrating AI evaluation and agent workflows with automation frameworks and CI/CD using GitHub Actions, Allure, TestRail, or equivalent tools.
• Experience testing AI security, privacy, access control, and guardrails.
• Experience measuring accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnosing distributed failures.
• Experience with AI tracing, monitoring, and production evaluation using Langfuse, LangSmith, or comparable tools.
• Familiarity with AWS, Azure, or GCP cloud AI services and agent platforms for deploying and validating AI and ML solutions.
• Experience deploying and operating agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.
• Experience managing defects through resolution in Jira or an equivalent tool.
• Ability to work with domain experts, product, and engineering on ground truth and coverage gaps.
• Experience contributing to release readiness for an assigned product area.

• Playwright and TypeScript experience for UI and API automation.
• Real-device and cross-browser execution using LambdaTest, Sauce Labs, or an equivalent platform.
• Knowledge of AI-driven test optimization and coverage analytics using SeaLights or an equivalent platform.
• Experience contributing reusable components to shared automation frameworks and reviewing automation code through Git and GitHub workflows.
• Experience managing test data and troubleshooting test environments needed for reliable execution.
• Experience triaging security findings and validating operational readiness, monitoring, rollback, and recovery controls.
• Experience producing audit-ready evaluation reports and improving shared evaluation patterns, automation libraries, and datasets.