Photon Logo

Photon

QA Lead (Automation+Performance)- Dallas, TX

Reposted 3 Months Ago
Be an Early Applicant
In-Office or Remote
Hiring Remotely in United States
Expert/Leader
In-Office or Remote
Hiring Remotely in United States
Expert/Leader
Lead QA Automation for agentic AI products by designing eval pipelines, golden datasets, and automated tests for tool-use, hallucination detection, latency/token monitoring, and regression across models and prompts. Integrate performance testing into CI/CD and collaborate with AI engineers to convert requirements into measurable automated evaluations.
The summary above was generated by AI

We are seeking a QA Automation Lead who is ready to move beyond traditional "Pass/Fail" testing. In this role, you will design and build automation frameworks specifically for Agentic AI products. You will focus on evaluating the performance of autonomous agents, ensuring they follow logical reasoning paths, call the correct tools, and provide accurate, safe outputs.

Your mission is to build the "evaluations" (Evals) that define what high-quality AI behavior looks like, moving the needle from unpredictable experiments to production-grade software.

Key Responsibilities

  • Non-Deterministic Testing: Develop automation strategies for probabilistic outputs, using model-based evaluation to "test the tester."
  • Building "Eval" Pipelines: Create and maintain "Golden Datasets" to benchmark agent performance across different versions of prompts and models.
  • Tool-Use Validation: Build automated tests to verify that agents call the correct functions/APIs with the right parameters in complex multi-step workflows.
  • Regression Testing for Prompts: Monitor how subtle changes in prompt engineering or model updates (e.g., moving from GPT-4 to Claude 3.5) affect the product’s reliability.
  • Latency & Token Monitoring: Integrate performance testing into the CI/CD pipeline to track agent reasoning time and cost-efficiency.
  • Hallucination Detection: Develop automated checks to identify and report AI hallucinations, bias, or "jailbreak" attempts.
  • Collaboration: Work closely with AI Engineers to translate "vague" business requirements into measurable, automated test cases.

Required Skills & Qualifications

  • Experience: 10+ years in QA Automation, with a recent focus on AI/ML or LLM-based applications.
  • Python Proficiency: Expert-level Python skills (the industry standard for AI testing) and experience with testing frameworks like Pytest.
  • AI Testing Tools: Familiarity with AI evaluation frameworks such as LangSmith, DeepEval, RAGAS, or Promptfoo.
  • API & Backend Testing: Deep experience with Playwright, Selenium, or Cypress for UI, but a heavy focus on API-level testing and database validation.
  • Statistical Mindset: Understanding that AI testing often requires "scoring" (e.g., 85% accuracy) rather than a simple binary pass/fail.
  • Data Skills: Ability to work with SQL and JSON to validate data retrieved by agents during RAG (Retrieval-Augmented Generation) processes.

Preferred Qualifications

  • Experience testing Multi-Agent Systems (where one agent tests another).
  • Knowledge of Prompt Engineering and how it influences software behavior.
  • Background in Investment Banking or Fintech (if applicable) to understand high-stakes data accuracy.

Compensation, Benefits and Duration

Minimum Compensation: USD 38,000
Maximum Compensation: USD 133,000
Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
This position is not available for independent contractors
No applications will be considered if received more than 120 days after the date of this post

Photon Hyderabad, Telangana, IND Office

DLF Cyber City 6th Floor, Block 3 Gachibowli , Hyderabad, India, 500032

Photon Hyderabad, Telangana, IND Office

Hyderabad, India

Similar Jobs

2 Hours Ago
Remote or Hybrid
Junior
Junior
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Handles inbound calls and warm leads, consults customers on insurance needs, recommends appropriate coverages and products, and converts leads into policyholders. The role includes paid training and Property & Casualty licensing support, customer relationship management, sales conversion, and representing the Liberty Mutual brand. Employees work remotely on a fixed weekday and weekend schedule and must maintain a dedicated home workspace with reliable wired internet.
Top Skills: Cable InternetDsl InternetFiber InternetPcWired High-Speed Internet
2 Hours Ago
Remote or Hybrid
Senior level
Senior level
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Owns product vision, strategy, roadmap, backlog, prioritization, and delivery for a workers compensation transformation program. Partners with business and technology stakeholders, leads Agile ceremonies, defines actionable user stories, manages KPIs, resolves delivery barriers, evaluates costs and feasibility, and ensures compliance with industry constraints. Drives customer-focused design, usability testing, stakeholder alignment, and value delivery across squads and portfolios.
Top Skills: Agile
2 Hours Ago
In-Office or Remote
Junior
Junior
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Coordinates nursing home-to-community transitions for Medicaid members. Supports applications, documentation, transportation, housing placements, household needs, and state reporting. Collaborates with members, care managers, nursing facilities, housing providers, and community organizations; monitors transition progress and ensures timely follow-up. Requires experience serving behavioral health or long-term-care populations, community provider knowledge, Microsoft Office proficiency, regional travel, a driver’s license, and reliable transportation.
Top Skills: ExcelMS OfficeMicrosoft OutlookMicrosoft Word

What you need to know about the Hyderabad Tech Scene

Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account