AI or ML Engineer - Python, QA Testing

Requisition Number: 2394623
Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.
We are looking for an AI Quality Engineer to help ensure our agentic AI systems are accurate, reliable, secure, and safe. You will define and execute a quality strategy spanning manual testing, test automation, release evaluation, and production monitoring. You will build an automation framework for agent responses, retrieval, tool use, and multi-step workflows, and work with engineering and product teams to turn failures into lasting improvements.
Primary Responsibilities:

• Define and execute a risk-based test strategy covering functional, API, integration, end-to-end, regression, performance, reliability, safety, and security testing for AI agents
• Build and maintain automated evaluation and test frameworks for prompts, model outputs, retrieval, tool calls, agent handoffs, and multi-step workflows
• Perform manual exploratory and scenario-based testing of new capabilities, ambiguous requests, edge cases, and real-world failure modes
• Create and maintain representative test datasets, expected outcomes, and evaluation rubrics. Use deterministic assertions, LLM-as-judge, and human review where each is appropriate
• Run quality checks for every release and for material prompt, model, tool, or data-source change. Integrate repeatable tests into CI/CD pipelines as release gates
• Monitor production quality using measures such as task completion, answer accuracy, groundedness, hallucinations, tool-call success, latency, cost, and user feedback
• Detect drift and quality regressions, set alert thresholds, investigate failures, and expand test coverage based on production findings
• Evaluate risks including prompt injection, sensitive-data exposure, bias, and unauthorized or unsafe agent actions
• Publish quality dashboards and release-readiness reports. Partner with engineering, product, and SRE teams on defect triage, root-cause analysis, and corrective actions
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications:

• Bachelor's degree in Computer Science or a related field, or equivalent practical experience
• 5+ years of experience in software quality engineering or test automation, including 2+ years testing AI/ML, generative AI, or LLM-based systems
• Hands-on experience evaluating LLM applications with test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review
• Experience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests
• Experience integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications
• Experience using logs, traces, dashboards, alerts, and user feedback to investigate production quality issues
• Solid Python skills and experience building or extending test automation frameworks, such as Pytest or Playwright
• Working knowledge of QA methods, defect lifecycle management, Agile delivery, and the software development lifecycle
• Understanding of RAG, prompt engineering, tool use, and multi-step agent workflows
• Demonstrated solid analytical, problem-solving, communication, and collaboration skills

Preferred Qualifications:

• Experience with evaluation or observability tools such as LangSmith, Langfuse, Arize, DeepEval, Ragas, Vertex AI evaluation, or OpenTelemetry
• Experience with GCP, Vertex AI, or Gemini; experience testing applications on AWS or Azure
• Experience with JavaScript/TypeScript or Java, Selenium, advanced Playwright framework development, or browser testing platforms such as BrowserStack or Sauce Labs
• Experience with adversarial testing, responsible AI, and the OWASP Top 10 for LLM Applications
• Experience with A/B testing or canary evaluation of prompt and model changes
• Experience testing conversational AI, multi-agent or MCP-enabled solutions, event-driven services, or API orchestration
• Experience using AI to assist with test generation, failure analysis, or quality reporting
• Experience with performance and load testing tools such as JMeter or LoadRunner, or operational tools such as Dynatrace or SonarQube
• Experience with cloud migration, data migration, or data reconciliation testing
• Healthcare or other regulated-industry experience, including knowledge of HIPAA, data privacy, and security requirements. Experience with claims, eligibility, prior authorization, provider, contact center, or digital health systems

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.