Role: AI QE engineering
Location: Chicago, IL or Irving, TX (Onsite)
Experience: 10+ Years
Engagement: Contract
Job Description:
We are looking for an experienced Sr. AI Quality & Reliability Engineer to drive the design, development, and execution of AI Quality Engineering initiatives supporting enterprise AI transformation. The candidate will be responsible for AI validation, AI-assisted testing, runtime quality controls, and scalable testing frameworks to ensure the reliability and quality of AI-powered applications.
Key Responsibilities:
AI Quality Engineering & Technical Execution
- Design and execute quality engineering activities for AI-powered applications, LLM-enabled workflows, intelligent automation solutions, agentic systems, and enterprise AI platforms.
- Develop AI-native testing approaches, validation processes, runtime quality controls, reusable testing accelerators, and scalable testing workflows.
- Modernize traditional QA practices to support AI-enabled applications and healthcare workflows.
- Establish AI quality standards, validation approaches, release readiness processes, and governance-aligned testing practices.
- Manage AI validation activities, defect management, test execution, release support, and quality improvement initiatives.
AI Validation, Testing & Runtime Assurance
- Perform functional testing, prompt testing, workflow testing, regression testing, and runtime quality assurance.
- Collaborate with AI Engineering, AIOps, LLMOps, Security, Governance, Clinical, and Data teams.
- Support runtime monitoring, distributed tracing, observability, incident management, and production reliability.
- Develop AI evaluation frameworks, validation datasets, quality scoring approaches, and automated testing workflows.
- Support drift detection, human-in-the-loop validation, and AI monitoring initiatives.
- Drive AI-assisted testing, intelligent automation, and reusable test automation frameworks.
- Validate healthcare workflows, payer operations, and AI-enabled business processes.
Delivery Coordination & Operational Support
- Coordinate testing activities, sprint execution, issue tracking, risk identification, and release support.
- Evaluate testing readiness, implementation dependencies, and quality risks.
- Support tooling evaluations, automation initiatives, and testing modernization.
- Establish scalable testing processes and reusable quality engineering frameworks.
Collaboration, Mentorship & Continuous Improvement
- Mentor quality engineers and analysts and provide technical guidance to delivery teams.
- Communicate testing risks, implementation issues, and recommendations to technical and business stakeholders.
- Promote engineering best practices, continuous improvement, and responsible AI adoption.
- Evaluate emerging AI testing, observability, automation, and runtime assurance technologies.
Required Qualifications:
- 10+ years of overall IT experience.
- 5+ years of experience in Quality Engineering, Quality Assurance, Software Testing, or Enterprise Application Delivery.
- 2+ years of technical leadership experience in testing initiatives, automation, or enterprise delivery projects.
- Strong experience in Quality Engineering / QA across enterprise platforms, APIs, operational workflows, healthcare applications, or integrated business systems.
- Experience with test automation frameworks, SDLC, Agile methodologies, defect management, regression testing, release validation, and production support.
- Understanding of AI/LLM validation, prompt testing, AI-assisted testing, and runtime quality practices.
- Strong analytical, communication, collaboration, and leadership skills.
Preferred Skills:
- GenAI and LLM Testing.
- AI Model Evaluation and Validation.
- Prompt Testing and Engineering.
- Agentic AI Testing.
- AI Observability and Runtime Monitoring.
- AIOps / LLMOps.
- Healthcare / Payer Domain Experience.
- Test Automation and API Testing.
