Go back

Senior QA Engineer (Conversational AI)

About the Role:

We are seeking a detail-oriented Senior QA Engineer (AI & Automation Testing) to join our technical team in ensuring the reliability, accuracy, and security of Candid Chat AI across both web and mobile platforms.

In this role, you will go beyond traditional software testing by actively evaluating Large Language Models (LLMs), conversational flows, and system integrations. You will design, execute, and automate tests to benchmark AI output quality, conduct rigorous adversarial and safety testing, and integrate automated quality gates into our CI/CD pipelines.

Schedule:

  • Monday to Friday, 10:00 AM to 7:00 PM (Australian Eastern Time, 40 hours per week)

Key Responsibilities:

  • AI & Conversational Testing: Test end-to-end conversational AI flows across web and mobile interfaces to ensure seamless user experiences.
  • Model Evaluation & Quality Assurance: Systematically evaluate AI responses for factual accuracy, context relevance, tone consistency, and hallucination detection.
  • Adversarial & Safety Testing: Perform red-teaming and adversarial testing, including prompt injection, jailbreaking attempts, and guardrail verification to ensure model safety and compliance.
  • Regression Testing: Design and execute regression test suites whenever model versions, system prompts, or backend configurations are updated.
  • Test Automation: Build and maintain UI and API automated testing frameworks for frontend applications and underlying backend services.
  • Benchmarking & Datasets: Construct and maintain benchmark test datasets to track, measure, and score AI response quality over time.
  • CI/CD Integration: Integrate automated AI quality and regression checks directly into the continuous integration and deployment (CI/CD) pipelines.
  • Bug Documentation & Escalation: Identify, isolate, and document AI-specific bugs, model drift, and system anomalies with clear reproduction steps for Machine Learning and Engineering teams.

Must-have Requirements:

  • Experience: 4+ years of experience in Software Quality Assurance, with strong manual and automation testing experience.
  • UI & API Testing: Proven experience in UI testing, API testing, and test automation using tools such as Selenium, Playwright, Cypress, Postman, REST Assured, or Python.
  • Technical Proficiency: Strong scripting skills in Python or JavaScript, with experience in CI/CD tools such as GitHub Actions, GitLab CI, or Jenkins.
  • Adaptability to AI Testing: Ability to quickly learn and adapt testing approaches to conversational AI, chatbots, and LLM-powered applications, including evaluating response accuracy, relevance, consistency, and hallucinations.
  • Communication & Detail: Strong analytical and problem-solving skills with the ability to communicate complex model behavior issues clearly to engineering teams.

Nice to have:

  • Industry background: Background in financial technology, SaaS, or high-growth tech environments.
  • Conversational AI Experience: Previous experience testing conversational AI, chatbots, NLP models, or LLM-powered applications.
  • LLM Evaluation & Safety Testing: Hands-on experience evaluating hallucinations, prompt safety, model responses, and guardrails.
  • Adversarial Testing: Familiarity with common LLM vulnerabilities, prompt injection techniques, jailbreaking attempts, and AI security testing.
  • Specialized Testing Experience: Experience testing AI-driven conversational flows across web and mobile applications, including end-to-end testing of complex interactions.

Independent Contractor Perks

  • Permanent work from home
  • Immediate hiring
  • Health Insurance Coverage for eligible locations

Note:

Please click the Apply Now button to complete your application, including the assessment questions, technical check, and voice recording. Your hourly pay rate will be established based on your performance in the application process; submissions with all requirements fulfilled will receive priority review.

One application, multiple possibilities: When you apply, our team reviews your background against all current openings, not just this one! If a different role fits your skills, we'll get in touch! We also encourage you to keep exploring our job board and apply directly to any position that excites you.

Senior QA Engineer (Conversational AI)

Job Category

Information Technology

Job Type

Full Time (35 hours or more per week)

Work Schedule and Timezone

Monday to Friday, 10:00 AM to 7:00 PM Sydney, NSW

Published on

Nov 16 2026

BruntWork will never ask you for money or any other form of payment. If someone claiming to represent BruntWork is requesting a payment from you, please let us know at applications@bruntwork.co

“BruntWork made the entire recruitment process smooth, transparent, and stress-free. They matched me with a client that genuinely fits my skills and values — and the support didn’t stop at placement... A reliable, professional partner I’d recommend without hesitation.”

— Zyrrah D, Bookkeeper

Google rating
4.9/5
Glassdoor rating
4.9/5