Anaplan
Senior Sdet For Ai
Overview
We're pioneering a new role focused exclusively on quality assurance for GenAI/Agentic systems. As our Senior SDET for AI, you'll develop testing strategies, evaluation frameworks, and quality metrics specifically designed for LLM-powered applications.
About Anaplan
At Anaplan, we are a team of innovators focused on optimizing business decision-making through our leading AI-infused scenario planning and analysis platform so our customers can outpace their competition and the market.
Requirements & Eligibility
- Deep understanding of GenAI technologies including LLMs, prompt engineering, and AI application patterns.
- Strong hands-on Python coding experience.
- Expert experience with test automation frameworks and scripting (Python, JavaScript, Selenium, Pytest).
- Strong background of software testing methodologies (functional, integration, regression, performance, security testing).
- Proven ability to design test cases and evaluation criteria for non-deterministic systems.
- Strong analytical and problem-solving skills with attention to detail.
- Experience with API testing tools (Postman, REST Assured) and backend testing.
- Strong experience with CI/CD pipelines and automated testing integration.
Key Responsibilities
- Design and architect comprehensive test plans, evaluation frameworks, and quality metrics designed specifically for LLM-powered applications and agentic workflows.
- Develop robust, end-to-end automated test suites in Python across evaluations, API checks, UI checks, and performance tests.
- Create automated prompt testing systems and regression suites to proactively detect unintended changes in model behaviour.
- Establish multi-dimensional evaluation criteria to measure and track GenAI quality across accuracy, relevance, safety, consistency, and latency.
- Perform adversarial testing (red-teaming) to identify and mitigate model hallucinations, biases, or security vulnerabilities.
- Develop internal tools and testing frameworks that enable software engineering teams to seamlessly validate their own GenAI implementations.
- Deploy real-time monitoring and alerting systems to quickly detect and flag quality degradation of AI features in production.
- Own the overall quality and testing strategy for your workstream, maintaining high-fidelity test datasets ("golden examples") and defining QA best practices.
Disclaimer: Trace Hiring is an independent job board. We are not directly affiliated with Anaplan. Please verify all details on the official company application portal.