Senior QA Engineer

What is the project, and why should you care?

The solution is an early-stage, AI-powered mobile app designed to help users capture, structure, and reflect on their thoughts through natural conversation. The product is inherently non-deterministic and experiential, focusing on intuitive human–AI interaction. As such, we are crafting a deeply personal and emotional user experience via a digital tool.

You will be an excellent fit for this position if you have:

• At least 5 years of hands-on experience in Quality Assurance within the software development industry
• At least 1 year of practical experience testing AI-powered applications, including but not limited to solutions built on large language models (LLMs) and other generative AI technologies
• Experience testing AI-driven applications, particularly those involving large language models (LLMs), natural language processing (NLP), or generative outputs, understanding how to evaluate relevance, tone, and coherence rather than deterministic correctness.
• Understanding of RAG (Retrieval-Augmented Generation) architecture.
• Hands-on experience with AI agent evaluation tools, such as Vertex AI Evaluation and TruLens.
• Familiarity with prompt engineering and the ability to assess how changes to prompts or context affect the quality and consistency of AI-generated responses.
• Understanding of AI system behavior and limitations, including bias, hallucination, and variability, with the ability to design meaningful tests and edge cases to evaluate these aspects from a user perspective.
• Hands-on experience in end-to-end testing of three or more distinct mobile applications in real-world, production environments.
• Demonstrated ability to perform exploratory and user-centric testing, with a strong focus on evaluating the quality of subjective user experiences, such as conversational flow, tone, and emotional resonance in AI-driven interactions
• Strong startup mindset, comfortable operating in fast-paced, evolving environments with minimal structure; eager to build testing approaches from scratch and iterate quickly based on product feedback
• Proven track record of thriving in high-ambiguity, early-stage projects, bringing structure and insights without relying on predefined processes or rigid documentation
• Proficient in QA tools, automation frameworks, and scripting languages such as Python
• Proven experience in API testing (REST/GraphQL) using tools like Postman, Swagger, or custom Python-based test scripts
• Proficient in creating and executing automated tests using Postman and related tools
• Familiarity with test automation tools such as Selenium, PyTest, or Robot Framework
• Experience working with vector databases, SQL-based databases, and document-oriented databases.
• Strong ability to work independently with minimal supervision, proactively identifying issues and driving solutions
• Experience working in Agile/Scrum development environments, participating in sprint ceremonies, and continuous delivery cycles
• Excellent communication skills (both written and verbal) and a strong ability to collaborate effectively in a team-oriented environment
• English C1

Here are some of the things you’ll be working on:

• Work hands-on with real AI model outputs, helping train and fine-tune quality indicators from a QA perspective
• Engage in continuous learning to stay up-to-date with advancements in AI quality assurance, tooling, and conversational interface testing
• Provide feedback on design and UX decisions, especially when they influence usability or affect subjective experience quality
• Contribute to internal documentation around testing heuristics, guidelines for prompt quality, and edge case handling
• Actively support UAT (User Acceptance Testing) and contribute to feedback cycles from internal or early users
• Own testing responsibilities across the product lifecycle and ensure timely, high-quality delivery of assigned tasks
• Collaborate closely and respectfully with cross-functional team members, including stakeholders across multiple geographies
• Contribute to the design of testing approaches for complex, non-deterministic user experiences, particularly in AI-generated content
• Provide clear, structured QA status updates and health reports throughout the development cycle
• Proactively identify risks and collaborate with the Service Delivery Manager (SDM) and product leadership to define mitigation strategies
• Ensure each release meets both functional and experiential quality benchmarks, aligned with the evolving vision of the product
• Participate in defining and refining lightweight QA processes that support rapid iteration in a startup environment
• Contribute to shaping the definition of “quality” for a new category of AI-driven experiences
• Maintain a strong user advocacy mindset and act as a guardian of the end-user experience
• Design and execute targeted test scenarios to evaluate AI-specific risks such as hallucinations, bias, and inconsistent behavior, ensuring responsible and reliable outputs.
• Assess the impact of prompt changes and context variations on AI-generated responses, working closely with product and NLP teams to improve stability and coherence.
• Develop lightweight benchmarking frameworks or tools to continuously evaluate the quality of conversational UX and generative content across iterations.
• Test integrations between AI systems and underlying vector/document/SQL databases, ensuring relevant and performant retrieval in RAG-based architectures.
• Contribute to prompt engineering strategy by experimenting with prompt templates and configurations and reporting on their effectiveness in meeting product goals.