AI Quality Engineer

Our requirements

• Professional experience in quality engineering, software testing, test automation, data science, software engineering or a related field (we can consider different seniority levels).
• Experience or strong interest in working with AI-powered solutions, either through developing, testing, evaluating or supporting them.
• Good Python proficiency and experience writing clean, maintainable code for analysis, automation or testing purposes.
• Interest in understanding how AI systems behave, how reliable they are, and how their quality can be measured.
• Strong analytical and problem-solving skills, with the ability to investigate issues and identify root causes.
• Ability to translate business objectives, customer outcomes and risks into practical evaluation approaches.
• Strong communication skills and the ability to explain findings to both technical and non-technical stakeholders.
• A curious, proactive mindset and a genuine interest in AI, emerging technologies and system robustness.

Optional

• Experience evaluating, testing or monitoring AI-powered systems in production environments.
• Working with structured data, especially JSON, logs and tabular evaluation results.
• Understanding of LLM behaviour, tokenisation, context limits, parameters and non-determinism.
• Understanding of multi-turn context, RAG, grounding, faithfulness, hallucinations, guardrails, attacks (like prompt injection etc.) and tool/function calling.
• Software engineering practices: Git, modular code, configuration management and automated tests.

Your responsibilities

• Translating business expectations, user outcomes and operational risks into measurable evaluation criteria.
• Defining what successful, partially successful and unsuccessful system behaviour looks like, including the gradient between clear success and clear failure.
• Designing evaluation approaches for conversational, multi-turn and non-deterministic AI systems.
• Building and maintaining representative evaluation datasets covering common, rare, boundary and high-risk scenarios.
• Assessing dataset quality, including coverage, class balance, duplicates, data leakage, label quality, synthetic-versus-production differences and dataset freshness.
• Selecting appropriate manual, deterministic, model-based and statistical evaluation methods.
• Analysing errors by category, user journey, risk level and likely root cause.
• Distinguishing model failures from failures in retrieval, chunking, ranking, prompts, tools, guardrails, integrations, source data and evaluators.
• Validating automated evaluators, including LLM-as-a-Judge, against human judgement.
• Detecting blind spots, noise, bias and misleading aggregation in evaluation results.
• Creating baselines, controlled comparisons and regression tests that allow meaningful comparison between system versions.
• Prioritising tests according to the probability and impact of failure rather than coverage alone.
• Communicating results, uncertainty, limitations and residual risks in a form that supports release and product decisions.
• Proactively identifying emerging failure modes based on system changes, production observations and research.

What we offer

• Contract of employment
• Performance Bonus
• Private medical care (ENEL-MED)
• Cafeteria system of benefits (e.g. MultiSport card)
• Additional parent privileges:
• baby leave – 6 weeks leave for second parent
• shortened working time after parental leave
• Employer-funded group insurance
• Employee Shares Plan
• Extra day off for birthday
• Paid time off to volunteer
• Additional leave acquired over the years in the company (even 5 days more in a year)
• Wellbeing Leave - additional two weeks paid leave after 10 years of service
• Psychological support consultations
• Access to the e-learning platforms
• Internal employee referral program
• Hybrid working model from Poland with a few days in Warsaw office per month
• Attractive office location in Warsaw - Gdański Business Center
• Additional entitlements for people with disabilities - Work Without Barriers
• Work environment that promotes diversity - Diversity Charter

Benefits

• sharing the costs of sports activities
• private medical care
• sharing the costs of professional training & courses
• life insurance
• remote work opportunities
• flexible working time
• saving & investment scheme
• birthday celebration
• employee referral program
• extra leave

Aviva Services Excellence Centre