AI Quality Engineering Team Lead

Our requirements

• Several years of relevant professional experience in quality engineering, software testing, software engineering, data science or a related technical field.
• Hands-on experience evaluating AI, large language model, RAG or agent-based applications.
• Practical Python skills and experience with APIs, logs, traces, structured datasets and SQL or similar
• analysis.
• Ability to design repeatable evaluations and automation, not just manual testing or vendor scoring.
• Good understanding of sampling, thresholds, trends and non-deterministic system behaviour.
• Experience with automated testing, version control, CI/CD and maintainable engineering practices.
• Ability to turn technical findings into decision-quality evidence for business, risk and engineering stakeholders.
• Experience coaching or leading engineers in an emerging or ambiguous domain.
• Strong communication and influencing skills, with the confidence to challenge weak evidence.

Optional

• Experience leading projects, coordinating technical workstreams or managing people.
• Experience with AI evaluation or observability tools such as DeepEval, Arize or similar platforms.
• Experience designing shared frameworks, reusable services or platform capabilities.
• Knowledge of responsible AI, model risk, controls, privacy or customer-outcome requirements.
• Experience in financial services, insurance or another regulated industry.
• Exposure to cloud AI systems, observability and production support.

Your responsibilities

• Leading the quality engineering approach for AI solutions, from risk assessment through release and monitoring.
• Evaluating multi-turn, RAG and agentic AI behaviour, including grounding, guardrails and tool use.
• Building representative datasets and assessing coverage, balance, leakage, label quality and freshness.
• Assessing correctness, relevance, hallucination, safety, bias, privacy and behavioural drift.
• Investigating unexpected outcomes across data, prompts, models, retrieval, integration, orchestration and evaluators.
• Building reusable evaluation components, automated test pipelines, reporting and evidence-generation patterns for multiple AI products.
• Combining test methods and validating automated evaluators against human judgement.
• Using baselines, key metrics and risk-based regression tests to support release decisions.
• Integrating evaluation into CI/CD, observability and engineering workflows with traceability across datasets, prompts, models and metrics.
• Producing concise, auditable findings that explain confidence, uncertainty, customer impact and residual risk.
• Partnering with product, engineering, architecture, data, risk, compliance and quality stakeholders to influence delivery and release decisions.
• Coaching and guiding AI Quality Engineers, planning team work, reviewing outputs and building a collaborative culture of constructive challenge and learning.

What we offer

• Contract of employment
• Performance Bonus
• Private medical care (ENEL-MED)
• Cafeteria system of benefits (e.g. MultiSport card)
• Additional parent privileges:
• baby leave – 6 weeks leave for second parent
• shortened working time after parental leave
• Employer-funded group insurance
• Employee Shares Plan
• Extra day off for birthday
• Paid time off to volunteer
• Additional leave acquired over the years in the company (even 5 days more in a year)
• Wellbeing Leave - additional two weeks paid leave after 10 years of service
• Psychological support consultations
• Access to the e-learning platforms
• Internal employee referral program
• Hybrid working model from Poland with a few days in Warsaw office per month
• Attractive office location in Warsaw - Gdański Business Center
• Additional entitlements for people with disabilities - Work Without Barriers
• Work environment that promotes diversity - Diversity Charter

Benefits

• sharing the costs of sports activities
• private medical care
• sharing the costs of professional training & courses
• life insurance
• remote work opportunities
• flexible working time
• saving & investment scheme
• birthday celebration
• employee referral program
• extra leave

Aviva Services Excellence Centre