Praca w miastach:Praca IT Warszawa•Praca IT Kraków•Praca IT Wrocław•Praca IT Szczecin•Praca IT Poznań•Praca IT Lublin•Praca IT Katowice•Praca IT Bydgoszcz•Praca IT Łódź•Praca IT Białystok•Praca IT Toruń•Praca IT Gdańsk
Stanowiska:Grafik komputerowy praca•Tester gier praca•Programista praca•Tester oprogramowania praca•Frontend Developer praca•Data scientist praca•Tester manualny praca•Tester praca•Analytics praca•Tester aplikacji praca•Programista java praca•Web developer praca
Technologie i narzędzia:Python praca•Cyberbezpieczeństwo praca•Java praca•Bazy danych praca•Cisco praca•C++ praca•Javascript praca•SQL praca•Business Intelligence praca•Delphi praca•Wordpress praca•Testowanie aplikacji praca•C# praca•PHP praca
Specjalizacje:Backend praca•Devops praca•Frontend praca•Full-stack praca•GameDev praca•Mobile praca•Embedded praca•Architecture praca•QA/Testing praca•Security praca•AI/ML praca•Big data science praca•Helpdesk praca•IT admin praca•Agile praca•Product Management praca•Project Management praca•UX/UI praca•Business analytics praca•SAP & ERP praca•System analytics praca•PO/PM praca•Scrum master praca•Analityka praca•Data analytics & BI praca
Offer summary
(Summary generated by AI based on the full job description)
The project focuses on AI, Large Language Models (LLMs) and AI model quality evaluation. Key requirements include experience in designing and validating evaluation metrics, working with ground-truth datasets, analyzing model errors, and developing methods for automated and human-feedback-based evaluation. Responsibilities cover metric design, scoring optimization, and collaboration with teams on evaluation solutions.
Senior Data Scientist – AI/LLM Evaluation
Company: SQUARE ONE RESOURCES sp. z o.o.
from: 28 August 2026
to: 27 September 2026
140 - 170 złnet (+ VAT)/ hr.B2B contract (full-time)
Offer parameters
level:senior • expert
working mode:remote
location:Warszawa, Masovian
Warszawa, Masovian
Requirements
Expected technologies
LLMs
RAG
LLM
AI
NLP
3D
Our requirements
- Proven experience as a Senior Data Scientist or in a similar role.
- Strong data science, statistical analysis, and experimentation skills.
- Hands-on experience with AI/LLM evaluation.
- Strong understanding of LLMs and agentic workflows.
- Experience designing, validating, and optimizing evaluation metrics for AI/ML models.
- Strong analytical skills and experience investigating model errors, including false positives and false negatives.
- Experience working with ground-truth datasets and defining quality criteria.
- Ability to independently analyze experimental results and translate findings into actionable recommendations.
- Strong problem-solving and analytical mindset.
Optional
- Basic knowledge of 3D, geometry, or rendering.
- Experience with LLM-as-a-Judge / LLM-based evaluation systems.
- Experience with human-in-the-loop evaluation methodologies.
- Previous experience evaluating AI agents or agentic systems.
- Familiarity with advanced approaches to evaluating LLM-generated content and outputs.
Your responsibilities
- Design and validate evaluation metrics and ground-truth datasets.
- Analyze evaluation quality, including accuracy, false positives, and false negatives.
- Optimize metric normalization and scoring methodologies.
- Develop and improve LLM-based judges for automated evaluation.
- Design and implement human-feedback-based evaluation approaches.
- Explore and develop new evaluation metrics, including:
- complexity,
- prompt adherence,
- output quality and correctness.
- Analyze experiment results and identify opportunities to improve model performance and evaluation methodologies.
- Collaborate with Data Science, AI/ML, and Engineering teams to develop evaluation solutions for AI models and agentic workflows.
About the project
We are looking for a Senior Data Scientist to join an international project focused on Artificial Intelligence, Large Language Models (LLMs), and AI evaluation.
The project focuses on developing and improving methodologies for evaluating the quality and performance of AI models and agentic workflows. The role involves designing evaluation metrics, building ground-truth datasets, analyzing model performance, and developing advanced approaches to automated and human-feedback-based evaluation.
This is how we organize our work
This is how we work
at the client's site
SQUARE ONE RESOURCES sp. z o.o.
At Square One Poland we link IT experts with the business. With over 25 years of experience, we specialize in recruitment processes on a global scale. Despite years of experience, we still have a startup DNA and this is our advantage. Our offices are located in London and Warsaw, however, we can reach clients from all over the world, from start-ups to big worldwide corporations.
This is how we work
Senior Data Scientist – AI/LLM Evaluation
140–170 zł / hr. (B2B)
I apply to:
SQUARE ONE RESOURCES sp. z o.o.
Warszawa, Masovian
Need more information?
- Make sure the body of the offer doesn’t already include what you’re looking for.
- Ask a question if you need more information you’re interested in.
- We’ll forward your question to the employer and aim to provide a response within 3 business days.