Senior Python Engineer (AI evaluations platform)
Offer summary

(Summary generated by AI based on the full job description)

The project focuses on AI optimization and LLM performance issue detection by designing and implementing complex Reinforcement Learning environments. Required skills include Python, FastAPI, Django, PostgreSQL, AWS. Responsibilities cover building task pipelines, reward models, scalable environments, and APIs for agent management. Tasks include automatic evaluation, telemetry, and performance optimization. Benefits include private medical care, Multisport card, flexible working hours.

newyou can start ASAP

Senior Python Engineer (AI evaluations platform)

Company: ACAISOFT POLAND Sp. z o.o.

from: 11 June 2026
to: 11 July 2026
26 880 - 33 600net (+ VAT)/ mth.B2B contract (full-time)
Offer parameters
level:senior
working mode:remote
Warszawa, Mokotów
Warszawa, MokotówAleja Niepodległości 18View on map

Requirements

Expected technologies

Python
FastAPI
Django
PostgreSQL
AWS

Optional technologies

Next.js
Material UI

Operating system

macOS

Our requirements

  • Ability to work 2:00 p.m. - 10:00 p.m. (CEST)
  • 5+ years as a Python Engineer
  • Experience with frameworks like FastAPI or Django
  • Skills in designing REST APIs
  • Experience with relational databases (PostgreSQL)
  • Experience with cloud environments like AWS

Optional

  • BS/MS in Computer Science, Mathematics, Statistics, or a related quantitative field
  • Experience with Next.js and Material UI
  • Familiarity with vector and columnar databases

Your responsibilities

  • Design and implement RL environments that support large-scale agent evaluation and reinforcement learning experiments
  • Build task generation pipelines, dynamic datasets, and scripted environments with controlled complexity and stochasticity
  • Develop verifiers and reward models to automatically score trajectories and evaluate model reasoning
  • Collaborate with infrastructure and systems engineers to ensure environments are scalable, reproducible, and instrumented for detailed telemetry
  • Design APIs and orchestration frameworks for running, resetting, and evaluating agents across environments
  • Partner with research and customer teams to translate open-ended specifications into verifiable, testable systems
  • Optimize environment performance, logging, and reward reproducibility across distributed setups

About the project

You will be cooperating with a leading provider of AI evaluation and optimization solutions, trusted by multinational companies to optimize AI agents and detect performance issues in large language models.
In this role, you will cooperate with a leading provider of AI evaluation and optimization solutions, directly contributing to a product that helps multinational enterprises optimize AI agents and detect LLM performance issues. You will be responsabile for designing and implementing complex Reinforcement Learning (RL) environments end-to-end, working at the intersection of research, systems engineering, and product. Collaborating with infrastructure and research teams, you will turn conceptual specifications into working environments with verifiable reward structures, automated verifiers, task generation pipelines, and reproducible simulations. These setups will span from API- and web-based tasks to multi-agent simulations, structured reasoning challenges, and knowledge-work environments used at scale by both researchers and customers.
Due to the client’s time zone, we would appreciate a candidate who can work until 10:00 p.m.
Join us and make a real impact!
If you’re ready to broaden your horizons and work with an innovative company at the forefront of AI, we’d love to hear from you. You’ll help build the environments that shape how future AI systems are trained, evaluated, and aligned - and collaborate with world-class engineers and researchers on one of the most important technical challenges of our time.

This is how we organize our work

This is how we work

at the client's siteyou focus on a single project at a timeyou can change the projectyou focus on product developmentagile

Team members

backend developerfrontend developerfullstack developertechnical leaderdata scientist

This is how we work on a project

  • Continuous Deployment
  • Continuous Integration
  • DevOps
Company

What we offer

  • Private medical care
  • Multisport card
  • Friendly, informal atmosphere, and direct contact with everyone in the company
  • Modern technologies

Benefits

  • sharing the costs of sports activities
  • private medical care
  • remote work opportunities
  • flexible working time
  • extra social benefits
  • baby layette
  • school layette
  • employee referral program
  • charity initiatives
  • Gift vouchers for kids (birthdays, Christmas, Child's Day)

Recruitment stages

  • 1.
    HR call (max 10 min.)
  • 2.
    Phone call with Technical Manager (max 10 min.)
  • 3.
    Tech check with the Acai team (~ 30 min.)
  • 4.
    Technical interview(s) with our client (30 - 60 min.)

ACAISOFT POLAND Sp. z o.o.

Acaisoft is a Polish company partnering with industry leaders across healthcare, telecom, fintech, and insurance. We specialize in delivering cloud-native, data-driven, and scalable solutions for enterprises operating in fast-moving environments. We currently serve around 30 active customers worldwide, mostly based in the U.S. We specialize mainly in:
* Data Engineering & AI
* Cloud-Native Solutions, including Cost Optimization
* Platform Engineering
Since technology is constantly being developed at such a fast pace, we always strive to be one step ahead of the market and keep up with the latest solutions.
Senior Python Engineer (AI evaluations platform)
26.9k–33.6k zł / mth. (B2B)
I apply to:
ACAISOFT POLAND Sp. z o.o.
Warszawa, Mokotów
Pracodawca zbiera zgłoszenia przez swój system.
Przejdziesz na zewnętrzny formularz.

By clicking "Aplikuj" you confirm that you've read and accepted our Terms and Conditions.



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Need more information?

  • Make sure the body of the offer doesn’t already include what you’re looking for.
  • Ask a question if you need more information you’re interested in.
  • We’ll forward your question to the employer and aim to provide a response within 3 business days.

Share this offer