AI Evaluation Engineer (Python, QA or Security)

About the role

Mindrift is building a dataset to evaluate AI coding agents on real‑world developer tasks. As an AI Evaluation Engineer you will design realistic development environments and create challenging, measurable tasks for AI agents.

Key responsibilities

• Construct virtual company environments with codebases, infrastructure, tickets, documentation and conversation histories.
• Design tasks from intermediate states of these environments, define clear success criteria and ensure they are solvable by an AI agent.
• Write functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, analysing agent failures and refining evaluation criteria.

Required profile

• 5+ years of software development experience.
• Strong English proficiency (minimum B2).
• Hands‑on experience with Python (FastAPI) and JavaScript/TypeScript (React).
• Familiarity with Docker, PostgreSQL, Kafka and Redis.
• Proven ability to write functional and integration tests.

Required skills

• Python
• FastAPI
• JavaScript
• TypeScript
• React
• Docker
• PostgreSQL
• Kafka
• Redis
• Functional testing
• Integration testing

What we offer

• Project‑based freelance work with a flexible schedule.
• Compensation up to $40 USD per hour, based on level and pace.
• Tasks estimated at ~20 hours each, allowing you to set your own workload.