Evaluation of AI Agents: Quality vs Cost

We can use various agents: Claude Code, Codex, Aider with DeepSeek, pi.dev, Mirror with Beam and others, but do we choose them based on their real capabilities or marketing? Let's check in practice: give them the same tasks and see what happens. We will perform tasks from simple to tricky. Each agent works alone, then we compare results. What we look at Whether the solution works and handles edge casesHow fast and efficient the generated code wasToken usage and their costHow many times a human had to step in and fix the direction The same model in a different tool can perform differently. We want to see how the harness changes the model's results on the same task. Evening plan Quick intros, who starts with what.Rounds from simple to tricky, with various challenges.Results board.Discussion, why one succeeded and another didn't; what stacks, prompts, configurations we have. Technical requirements: Linux or macOS. It's good if you have your own tokens for the agent; if not, please inform before the event. For whom: for everyone who codes with AI and wants to see how different setups perform on real tasks. Have an idea for a tricky task? We'll be able to run and test it if we have time.