In the Project Swap experiment, 201 Anthropic employees sent Claude agents to trade books for them. After a five-minute chat the agent ordered pairs of books the way the person did in 61% of cases. The biggest loss was not in the haggling but in how well the agent had understood the person.
- 61% agreement, against 50% for random guessing, 53% for ranking by popularity and 55% for “people who liked X” recommendations.
- 85% of the gap to the best possible outcome came from imperfect understanding of taste, and 15% from the bargaining itself.
- People would give an agent about 30% of their yearly book budget, against about 40% to a well-read friend.
Five minutes of conversation, and the agent has to know which book you will like more than another. How many times out of ten will it get it right?
About six. It sounds low, but it beats popularity and beats the classic “people who liked this also liked that” recommendations.
The weak spot
It is not the bargaining. The agents bargained well. The weak spot is the input: what the agent knows about you before it goes off to buy.
That matters directly to anyone building such systems. You can give it the smartest negotiating strategy, but if the agent misunderstood what you want, it will bring you an excellent deal on the wrong thing. People who wrote more in the conversation were, on the whole, understood better.
And one detail that says a lot: those who saw that Claude's summary had missed something they said would give it 23% of their budget. Those who saw no gap - 34%.
Trust, it seems, depends also on whether the agent shows you it heard you.