Evaluate probability estimation of agents

[kongzii]

The following experiment for each agent:

1. Bucketize all bets by probability, e.g. 0-10% bucket, 10-20% bucket , etc.
2. In each bucket, get the real probability of markets in them: len(markets resolved as yes in this bucket) / len(all markets in this bucket)
3. Optionally, consider only bets with confidence >=

The real probability should be within the bucket. We can use that to either calculate "probability accuracy" or MSE between the bucket and real probability.

It would be nice to have it on Dune, but it should be fast to implement in this script so probably best to get the results right away: https://github.com/gnosis/prediction-market-agent-tooling/blob/main/examples/monitor/match_bets_with_langfuse_traces.py

[gabrielfior]

Possibly a duplicate of gnosis/prediction-market-agent#477

[kongzii]

Possibly a duplicate of gnosis/prediction-market-agent#477

Right! I was looking for "probability" in issue titles. Will close this.