Should skips really get reward 1?

[ctjlewis]

For #158, to handle situations where math_verify throws an error, we just skip it with existing behavior.

But what is the reasoning behind awarding reward 1, the same as a correct answer? I'd assume we just return None and actually skip it.

[Some-random]

Agreed. I also don't think we should give an reward of 1 when math_verify fails to work

[ctjlewis]

cc @qgallouedec

[qgallouedec]

It doesn't really matter, the rewards are normalized anyway.

[qgallouedec]

The only problem is logging. If it turns out that half of your dataset has a non-parsable solution, then you'll always have a logged reward of at least 50%, which is misleading. But in terms of optimization, it doesn't make any difference, because rewards are normalized per group.

Ideally, we'd like to be able to remove non-parsable solution samples, or more generally, to return None when the reward function doesn't apply to the sample for whatever reason, but this requires some changes in the way the advantage is counted on the TRL side. We could add this feature in the future, but for now, it's an edge case. Who works with a 50% non parsable dataset?

[ctjlewis]

OK, if it fails once it fails for the whole group, so this is not relevant, closing.