Because they are doing it to compute quality metrics not to implement RLHF. It’s not training data.

Every decision they take based on evals influences the model.

/"directly"/

> For this to be true they would have to throw away labeled training data.

That's how validation works.

Is there a reason not to use validation data in your next round of training data? Or is it more efficient to reuse validation and instead get more training data?

You'd have to recreate your validation if you trained your model on it every iteration and then they wouldn't be consistent enough to show any trends

I'd have thought that if you kept the same validation you'd risk over fitting.

Clearly that does make it hard to measure. I'd think you'd want "equivalent" validation (like changing the SATs every year), though I imagine that's not really a meaningful concept.

Key word: "directly"

It does so indirectly, so it's a true albeit misleading statement.

It's not part of the inner feedback loop. It's part of the outer feedback loop that they use to decide if the inner loop is working.

They probably don’t do it at a scale large enough to do RLHF with it, but it’s still useful feedback the people working on the projects / products.

More recent models actually use "reinforcement learning from AI feedback", where the task of assigning a reward is essentially fed back into the model itself. Human feedback is then only used to ground the training, on selected examples (potentially even entirely artificial ones) where the AI is most highly uncertain about what feedback should be given.