Source item

I think there is something important going on here that is worth figuring out: 1) most models overfit to evals. that said, we need some evals to tell a model if they are good bullshit detectors before we go off and burn a bunch of money on returning nonsense 2) there needs to be an awareness of sycophancy and generally being agreeable isn't always the best thing 3) i think using a mixture of llms and humans to judge would be valuable