hold against

verb1

1. •
   To think less of (someone) for (something they have done).

   • He was awful to me when we were young, but I don't hold that against him. We were children then.