bitter pill to swallow

noun1

1. •
   (figurative) Something unpleasant that must be accepted or endured.

   • [T]o see himself dethroned, the object of her contempt, was a bitter pill to swallow. — 1886