worm in the apple

noun1

1. •
   A serious flaw in something otherwise good; a problem that ruins everything else.

   • The worms in the apple, of course, are Baby Boom liberals who latched on to the one institution they could control—the university. — 2007