go quietly

verb1

1. •
   To leave one's current location or position without protest or kicking up a fuss.

   • "Look you, Mynheer Poots," exclaimed Philip, red with passion; "you have but to choose, — will you go quietly, or must I take you there? You'll not trifle with me." — 1874