wrong side of the tracks

noun1

1. •
   The part of town that is not inhabited by the wealthy. An area where the working class, poor or extremely poor live.

   • He grew up on the wrong side of the tracks, but he made a success of himself.