low on the totem pole

adjective1

1. •
   (offensive) Accorded relatively little importance.

   • Gaston said, however, that the boycott would not substantially affect Negroes because they are "so low on the totem pole." — 1965