Early boarding/de-planing aircraft

Why is it that those passemgers who need to board the aircraft early because they need more time also de-plane early and hold up the rest of the passenger who need to make connecting flights, etc? How you figguh?