They are called special infected and their rarity and scarcity has nothing to do with their status as special infected. It is purely whether they have an ability or status effect that varies from the basic zombies.
Believing that scarcity is what defines “special” is purely your own personal take. TFP defines special infected by abilities and stats and the frequency of the appearance of ANY entity is purely a function of gamestage. Cops and spiders are super rare on day 1 during the day outside of downtown areas. But nighttime will increase the chance of sighting a spider and downtown areas increase your gamestage so again more frequency of seeing special infected.
This will be true of the bandits and any tiers or specials within that mob group that they develop.