Tbh, the part where some zeds surround a trader in which we are doing something and they can climb on each other to get our asses is pretty entertaining. I used one trader, to clean a small military spot between my base and the trader (with T2 melee weapons, and a pipe pistol, day 4 on warrior difficulty). Few of them hopped in the trader compound and there was a lot of jumping and fighting, hiding for stamina also.
It is a nice touch, as Roland said it would really be cool to have these places being destroyable, but not till the point where Trader Someone stays in the middle of some town or city with only his desk and doing its stuff, surrounded by the ruins of his lifetime build spot. Could be an option in the menu, too.
Or make them fully destroyable with notification that some horde event is heading on their way, to keep them alive and be able to get to late game stages and to The Duke. Or if unable to do it, a repair mission, to keep them surviving, cuz we would need them for stuff.