I don't think we'd ever end up joining a bandit clan. I honestly hope not and I can't even imagine how that'd fit into existing mechanics. But in the end, the final point is the right one. Yeah, I don't think traders are going to give you a quest to go through a POI with nothing but a shiv, but traders aren't the only ones in the wasteland and there are definitely some eccentric folks there. There's room for some more challenging/varying quests. Just gotta think outside of the box of 'Quests come from traders'.