Well, remember a few things: One, you just got out of jail, so they already don't like you.
But more importantly, two, you're breaking into their homes stealing their ♥♥♥♥. Peasants inside houses are the ONLY "hostiles" outside of bandits (they have knives) and guards at two prefabs (their base). Beyond that, they'll ignore you or run away.
And, out of the ones in the houses, only a percent of THOSE will fight you.