It despawned while were playing. Also, we only summoned a few, less than 10 on mine and only 2 with my friend. After an hour or two, after doing a quest, when we came back, everything was lost. Summoned it again, came back a few minutes later, without doing a quest, all pets and guards despawned.