Think of it this way. The map was created with every object/material instanced. Once you remove that object instance, it has to insert a value to indicate a change from the OG map creation was made. The more changes you make.
A POI might have a very small value if it is a preset referenced elsewhere, but the moment you destroy part of that POI, the reference has to be modified. Ie the reference gets larger to account for the changes. The more references it has to log to ensure all players experience the same game world, the larger it will become.