In our case I'm certain it's a host issue. Or more specifically, being the host is exacerbating the issue. But we're not on a server, one of us is creating the game and the other joining it. We have to restart whenever the host starts to get too visually bugged. In your case, since nobody's machine is doing double duty as host... maybe it's just taking longer to happen? On our end the client may not be playing long enough to see the phantom blocks by the time the host is. In which case, if we go back to it being purely a memory leak... then hosting a server seems to exponentially increase said leak. Which would also mean that changing all the video options in the world won't help (and that seems to be the case).
Sadness. But it's the price of experimental I suppose.