The place I was on about, was on the same road as the market, but a bit further down. You'd come out of the main drag of the shopping centre, and find a market across the road, anyway, turn right instead of crossing the road, and it was about 50 yards down there.
From what I remember, that shop later moved up to another place, that was situated in a big triangle, just off the side of what was then, the main shopping mall.
I went back there a couple of years ago. The place had changed pretty much beyond my recognition.
I used to go to a car boot sale, half indoors, half outdoors, heading on the Highworth road out of Swindon, and that's where I picked up Kate Bush's Sensual World album on CD a good three months before it was officially released, so yeah, I'm pretty sure a fair amount of stuff found its way "unofficially" out of EMI Swindon