The brown marks are probably iron fallout, all cars have some but its very obvious on white, buy some ironx and use that on the paint to sort that out.
The white spots sound like they might be flat spots where the aggressive sponge has dulled the surface maybe? You got any photos?