Go on the ciruit and do some sessions without a passenger and then with and you will be amazed at the difference, not just speed, the biggest difference is in braking and cornering, you can carry more speed round a corner when lighter. However, on the road you won't really notice much difference (at least I hope) as you won't be driving near the edge of the envelope.
This is why the elise is so effective, in the words of the late grate colin chapman for more performance just add lightness.