Never driven the S5, but I have the A5 3.0TDi, and it was very fast and very easy to go fast. Its almost as quick as an S5, and a re-map would sort that out.
I love the look of the S5 though, a stunning car.
If the fuel economy of my old RS4 is anything to go by, then it will be poor on fuel!, but if you are only using it as a second car then not an issue.