i am no automotive engineer, but someone explained it to me in a way that seemed believable. if someone has a better/more correct explanation i genuinely would love to hear it.
bhp is the outcome of the amount of torque an engine produces mutliplied by the number of revs the engine is producing and some mathematical constant. this is why even though petrol cars produce their max torque low down, they tend to produce their peak bhp higher up the rev range. the higher revs mean the bhp is still higher.
diesel is more explosive than petrol, and gram for gram you get more energy from combusting diesel than petrol. this means that the engine components need to be stronger (i.e. heavier) and this means they can't rev as much. this means that despite being able to produce more torque than a like for like petrol, they never rev high enough to be able to match their bhp.
tbh i think nowadays if you're planning on buying a new car you really need to sit down and work out whether diesel is the better way to go. before i returned to uni i was looking at replacing my A3 with an A5 (either the 2.0T quattro or the 3.0TDI quattro) and i worked out that even though they have roughly the same performance, i would have had to drive the derv about 40k miles before the higher initial asking price and the higher price of diesel at the pumps was offset by the better fuel mileage of the diesel car. that combined with the fact that the 2.0T is lighter (and would hence more likely be more fun to drive) meant that i felt it was the better option. just a shame life got in the way!