Personally, if you are going to do it, do it properly
The rear silencer is slightly restrictive, plus to have a 3" downpipe and center youd need a special adapter to reduce it down to 2.5/2.75 (i cant remember which it is atm lol), and as the factory silencer has the rear silencer and center silencer as one piece, youd have to cut the silencer out and join it up.
The main bottleneck is the factory downpipe as its smaller and has a precat which impedes exhaust gas flow, so yes there will be big benefits just from that.