Why would you use RTV sealant?
You can't do this with RTV.. it's not about sealing it, it's about keeping noise from escaping the vehicle and removal/prevention of rattles. Unless I'm mis understanding how you are doing it.

I have only done the inside of my door (removed the door cards) as that's where most of the rattles come from. When I go to upgrade the speakers I may take the door skins off but it's a lot of effort what I can't see to be any gain.