Andy, mine was really stiff as well and like you had the feeling i was going to break it. You will need to drop the door card (you don't need remove completely, just support it so you can get behind it)
You need to push from the top and pull from the bottom at the same time, also you need pry away from the sides all at the same time
From memory it's not a straight pull/push, you need to push switch down and back and then roll forward, if that makes sense HTH