Annoying isn't it. Did you feel any kind of torque steer from that? I was surprised by how violent the torque steer was on mine from the insert popping out, and mine's 100lbft less than yours!
The upper insert started walking out on mine as well, hence tying that down with the bolts also. But not inserted too far otherwise the pendulum arm will contact the bolts when accelerating. I don't think any kind of glue or silicon will be man enough to keep them in place unfortunately.