I get that. Is the gimbal also drifting when airborne? If the answer is yes, and you have done everything else, what is there to loose by doing the IMU cal? After all the gimbal is communicating with IMU all the time, even when the drone is on the ground. If IMU's gyro or accelerometer is sending incorrect info to the gimbal, it could cause the gimbal to drift. This is a close loop system and feed of incorrect info can cause all sort of issues, gimbal drift including.
Not saying IMU cal will fix your issue but since nothing else has, there is nothing to loose by doing the IMU cal, I would have thought