I take it this drift isn't in P-GPS mode? That would be odd. So I assume it is in Atti. You make an interesting point with regards to the FC being possibly slightly out of true when landing gear is raised.
Like you, I have designed and built many multi rotors from scratch and I have never been an 'IMU calibration junkie'. I prefer to get it right and then leave well alone unless mod values are out ocalibrated message presents itself.
I have not experienced any drift and have only calibrated with gear down and then only twice.....once the day I unboxed the Inspire and once more when I upgraded the FW to .17.
With regard to your drift problem, could you maybe shim the Inspire's legs in the opposite direction when you do the calibration. Only by a fraction (a few sheets of paper etc). The first time you do this will be a bit of trial an error until you get the shim value correct but then for future calibrations you will know what legs to shim and by how much?