If you have completed a good IMU calibration and you keep getting this message then - yes.
The FC will only switch to the 2nd/backup or third IMU if it considers the data coming from the primary is suspect.
This would only happen in rare instances where there were wide enough differences between one and the other two that the algorithms consider there is a problem.
Having two IMU’s is great but obviously for true redundancy, three is required.