I have this problem, which I noticed the first time I attempted to fly indoors. My Inspire drifted to its left side. I thought it was an air current, until I rotated the craft 180 degrees, and it still drifted to its left, although now it was drifting towards the opposite wall.
I believe my drift is due to a misalignment of my Inspire's legs. Here is a photo that shows the front right rotor:
My cell phone is directly in front of the Inspire, in line with the right front (shown) and right rear legs. The tilt is distinctly in, with the top of the leg closer to the body than the bottom. The other leg is tilted out, with the top farther away from the body than the bottom:
So either my right side is incorrectly tilted in, or my left side is incorrectly tilted out. Either way, the thrust goes down and to the right, and it drifts to the left.
There is nothing at all loose on this craft; this is how it was shipped to me.
I never noticed this because I always fly in GPS mode, and it always compensates. But this explains why I find it so hard to fly in a straight line. It looks as though I could loosen the screws in the plastic sleeve, rotate that arm, then tighten them. But I'm not sure what it should look like.