We don't need to just look at the scale of the new modelview matrix, but at the one we get when multiplying the new one with the current one. Test case attached.
