I don't think it's that questionable. Depends on how you want to interpret "the office of the President." Does that mean the President specifically, or does that mean President, VP, his white house staff, and his cabinet members?
So you're saying a 2-term President could never be Secretary of...