“This wasn’t frivolous,” the former secretary of defense who’s served under eight presidents says