(CNN) — Rep. Jason Chaffetz blasted the U.S. Secret Service for trying to smear him by leaking information about his prior job application to the agency.

“It’s a little bit scary. The Secret Service diving into my background as a sitting member of Congress?” said Chaffetz, a Utah Republican, to CNN’s Wolf Blitzer on The Situation Room. “It’s not about me, but it is about: What are they doing over there? These people are trusted with guns by the president for goodness’ sake.”

An investigation released Wednesday by the Department of Homeland Security watchdog found that a top official at the agency, Ed Lowery, encouraged the office in an email that, “some information that he might find embarrassing needs to get out. Just to be fair.”

A few days later, The Daily Beast reported that Chaffetz had been rejected by the agency in 2003 he oversees as a committee chairman.

Chaffetz serves as chairman of the House Oversight Committee, which is leading an investigation into the Secret Service’s misbehavior including intruders around the White House and drinking on the job.

In late March and early April, 45 different Secret Service employees looked at Chaffetz’ sensitive personal file, the report said. And only once did managers try to prevent the violations of department policy.

“This episode reflects extremely poor judgment and a lack of care on the part of a number of Secret Service employees,” Inspector General John Roth said in the report.

Chaffetz said both the directors of DHS and the Secret Service had called him to apologize.

“That ain’t good enough,” he said. “I worry that if they’re doing this to me, they’re doing it to who knows how many other people.”

The-CNN-Wire ™ & © 2015 Cable News Network, Inc., a Time Warner Company. All rights reserved. (PHOTO: CNN)