WASHINGTON — (CNN) Ben Carson declined an offer from President-elect Donald Trump to join his cabinet as Secretary of Health and Human Services, a close Carson adviser told CNN on Tuesday.

Carson confidant Armstrong Williams said the former neurosurgeon and top Trump surrogate will continue to advise Trump from outside the White House but declined Trump’s offer to join his administration as a cabinet-level official.

“He’s never run an agency and it’s a lot to ask. He’s a neophyte and that’s not his strength,” Williams said, despite the fact that Carson vied for the Republican nomination to be the next president of the United States.

Carson had been rumored as a potential pick to head up the Department of Health and Human Services or the Department of Education.

The Trump transition team did not respond to a request for comment.

Williams told The Hill newspaper, “Dr. Carson feels he has no government experience, he’s never run a federal agency. The last thing he would want to do was take a position that could cripple the presidency.”

Carson told The Washington Post that he is “leaning” toward working “from the outside and not from the inside.”

“I want to have the freedom to work on many issues and not be pigeonholed into one particular area,” he told the newspaper.

Carson declined to tell the Post whether he was offered a specific post, but said “I was offered things. … I don’t want to go into the details.”

Carson was a key adviser and top surrogate during Trump’s presidential campaign, joining him at numerous campaign rallies and helping Trump with his outreach efforts in the African-American community.