(CNN) Conservatives in the House are saying not so fast to the idea of Speaker Paul Ryan.

On Tuesday night, the Wisconsin Republican laid out his vision to his GOP colleagues and said he would be willing to serve as speaker — if the whole conference could unite behind him.

But Wednesday morning, many of the conservatives that consistently fought outgoing House Speaker John Boehner and helped scuttle the bid to replace him by Majority Leader Kevin McCarthy signaled they aren’t ready to jump on the bandwagon for Ryan.

“I thought we did the coronation last night,” Texas Rep. Louie Gohmert cracked Wednesday morning when asked if he would support Ryan.

Gohmert had been backing Florida Rep. Daniel Webster and said he was still supporting Webster. And while Gohmert was never likely to be a vote Ryan would pick up, having voted for himself instead of Boehner in past speaker elections, key members of the conservative House Freedom Caucus were also not rushing to endorse Ryan.

Boehner has scheduled an election for Republicans to pick their nominee to be his successor on October 28, with the full House to then vote on the next speaker the next day. Ryan is expected to meet with the Freedom Caucus on Wednesday and sit down with the other caucuses from whom he seeks support, as well, including the conservative Republican Study Committee and the moderate Tuesday Group.

Ryan’s conditions includes rules changes that would make it more difficult to overthrow a sitting speaker – a provision that Rep. Raul Labrador, R-Idaho, a leader of the Freedom Caucus, said is a “non-starter.”

“We have to remember everything in the House and everything in Congress is about checks and balances,” Labrador said. “We have two houses of Congress for that reason. We have three branches of government for that reason, and one of the reasons that every board has the freedom to vacate (the chair).”
And fellow Freedom Caucus member Rep. Mo Brooks said in addition to that concern, he takes issue with Ryan’s track record on immigration. “Paul Ryan’s support for amnesty and open borders, that is a significant factor,” the Alabama Republican said.

In still another line of attack, caucus member Rep. Tim Huelskamp, R-Kansas, said Ryan’s insistence that he spend time with his family on weekends and not carry on the grueling record of fundraising established by Boehner on his own shoulders is a major problem.

“You can’t do the job 9-5, Monday through Friday,” Huelskamp said, noting Boehner’s raising of more than $50 million for the party. “The speaker has to work on weekends. I’m not running for speaker, I’ve got young kids at home. … The time commitment is not 40 hours a week.”

Freedom Caucus previously endorsed Webster
The Freedom Caucus endorsed Webster in the race before McCarthy’s dramatic announcement he was dropping out and Ryan’s subsequent decision. Caucus rules state an endorsement requires at least 80%, and so it would take another four-fifths vote to rescind that endorsement and transfer it to Ryan.

And there’s no guarantee an endorsement means all members would vote the same way, Brooks noted, adding he had always supported Rep. Jason Chaffetz, R-Utah, for the spot.

Huelskamp said he was still supporting Webster, but said he was open to hearing from Ryan.

And many of the caucus members did the same.

“On the positive side. Paul Ryan is a very charismatic individual, he’s an eloquent spokesman, he’s a good fundraiser, all of those are positive factors. So you have to weigh all of these factors,” Brooks said.

“This place has to change. It can’t be business as usual, status quo. There are a number of candidates, including Paul, who understand we need to change the way this place operates,” caucus Chairman Jim Jordan said. “We just want to talk to him about what those specific changes are.”

Boehner says Ryan will win
Boehner says he believes Ryan will win over his colleagues.

“Listen, I think Paul is going to get the support that he is looking for,” Boehner told reporters Wednesday morning. “I thought he laid out a very clear vision of how he would run the speakership. And I thought the members responded very well to him.”

He added: “We all know Paul Ryan, right? He’s a very good member. He works hard. He’s very bright. and he has good relationships, I think, with all the wings of the party. That’s why I think he’ll be doing fine.”

Many Republicans fear that if Ryan doesn’t earn the conservatives’ support, the party will be in total disarray.

“If Paul Ryan can’t get 218, no one can,” Florida Rep. Carlos Curbelo told reporters.

The-CNN-Wire ™ & © 2015 Cable News Network, Inc., a Time Warner Company. All rights reserved. (PHOTO: CNN)