(CNN) — Paul Ryan initially didn’t want to be speaker, but he may now have to fight to keep his job.

A number of members of the conservative House Freedom Caucus — a group of roughly 40 members who have been a persistent thorn in the GOP leadership’s side — are refusing to say if they will back the Wisconsin Republican’s bid to keep his gavel if their party maintains the House majority.

The chairman of the group, Ohio Rep. Jim Jordan, pointedly declined to say in a nationally televised interview this week if he’d back Ryan. Several of the group’s top members, including Reps. Raul Labrador of Idaho, Mick Mulvaney of South Carolina and Mark Meadows of North Carolina, refuse to say if they’d like Ryan to return.

“I’m not commenting on leadership elections until after November 8,” Meadows told CNN Friday.

And others, including Rep. Dave Brat of Virginia, are already laying out a list of conditions for Ryan to meet before offering their endorsement.

“If he commits to regular order — in writing — and no Trans-Pacific Partnership until we reduce regulations and no amnesty bills and a commitment to reduce federal spending, then I will vote for any candidate who backs these conservative positions that 80 percent of Republicans share,” Brat, who voted against Ryan last year, told CNN Friday. “Each of these issues should just be reflexive for any Republican by the way; they should not take more than five seconds to say ‘yes.'”

Such demands indicate the challenges ahead for Ryan — if he wants to keep his job. Speculation has grown that Ryan may quit the speakership, but the Wisconsin Republican and his aides firmly dispute that notion.

Some conservatives in the group do back the speaker. Ryan has campaigned recently for two members of the Freedom Caucus — Iowa Rep. Rod Blum and New Jersey Rep. Scott Garrett. Both have praised the speaker and indicated they continue to support him.

If the GOP keeps its majority, it will be smaller and more conservative, given that a number of more moderate members could lose Tuesday. That would give more power to the right-wing of his conference.

To be reelected speaker with a smaller House GOP majority, Ryan will need the backing of most of those conservative members to reach the 218 votes he needs on the floor to win reelection. Along with Brat, nine other members voted against Ryan when he became speaker last year.

But other conservatives who backed Ryan last time, including Rep. Jim Bridenstine of Oklahoma, are refusing to commit to voting for the speaker now because of Ryan’s recent comments that he wouldn’t defend — or campaign with — Donald Trump.

“He should get out and campaign for him to win Wisconsin,” Bridenstine told CNN earlier this week.

On Saturday, Trump’s running mate — Mike Pence — will appear in Wisconsin at the same event as Ryan, as the men join Republican Sen. Ron Johnson who is battling to keep his seat. It will be the first time Ryan has appeared with anyone on the Trump ticket, since the speaker and the GOP nominee had a major falling out after the tape of Trump’s vulgar comments about women were leaked to the press.

Some House conservatives in the Freedom Caucus took unusual step this week of taking time off the campaign trail and returning to Washington days before the election to meet to discuss how they could flex their muscle in upcoming leadership election. In the meeting, a handful of members discussed changes they wanted Ryan to make, including giving rank-and-file members more power to select committee chairman and adding a new conservative to Ryan’s leadership team, sources said. Some of these conservatives are holding off saying how they will vote as a way to leverage their support to get some type of rules change they believe will boost their power inside the GOP conference.

Allies of Ryan noted to CNN that these members have done little to stump for fellow Republicans in tough races and pointed to the speaker’s extensive fundraising and campaigning tour this fall.

Will Ryan run?

But the more immediate question for Ryan is if he actually wants the job to begin with. Some of his allies believe that Ryan won’t want to negotiate with the hard-right faction to keep his job — or constantly run the House in perpetual fear that an agitated member could file “a vacate-the-chair” resolution that would prompt a no-confidence vote in the speaker.

Oklahoma Rep. Tom Cole, a supporter of Ryan’s, told CNN that the speaker is the “odds-on favorite as long as he runs again.”

“I don’t know he’s made that decision,” Cole said.

Ryan sought to put any speculation to rest that he might step aside in a radio interview on Friday with host Jerry Bader.

He brushed off reports suggesting he wouldn’t run as “the typical chatter you have every two years. They call it palace intrigue in the hill rags” and said “I am going to seek staying on as speaker.”

Asked about some conservatives not committing to backing the speaker, Ryan’s spokeswoman AshLee Strong told CNN, “the speaker’s only focus until Election Day is defeating Democrats and protecting our majority, and nothing else.”

Ryan has been barnstorming the country for House candidates — traveling to more than 20 states and 50 cities with more than 75 events since October, his office says. And such moves are bound to build goodwill with his caucus, especially if the GOP hangs onto the majority.

One senior House GOP leadership aide brushed off the notion that Ryan would walk away after his intense focus on maintaining the majority and putting together a detailed policy agenda to pursue next year.

“Paul is not going to criss-cross the country campaigning for members and spending all this time just to say ‘oh, I’m not going to run,'” the source told CNN.

Scenarios facing Ryan

There are different scenarios for the speaker after the elections. If the GOP loses seats, but keeps the majority, the more moderate GOP members will be gone, and the demands from the right will increase.

While some on the right could announce a challenge to Ryan, the House Freedom Caucus is very disorganized and previous efforts by its members to challenge leaders have failed. Leadership races are very personal, and members take advantage of the secret ballot rules and sometimes pledge support to more than one candidate.

But also if Ryan weighs a potential 2020 run, he could presumably decide being speaker is not best suited to meet his political ambitions.

Yet if the GOP loses the House, Ryan could run for House minority leader — or decide to step aside. The post-election turmoil for the heart and soul of the GOP is bound to play out in the leadership elections, and finger pointing over Trump is bound to get nasty.

But one of the worst scenarios facing Ryan is if he wins the House GOP’s vote to be renominated as speaker in November, but loses the vote on the full House floor in January to get the 218 votes needed to keep his gavel.