Rep. Kevin McCarthy, R-Calif., has become the first Speaker of the House to be removed through a parliamentary rule called a "privileged motion."
The House voted 216-210 to strip him of his power.
Eight Republicans, including Rep. Matt Rosendale, R-Mont., joined House Democrats to oust the speaker.
"As for all the pundits out there that say that this is a bad time to do this, and we shouldn't be taking this action right now, I will tell you, I'm tired of being lectured by people that have been here for 20 years, that have led us to $33 trillion of national debt," said Rosendale, who represents Montana's eastern district. "I think we better start doing things a little different."
Rep. Matt Gaetz, R-Fla., filed the motion to remove McCarthy after the speaker turned to Democrats to help pass a short-term spending bill to keep the government open.
The two talked about the decision shortly after the vote came down.
"I don't regret standing up for choosing governing over grievance," McCarthy said. "It is my responsibility. It is my job."
"The one thing that the White House, House Democrats and many of us on the conservative side of the Republican caucus would argue is that the thing we have in common, Kevin McCarthy, said something to all of us at one point or another that he didn't really mean and never intended to live up to," Gaetz said.
The decision to vacate the chair means the House will elect a new Speaker of the House at time when the government is funded until Nov. 17.
McCarthy has announced he will not run again for the speaker's job.
"It's devastating for a conference," said Rep. Nicole Malliotakis, R- N.Y., "I mean, 98% of the conference was with speaker McCarthy. He's done a good job. He did not deserve us. It's a complete injustice."
Congressman Rosendale says decisions on the speaker and spending need to happen now.
He fears the debt could reach $50 trillion in five years.
"We cannot maintain this trajectory," Rosendale said. "If we do, we're not talking about in the long distant future. But within the next five years, that 33 trillion will be $50 trillion. And $50 trillion of national debt will consume 50% of the revenue that the federal government collects annually. So if you think that we have to make difficult decisions now, in five years, they're going to be monumental."
He says his reasons for voting against former Speaker McCarthy come down to trust.
"Kevin McCarthy has violated the trust of the Republican conference many times over the last nine months and that's why he's no longer the Speaker of the House," Rosendale said.
Rosendale says in January he and some of the Republicans passed rules requiring 72 hours notice before voting on bills, asked for single subject bills and 12 individual bills to fund government.
"This isn't a big deviation or radical change," Rosendale said. "This was just a restoration of Congress."
He says the four appropriations bills passed last week illustrate that these rules were not implemented and McCarthy's talks with the President raised the debt ceiling by $4 trillion.
"We found out that he had also negotiated yet another deal with President Biden to tie in our border security with Ukraine funding," Rosendale said. "And basically, at that point, confidence was lost. The appropriation bills aren't being done. And that's when the motion to vacate was brought forward."
Rep. Ryan's Zinke, R-Mont, who represents Montana's western district, voted against declaring the office of speaker vacant and may issue a statement on Wednesday.
Rep. Harriet Hageman, R-Wyo., also voted against ousting McCarthy and sent a written statement:
"There are far too many pressing concerns facing this nation for Congress to be pulled into such a self-serving stunt as one member is selfishly trying to torpedo the business of the House. Federal spending is out of control, we have just begun an impeachment inquiry to get to the bottom of the Biden family influence peddling scheme, our southern border is in chaos as cartels smuggle across enough fentanyl to kill every American, and our economy is in tatters while we see families unable to afford medication, electricity, and groceries.
There’s nothing Democrats and the media like better than Republican-on-Republican infighting, and this distraction focuses all attention on that, rather than on the serious issues at hand. We have 43 days to pass the remaining 8 appropriations bills and I am focused on continuing to pass good legislation to secure our border, reduce our national debt, lower energy costs, and put parents back in charge of their children’s education."
According to Rosendale 90% of the budget could be in place if the Senate passed the bills transmitted from the House.
"Kevin McCarthy's not trustworthy," said Rep. Adam Schiff, D-Calif. "We don't trust him. His own conference doesn't trust him. We just lurch from crisis to crisis with him as the Speaker."
"Democrats save the day," said Rep. Ilhan Omar, D-Minn. "And (McCarthy) goes on TV hours later and says Republicans did it, Republicans were leading. That's insane."