MPs today defeated the Government on an amendment which could thwart efforts by the next prime minister to force through a no-deal Brexit.
The Commons voted by 315 votes to 274, majority 41, to approve an amendment from Labour's Leeds Central MP Hilary Benn which aims to strengthen measures designed to try to stop Parliament being suspended to allow the UK to crash out of the EU.
Margot James resigned as minister for digital after voting against the Government and was among 17 Tories who rebelled by backing the amendment.
Chancellor Philip Hammond, Justice Secretary David Gauke, Business Secretary Greg Clark and International Development Secretary Rory Stewart did not take part in the vote.
Theresa May was said to be "disappointed" that several ministers failed to support the Government.
A Downing Street spokesman said: "The Prime Minister is obviously disappointed that a number of ministers failed to vote in this afternoon's division and issued a warning to the MPs over whether their roles will still be available under her successor.
"No doubt her successor will take this into account when forming their government."
Ms James said fears that Boris Johnson may prorogue Parliament if he becomes the next prime minister were behind her decision to rebel.
She told BBC Two's Newsnight: "The fact that Boris Johnson, and he hasn't won yet by the way but obviously he may, is not ruling out proroguing parliament, I felt that this time that rather than just abstain I would vote for the amendments that will make it more difficult."
And she said she thought "quite a number of people" would quit the Government and do "everything they can with myself and others to make sure we leave with a deal" if Mr Johnson wins the contest.
Mr Hammond, who abstained, tweeted: "The Conservative Party has always, at its core, had a fundamental belief in the importance of strong institutions - and in a representative democracy there can be no more vital institution than it's (sic) Parliament.
"It should not be controversial to believe that Parliament be allowed to sit, and have a say, during a key period in our country's history."
Tory leadership contender Jeremy Hunt said he thought he was given permission to miss the vote, but was mistaken.
He tweeted: "I missed votes today because I thought I was slipped and it turns out I was not. Apologies to my colleagues & Whip. My position is that parliament should NOT restrict the hands of an incoming govt in this way & I remain opposed to how parl voted."
His rival Mr Johnson voted against the move to block prorogation.
The vote came after the Lords strongly backed a bid to block Parliament being suspended in order to facilitate a no-deal exit by a margin of 103 votes on Wednesday.
The change to the Northern Ireland (Executive Formation) Bill will require progress reports on restoring devolved government in Northern Ireland to be debated regularly in Parliament, effectively preventing it being prorogued.
Justice Secretary Mr Gauke said before the vote that proroguing Parliament would be "outrageous" - but he did not confirm whether he would back the measures to block it.
He told BBC Radio 4's Today programme: "At a crucial point in this country's history, if you like - that Parliament should not be able to sit, should not be able to express its opinion and its will, I think would be outrageous.
"I very much doubt that any prime minister would in fact suspend Parliament in these circumstances, but I can understand the concerns that a lot of my colleagues have."
Downing Street said forcing a report and debate every two weeks risks being "counter-productive" to the aim to restore the executive in Northern Ireland.
The Prime Minister's official spokesman told a Westminster briefing: "We have been very clear that the purpose of this Bill is to ensure the continuation of vital public services and effective governance for the people of Northern Ireland.
"In the light of the ongoing talks to restore the Executive, our view is that forcing a report and debate every two weeks on the progress of those talks risks being counter-productive to this overarching aim."
Front-runner for the Tory leadership Mr Johnson again refused at the final campaign hustings on Wednesday to rule out proroguing Parliament in order to meet his red line of getting the UK out of the EU by October 31.
Mr Hunt has insisted he would not use such a constitutional manoeuvre.
The amendment to the Northern Ireland (Executive Formation) Bill builds on an amendment passed in the Commons last week, when MPs backed a proposal from the pro-European Tory Dominic Grieve calling for fortnightly reports from the Government on the efforts to restore the powersharing executive.
The new Lords amendment would ensure these have to be debated in the weeks before the Brexit deadline.
The Bill could theoretically make it illegal for the Government to prorogue Parliament in the autumn if the powersharing executive in Northern Ireland has not been restored.