The stark warning from the Institute for Government comes as May held her first summit Monday with the leaders of Scotland, Wales and Northern Ireland since the June 23 referendum to leave the EU.
In a report released on Monday, the London-based advisory group warned that if the four leaders fail to agree on the “core planks” of the UK’s negotiating position before the prime minister invokes Article 50, they can expect “a serious breakdown in relations between the four governments and nations of the UK.”
Article 50 stipulates that any member state may leave the EU “in accordance with its own constitutional requirements.”
Scotland and Northern Ireland voted against Brexit, and politicians from both nations have expressed concerns about leaving the bloc.
"There is little common ground between these leaders on the future of the UK or almost anything else," said Akash Paun, a fellow at the institute.
While May has no constitutional duty to consult the first ministers, it would be a "reckless strategy" to impose a Brexit deal on the other nations, the report says.
Scottish First Minister Nicola Sturgeon and Welsh leader Carwyn Jones were expected to tell Prime Minister May that their national parliaments should have a say on the proposed deal.
The meeting came as British Bankers’ Association has warned that several banks were poised to withdraw from the UK over fears of a “hard Brexit.”
Under the “hard Brexit” formula, the UK may lose its preferential access to the EU's single market and customs union.
Sturgeon has said Scotland and the other two devolved administrations were in favor of the UK remaining in the single market. She has promised to hold a second independence referendum unless Britain remains in the single market.
May warned Monday that Sturgeon was "undermining" Brexit negotiations, saying the “great union” has been a “cornerstone of our prosperity in the past” and will be “absolutely vital to our success in the future.”
"Working together, the nations of the United Kingdom will make a success of leaving the European Union – and we will further strengthen our own unique and enduring union as we do so,” she added.
Sturgeon said the Downing Street meeting left her “deeply frustrated” and no clearer about the UK government's thinking than before.
“We discussed the UK's negotiating position in general, but it is safe to say we got no more information or detail on that than we had before we went into the meeting, and I got the strong sense the UK government itself doesn't know what it is trying to achieve,” she told BBC Scotland.
“That is why many parts of the meeting were deeply frustrating, because we felt as if we weren't getting any greater insight into the thinking of the UK government,” the first minister noted.