The man to replace him, Ibrahim al-Assaf, was no less of a surprise because he was among dozens of prominent Saudi royals and businessmen detained in Riyadh's Ritz-Carlton in November 2017.
Jubeir will be demoted to minister of state for foreign affairs, the foreign ministry said on Thursday citing a royal decree.
A royal decree was also issued to restructure the Political and Security Affairs Council, headed by Crown Prince Prince Mohammed bin Salman, which has seen Saudi Arabia embroiled in its worst scandal in decades.
Bin Salman has been widely associated with the October 2 killing of dissident journalist Jamal Khashoggi after entering the Saudi consulate in Istanbul.
The death sparked international outcry as it drew attention to the West's close ties with the kingdom whose destructive war on Yemen is a matter of constant criticism.
Various reports have verified that some members of a 15-member hit team, which traveled to Istanbul to carry out the killing, had been drawn from among bin Salman’s personal bodyguards.
Western intelligence agencies and Turkey have suggested that bin Salman bore ultimate responsibility for the assassination of Khashoggi.
However, the reshuffle announced on Thursday did not touch bin Salman portfolios, with analysts saying Jubeir has been used as a scapegoat in the scandal.
The decision, they said, did not surprise them because they had been expecting Jubeir to be sacked for some time, even before the Khashoggi affair.
"But I now think he's been used as another scapegoat in this issue," Marwan Kabalan, head of policy analysis at the Arab Center for Research and Policy Studies, told Doha-based Aljazeera broadcaster.
Jubeir, he said, was seen as a leftover from the former monarch King Abdullah who died in January 2015, prompting a widespread purge in the kingdom.
"I think he is out now - perhaps at the right time for Crown Prince Mohammed bin Salman. As I said, the crown prince wanted to use somebody as a scapegoat and also hold him responsible," Kabalan said.
Khashoggi was murdered at the Saudi consul general in Istanbul, which is administered by the foreign ministry.
King Salman also appointed Prince Abdullah bin Bandar bin Abdulaziz to replace Prince Khaled bin Ayyaf as chief of the National Guard.
General Khalid bin Qirar al-Harbi was named general security chief, while Musaed al-Aiban was named national security adviser.
Turki al-Sheikh, head of Saudi Arabia’s sports committee and a close adviser to Crown Prince Mohammed, was replaced and named as new chief of the Entertainment Commission.
Turki al-Shabana, an executive at broadcaster Rotana, was appointed the minister of information. Rotana is owned by billionaire investor Prince Alwaleed bin Talal, who was also detained at the Ritz last year and later released.
King Salman further ordered the creation of the Saudi Space Agency, to be chaired by his 62-year-old son, Prince Sultan bin Salman. Prince Sultan had been head of the Saudi Commission for Tourism and National Heritage (SCTH).