The outgoing secretary of state, who returned before dawn from a trip to Africa, did not speak to the president before his sacking was announced and was unaware of the reason for his sudden downfall, according to a top aide, AFP reported.
A senior White House official said the president wanted to reshuffle his team with a view to launching talks with North Korea, following last week's spectacular announcement of a meeting between Trump and Kim Jong-un.
Trump had scant words of praise for Tillerson, who had long been rumored to be about to be pushed out.
"A great deal has been accomplished over the last fourteen months, and I wish him and his family well," Trump said, thanking Tillerson "for his service."
But addressing reporters before leaving on a trip to California, Trump spoke openly of his disagreements with the former Exxon chief – including over the Iran nuclear deal – as he explained his decision to replace him.
"We got along actually quite well but we disagreed on things," Trump said. "When you look at the Iran deal, I thought it was terrible, he thought it was okay. I wanted to either break it or do something, he felt a little differently. So we were not really thinking the same."
In announcing Tillerson's sacking earlier Tuesday, Trump lavished praise on Pompeo, a former US army officer and congressman who led the CIA for nearly 14 months, saying he would do a "fantastic job!"
"He will continue our program of restoring America's standing in the world, strengthening our alliances, confronting our adversaries, and seeking the denuclearization of the Korean Peninsula," Trump added, calling him "the right person for the job at this critical juncture."
Trump called for Pompeo to be swiftly confirmed in the role – as his administration heads into the high-stakes talks.
To succeed Pompeo, the US president nominated Gina Haspel, a controversial career intelligence officer, to head the Central Intelligence Agency – the first woman tapped for the post.
Haspel has been reported to have overseen a CIA "black site" in Thailand where Al-Qaeda suspect Abu Zubaydah was waterboarded after the 9/11 attacks.
'Every intention fires remaining'
Tillerson has had a rocky tenure as the top US diplomat.
He was repeatedly forced to deny he had fallen out with Trump – vowing to remain in the post despite a sensational report that he once dubbed the president a "moron."
A respected figure in the oil business, the 65-year-old Texan's tenure at the State Department drew scorn from Trump's opponents, from former diplomats and from the Washington policy elite.
Tillerson had no diplomatic or political experience before becoming secretary of state.
During his time in the post, he was faced with an extraordinary array of foreign policy challenges, from North Korean nuclear threats to attacks on US diplomats in Cuba.
But his efforts were often overshadowed by Trump's un-diplomatic style and his streams of taunting tweets stirring international tensions.
Tillerson was thousands of miles away on a tour of African countries when Trump made the snap decision to meet Kim, and had suspended his schedule on grounds he was "unwell" before cutting short his trip.
Undersecretary of State Steve Goldstein, in a statement, made clear that Tillerson was caught off guard by his ouster.
"The secretary had every intention of remaining because of the tangible progress made on critical national security issues. He established and enjoyed relationships with his counterparts," Goldstein said.