Modric, 33, won his third successive Champions League in May before helping Croatia to a first World Cup final, BBC Sport reported.
Former Brazil and AC Milan forward Kaka, in 2007, was the last player other than Messi or Ronaldo to win it.
Ronaldo, who joined Juventus from Real Madrid in the summer, came second.
Barcelona forward Messi finished fifth, with Atletico Madrid frontman Antoine Griezmann and PSG attacker Kylian Mbappe – both of whom won the World Cup with France – third and fourth respectively.
The Ballon d’Or is voted for by journalists and organized by French magazine France Football.
The Ballon d’Or was first presented in 1956 and partnered with soccer’s world governing body FIFA as ‘The Best’ award from 2010 to 2015, until becoming its own entity again.
Victory caps a stellar year for Modric, who also was named the world's top male player at the Best FIFA Football Awards and received the Golden Ball award for the best player at the World Cup.
Speaking to BBC Sport, Modric dedicated the award to all of the other players who have missed out in the 10 years dominated by Messi and Ronaldo.
"Maybe in the past there are some players who could have won the Ballon d'Or like Xavi, Andres Iniesta or [Wesley] Sneijder but people finally now are looking at someone else," he said.
"This award is for all the players who probably deserved to win it and didn't. It was a really special year for me.
"It is difficult to express my emotion and how I feel in words. It is something unique for me. It was very special."
Modric described Ronaldo and Messi as "phenomenal players", adding, "To win means I did something really special on the pitch this year, that's why 2017-18 was the year for me."
Modric was one of eight Real Madrid players in the top 30, including Wales international Gareth Bale in 17th.
Lyon and Norway striker Ada Hegerberg, 23, won the inaugural Women's Ballon d'Or while Mbappe, 19, became the first winner of the Kopa Trophy – awarded to the best under-21 player and selected by former Ballon d'Or winners.