A "yes" vote would have deprived Paris of a foothold in the Indo-Pacific region where China is expanding its presence, and dented the pride of a former colonial power whose reach once spanned the Caribbean, sub-Saharan Africa and the Pacific Ocean.
Based on provisional results and with a participation rate of nearly 80 percent, the "No" vote stood at 56.9 percent around 1300 GMT, local TV station NC La 1ere reported on its website, Presstv Reported.
"The New Caledonians have chosen to remain French...It is a vote of confidence in the French republic, its future and its values," President Emmanuel Macron said in a speech on French television.
The referendum was the first auto-determination vote to be held in a French territory since Djibouti in the Horn of Africa voted for independence in 1977.
Voters in the largely self-governing territory had been asked the question, "Do you want New Caledonia to gain full sovereignty and become independent?"
Macron said he understood the disappointment of those who wanted independence, but added that the French state would ensure liberty, equality and fraternity for everyone.
"The only loser is the temptation of contempt, division, violence and fear; the only winner is the process of peace and the spirit of dialogue," Macron said.
Tensions have long run deep between pro-independence indigenous Kanaks and descendants of colonial settlers who remain loyal to Paris.
Over the past decade, relations between the two groups have improved markedly, but the "no" vote outcome was well below some early polls, which could encourage nationalists to try for a new referendum in coming years.
Some 175,000 out of the 280,000 people living on the archipelago were eligible to vote, with polls showing earlier in the week that the islands were expected to vote to remain a French territory.
Posters calling for a "no" vote said that "France is the only chance" while proponents of independence called in their posters to vote for "a multicultural, in solidarity, peaceful nation".
During a visit to the archipelago in May, Macron acknowledged the "pains of colonization" and saluted the "dignified" campaign for autonomy led by the Kanaks. He and his administration sought to strike a neutral tone on the vote.
New Caledonia's economy is underpinned by French annual subsidies of some 1.3 billion euros ($1.48 billion), nickel deposits that are estimated to represent 25 percent of the world's total, and tourism.
It enjoys a large degree of autonomy but depends heavily on France for matters such as defense and education.
First discovered by the British explorer James Cook, the New Caledonia archipelago lies more than 16,700 km (10,377 miles) from France.
It became a French colony in 1853. Under colonial rule the Kanaks were confined to reserves and excluded from much of the island's economy.
The first revolt erupted in 1878, not long after the discovery of large nickel deposits that are today exploited by French miner Eramet's subsidiary SLN.
More than a century later, in the mid-1980s, fighting broke out between supporters of independence and those who wanted to remain French, amid festering anger over poverty and poor job opportunities.
A 1988 massacre in a cave on the island of Ouvea left 19 indigenous separatists and two French soldiers dead and intensified talks on the island's future.
A 1998 deal provided for a referendum on independence to be held by the end of 2018. Under the terms of that deal, in the event of a no vote two further referenda can be held before 2022.