The state-run Tunis Afrique Presse agency gave a death toll of at least 70 people.
It was one of the deadliest shipwrecks involving refugees trying to reach Europe illegally this year.
The UN's International Organization for Migration said the sunken boat took to the sea from neighboring Libya, where renewed warfare between rival factions has gripped the capital, Tripoli, in the past five weeks. It put the number of victims at 50, with 16 survivors, Presstv Reprted.
The boat sank 40 miles off the coast of Sfax, south of the capital, Tunis, the Tunisian agency said. Fishing boats rescued the survivors, it said.
The Tunisian Defense Ministry said the boat had left from the Libyan port of Zouara on Thursday aiming to reach Italy.
Navy units have recovered only three bodies so far, it said in a statement.
"Another tragedy in the Mediterranean," the IOM said.
Libya's western coast is a main departure point for refugees from across Africa hoping to reach Europe by paying human traffickers, though numbers have dropped due to an Italian-led effort to disrupt smuggling networks and support the Libyan coast guard.
Earlier on Friday, UNHCR spokesman Babar Baloch said the United Nations had repeatedly voiced its concerns about the lack of boats to pick up migrants and refugees fleeing from "the horrendous and horrible situation" in Libya. It called on governments to step in.
Although the fighting in Libya has made the situation more difficult for people-trafficking rackets, international aid officials have warned that it could also prompt more Libyans to flee their country.
It was not immediately known from which countries the refugees involved in Friday's tragedy were.
According to the IOM, 2,297 refugees died or went missing in the Mediterranean last year.
Some 117 refugees who left Libya in a rubber dinghy in January went missing and most remain unaccounted for, according to the IOM.