A sailor who had been operating the boat, Ibrahim Ali Zeyad, said 80 refugees had been rescued after the incident.
Seven fishermen were also killed off Hodeida by gunfire from an unidentified source, while a further seven were killed in a auto near Mokha, hospital workers said.
Eric Christopher Wyss, of the International Committee of the Red Cross, described the scene at the port as "gruesome and heartbreaking".
The refugees were hit by light weapons fire in waters off rebel-held Hodeidah but managed to dock in the city's port, an official there said.
He said Yemen's government also must give an explanation for the attack and that those responsible must be brought to justice.
Josh Gad surprises NYC theaters ahead of 'Beauty and the Beast'
I felt this way when I saw the live-action version of Cinderella a few years ago. "We all make choices and I am making mine". It might have just been that the other characters were so much more entertaining that Watson's Belle got washed away.
Yemen's Shia rebels accused the Saudi-led air coalition of carrying out the attack.
The trafficker, Al-Hassan Ghaleb Mohammed, said the boat left from Ras Arra, along the southern coastline in Yemen's Hodeida province, and was near the Bab al-Mandab strait when the attack occurred. The Trump administration has ramped up the offensive begun by Barack Obama which has already killed hundreds of people, including women and children.
The civil war began after the Houthi militants with support from forces loyal to the former president ousted the UN-backed transitional government and occupied capital Sanaa militarily in September 2014. "There has been no firing by the coalition in this zone", he said.
The organization said the refugees, who were from Somalia, were probably heading towards Sudan.
Numerous wounded lost arms and legs in the strike, the Yemeni medical official told AP on condition of anonymity. 32 people, including 10 civilians, have been killed.
Civil wars stoked by the United States government and its European allies combined with historic drought conditions are pushing millions across the Africa continent to flee their homes.