Several refugee camps in southern Yemen host Somali refugees, although not in the Hodeida area.
The UNHCR said that as conditions worsen in Yemen, refugees are starting to use areas further to the north as a transit route.
It called on all sides in Yemen's war to protect civilians.
"UNHCR is appalled by this tragic incident, the latest in which civilians continue to disproportionately bear the brunt of conflict in Yemen," it said.
The ICRC's Eric Christopher Wyss, quoting survivors at the scene, said many people on the boat "were refugees from Somalia or Yemen, fleeing the conflict".
"It was a gruesome and heartbreaking scene. I saw many men, women and children either killed or horribly wounded," he said.
The IOM said it believed the boat had been headed for Sudan.
"We strongly condemn this attack and deplore such a tragic loss of life," the ICRC's director of operations, Dominik Stillhart, said in a statement.
"These people were themselves fleeing conflict, in search of safety and a better life. We call on the warring parties to conduct an immediate investigation into what happened."
The coast south of Hodeida has seen violent clashes over the past 24 hours, leaving 32 dead including 10 civilians, according to military and medical officials.
The fighting killed 14 rebels and eight soldiers supporting the UN-backed government of President Abedrabbo Mansour Hadi, military sources said.
That came as pro-Hadi forces sought to advance northwards from the port town of Mokha in an offensive to retake Yemen's western coast from Huthi rebel forces.
Saudi-led coalition aircraft had taken part in the assault, the same sources said.
Seven fishermen were also killed off Hodeida by gunfire from an unidentified source, while a further seven were killed in a car near Mokha, hospital workers said.
The UN's humanitarian chief Stephen O'Brien has called Yemen "the largest humanitarian crisis in the world", with two-thirds of the population in need of aid.
On Friday, a rebel missile attack killed at least 26 members of the pro-government forces in a camp east of the capital Sanaa, officials at a hospital in Marib town said.