Five bodies have been recovered while around 50 people were missing after the incident on Thursday, the International Organisation for Migration reported.
It wasn't immediately clear where the refugees came from, but in a similar incident on Wednesday, 120 migrants from Ethiopia and Somalia were forced overboard in the same area by smugglers who feared arrest if they were caught with human cargo.
Refugees and migrants hoping to reach oil-rich Gulf countries - many of them teenagers - are increasingly traversing a risky sea route to Yemen, despite the conflict raging there.
"They were in a boat with smugglers, who dropped them at sea before arriving at the shores".
Despite the war in Yemen, almost 55 000 migrants have left the Horn of Africa, to arrive in this country from January 2017, according to IOM.
Staff estimated the average age of passengers on the boat to be 16.
At least 22 migrants remained missing, the IOM said.
According to survivors' testimonies, the smuggler pushed them to the sea after he saw men he perceived as "authority types".
Despite the conflict, migrants still seek transit through Yemen, often towards its land border with Saudi Arabia. One-third are women. In spite of the war in Yemen, African migrants continue to choose this route because there is no central authority that can prevent them from undertaking their trip.
The journey is especially hazardous at this time of year due to strong winds in the Indian Ocean.
The group, which included Gambians and Senegalese, said they had been abandoned by their driver.
"They have some family, or others, who've succeeded and reported back home that it's easy to find a job".
The route is popular because it is cheaper than others, but when they arrive migrants often fall victims to abuse.
More than 111,500 migrants landed on Yemen's shores last year, up from around 100,000 the year before, according to the Regional Mixed Migration Secretariat, a grouping of global agencies.