The train was making its way from the capital Taipei to the city of Taitung.
Transport Minister Lin Chia-lung told reporters on the scene that the train was carrying about 490 people - higher than an earlier fire department figure of 350.
Taiwan's government said there were 496 people on the train, including 120 without seats.
The express train that derailed in a tunnel in eastern Taiwan was manufactured by Hitachi Ltd. and started operations in May 2007 as Taiwan's first tilting rail auto, the major Japanese conglomerate said.
The Taiwan Railways Agency said 146 passengers were sent to hospital in addition to the 51 confirmed dead.
Belize's Ministry of Foreign Affairs took to its official Twitter account and said, "Belize extends honest condolences to the Government and People of Taiwan following the sad news of a train accident in eastern Taiwan".
In a tweet, Taiwanese President Tsai Ing-wen said emergency services "have been fully mobilised to rescue & assist the passengers & railway staff affected".
Numerous passengers would have been traveling for the first day of the four-day Qingming, or Tomb Sweeping Festival - an annual pilgrimage to the gravesites of ancestors.
Pictures published by local news website UDN showed the front of the train inside the tunnel had been pulverised into a twisted mesh of metal.
"It suddenly came to a stop and then everything shook", one survivor told local television.
Footage released by the Taiwan Red Cross showed specialists with helmets and headlights walking on the roof of the stricken train inside the tunnel to reach survivors.
During the festival, people return to ancestral villages to tidy up the graves of their relatives and make offerings.
With the help of multiple tunnels and bridges, it winds its way through towering mountains and dramatic gorges before entering the picturesque Huadong Valley.
Taiwan's state-owned railways are generally reliable and efficient, but have had a patchy safety record over the years.
The crash quickly became Taiwan's deadliest rail disaster.
Taiwan's last major rail crash was in October 2018 when an express train derailed while rounding a tight corner on the northeast coast, killing at least 18 people and injuring approximately 175.
The driver of the eight-carriage train was later charged with negligent homicide. More than 200 of the 366 people on board were also injured.
The Apple Daily newspaper said the island's worst crash was in 1948 when 64 died.
In 1981, 30 were killed in a collision in northern Taiwan, and in 1991 another 30 died in a train crash.
Another crash in 1961 killed 48, while a 1978 crash left 41 dead.