About 70 people remain trapped in the wreckage and rescue efforts are ongoing, they added. Some carriages hit the wall of the tunnel, as a result of the derailment, the fire department said.
Pictures published by local news website UDN showed the front of the train inside the tunnel had been pulverised into a twisted mesh of metal.
In a tweet, Taiwanese President Tsai Ing-wen said emergency services "have been fully mobilized to rescue & assist the passengers & railway staff affected".
Weng said a construction site truck operated by the railway administration slid onto the track from a work site on the hillside above.
Railway authorities say four of the eight cars are badly damaged. "I felt like we were pushed from behind".
In fact, a yellow trailer vehicle could be seen lying on its side next to the derailed train at the tunnel entrance in the videos that have surfaced from the accident spot.
Another rescued woman said: "My whole body fell to the floor".
The train was en route from the northern part of the island to the southeastern city of Taitung.
The accident happened at around 9am local time (2am BST) and came on the first day of the four-day Qingming Tomb Sweeping Festival.
During the festival, people return to ancestral villages to tidy up the graves of their relatives and make offerings.
The lightly populated east is popular with tourists, many of whom arrive along the coastal railway lines to avoid treacherous mountain roads.
With the help of multiple tunnels and bridges, it winds its way through towering mountains and dramatic gorges before entering the picturesque Huadong Valley.
Taiwan's last major rail crash was in October 2018, when an express train derailed while rounding a tight corner on the north-east coast, killing at least 18 people and injuring almost 200.
The train involved in Friday's accident, the Toroko No 408 is one of Taiwan's newer models and requires tickets to be reserved in advance with no standing allowed.
The driver of the eight-carriage train was later charged with negligent homicide. More than 200 of the 366 people on board were also injured.
The island's worst crash in recent history was in 1991, when 30 passengers were killed and 112 injured after two trains collided.
Thirty were also killed in 1981 after a truck collided with a passenger train at a level crossing and sent coaches over a bridge in Hsinchu.