At the time of Sunday's attack, the prison held more than 1,700 inmates - mostly IS or Taliban fighters, a security source told AFP.
Khogyani said about 1,000 escaped prisoners had been caught, but it is unclear if any more are at large.
Officials said a total of 10 attackers were also killed.
Security forces also clashed with Islamic State fighters on Monday in another part of Nangarhar, according to Ghalib Mujahid, the governor of Behsud district, though he said he did not yet have any information on casualties. Eight attackers were killed, he said.
Islamic State's Amaq news agency quoted a military source as saying its fighters had carried out Sunday's attack but gave no further details.
The U.S. reached the peace deal with the Taliban in February, but a second, crucial round of negotiations between the Taliban and the political leadership in Kabul has yet to begin.
The attack highlighted the challenges ahead for Afghanistan, even as U.S. and North Atlantic Treaty Organisation forces begin to withdraw following a U.S. -Taliban peace deal struck earlier this year. The rest of the dead were believed to be prisoners, civilians and Afghan forces, although no official breakdown was given.
US Secretary of State Mike Pompeo, eager for a diplomatic win ahead of elections, discussed the prisoner issue by videoconference Monday with Taliban chief negotiator Mullah Baradar, insurgent spokesman Suhail Shaheen said.
Acting Interior Minister Massoud Andrabi said on Twitter that IS's new head in Afghanistan, Shahab Al-Mahajir, was a member of the Haqqani network, considered a Taliban offshoot.
One resident, Atiqullah, who only uses one name, said that rumors were circulating that the Taliban were regrouping around Kunduz city.
Of the five prisoners killed by the militants, at least three were members of the Taliban, showing the tensions between the two factions.
The telephone conversation also came as a ceasefire between the Afghan government and the Taliban announced by the latter on the occasion of Eid came to an end.
Washington has been pushing for an early start to intra-Afghan negotiations that have repeatedly been delayed since the Taliban accord was reached.
Officials would not confirm the number of prisoners now being discussed for release in exchange of the remaining hostages, but it is understood that nine Taliban prisoners, including some releases facilitated by the USA have been freed, while two others, accused of more serious crimes, awaited being freed by the government.
After the Taliban protested, even threatening violence would be resumed, President Ghani said he would convene a meeting of "elders" or a Loya Jirga to decide on the matter, which is now expected on Friday.
Nangarhar province has seen deadly IS attacks, including a suicide bomb that killed 32 mourners at a funeral in May. The Taliban has been engaged in an armed rebellion since 2001 when it was removed from power in a US-led invasion.
The objective of the attack is not clear, but hundreds of prisoners at the jail are members of the Islamic State group, AP reported.