Mr Swinney insisted that he made a decision to upgrade the results "as quickly as I possibly could".
Students taking Scottish Highers were given estimated grades by teachers, which were then moderated by exam board SQA.
It follows a Scottish Government U-turn due to anger over almost 125,000 results being downgraded from teacher estimates by the SQA's controversial moderation system.
"If you go to a school in a deprived area, there has been a deprivation algorithm applied to the schools so that those grades that teachers gave you on your coursework and all your pre-learns have been downgraded because, essentially, what the Scottish Government have been saying to poorer students is: "you should know your place, and you don't deserve those higher grades", he said.
On Tuesday, Mr Swinney announced he would be instructing the SQA to revoke downgraded marks and replace them with teacher estimates.
Pupils, parents, teachers and opposition politicians have been united in condemning the process.
She responded: "It was hard to see the reaction to last week's results".
He added: "The people who are now calling for my resignation would have complained if I had not announced what I announced yesterday to parliament".
He said: "We will listen to John Swinney's proposals in Parliament and if we are not satisfied with what he puts forward, we will vote to remove him".
But he said he had changed his mind after listening to the "heartfelt pleas" and "anguish" from young people who had been affected.
PROTECTIONS to prevent evictions of renters in Scotland look set to be extended after the Scottish government indicated that it intends to prolong emergency legislation into next year.
Nicola Sturgeon said there had been too much focus on the system rather than pupils in efforts to ensure results were as valid as in previous years.
The appeals process would have dealt with any "anomalies" in the moderated results, she said, while the SQA's equalities impact assessments showed the results were "fair".
She said the SQA treated teacher estimates with "utmost respect", but there was a "clear and unequivocal case for some moderation" because they were "not always accurate".
The downgraded results will now be withdrawn, reverting to the original estimates.
"The situation is not the fault of students and should not be on students to fix it - that's on us and we will set out tomorrow exactly how we intend to do that".
The SQA says schools will receive the new, confirmed results "as soon as possible and no later than the end of next week", with fresh certificates also being sent out to affected candidates.