Federal and state governments have been shaken by the rapid rise in virus cases - now well above 1000 - and by the crowds on Friday and Saturday on Bondi beach and at bars and clubs.
South Australia and Western Australia on Sunday said they would close their borders as of Tuesday to restrict the movement of people, following similar actions in the Northern Territory and Tasmania announced last week.
"We can't have the confidence that the social distancing will be followed to save lives".
"The failure of our public to do that will put people at risk", he said.
The Prime Minister has resisted calls for a blanket closure of schools across the country.
From today "stage one" restrictions will be introduced and will be reviewed on a monthly basis.
The Prime Minister said that the measures cover anything deemed a "principal place of socialisation".
Restaurants and cafes will be restricted to providing takeaway only, with dining now forbidden from midday.
Gatherings in closed spaces will be stopped from Midday Monday.
"We need you to do your bit, when it comes to social distancing, to keeping that healthy distance, to respecting and following the rules that we're setting down".
Australia has recorded more than 1,300 cases and seven deaths from COVID-19.
All states and territories instead agreed to shut: licensed sections of pubs and hotels, clubs, cinemas, casinos, entertainment venues, indoor sporting venues and indoor places of worship.
"Because the sooner we act, the more effective it will be".
He said there was no change in the medical advice that schools should remain open.
"It's true, most young people don't get significant disease". Restrictions on entering aged care homes have also been introduced.
All non-essential travel within Australia should be cancelled to tackle the spread of COVID-19, the prime minister has announced.
"Leaders agreed that we can not see children lose an entire year of their education as a result of school closures caused by COVID-19", he said.
He said work-related trips, the transport of essential supplies and travel on compassionate grounds could continue but people should cancel any other travel plans ahead of the upcoming Easter school holidays.
"If parents choose to keep their children home from school, parents must be responsible for the conduct of the children and to ensure they adhere to the social distancing arrangements in place", he warned.