The federal opposition leader said Mr Setka had undermined the labour movement and his comments were "completely unacceptable".

"I don't want him in the party I lead. It's that simple," Mr Albanese told reporters in Perth on Tuesday.

The Victorian secretary of the CFMMEU has been suspended by the party, but Mr Albanese said he will have a chance to "press his case".

Mr Setka claims he was taken out of context after reportedly telling a union meeting that Ms Batty's advocacy had led to men having fewer rights.

"People are making up lies about what I said. Every time I see Rosie Batty I want to give her a hug," Mr Setka told the New Daily.

"I would rather be called corrupt. It's not even an exaggeration of what I said. It's an outright lie.

"I've got people portraying me as a neanderthal. This is dirty politics and this is wrong."

Ms Batty says Mr Setka's comments open an opportunity to "continue conversations that we need to have".

"What rights does he feel he has lost as a man, or men are losing?" she told ABC Radio National.

"The right to behave in a violent and abusive, degrading way? Because if that's the rights that men have lost, or feel that they are losing, well then I am pleased that we are finally moving forward and making people who choose to be violent accountable."

Prime Minister Scott Morrison said Mr Setka should have resigned a long time ago.

"They can root out one Labor thug in the union movement, but there's plenty more where John Setka came from," Mr Morrison told reporters in Sydney.

"The CFMMEU is one of the most litigated against and charged unions in the country - their charge sheet is longer than your arm - and John Setka is just one of many."

After remaining silent on the issue for several days, Victorian Labor premier Daniel Andrews on Tuesday said he supported Mr Albanese's move.

"The comments made about her (Ms Batty) by Mr Setka are disgraceful and his refusal to apologise for them is appalling," Mr Andrews said in a statement.

"They cannot be defended in any context."

Mr Andrews, who has rallied alongside Mr Setka, ignored Liberal Party calls to take action against him last month when the union leader indicated he would plead guilty to two criminal charges following an incident at a Melbourne home.

He faces a charge of breaching a court order, along with another charge, details of which cannot be revealed for legal reasons.

Mr Albanese stressed his decision to seek Mr Setka's expulsion is not linked to his court case which is due to resume on June 26.

The Liberals also condemned him last year for likening the federal government to the Hitler regime and for tweeting a picture of his young children holding a sign telling the building commission to 'Go get f*#*ed'.

ACTU secretary Sally McManus has cut short an overseas trip to return home and deal with the crisis.

Ms McManus said in a statement Mr Setka's reported comments were "appalling and totally unacceptable" and he should withdraw them and apologise.

Australian Associated Press