It will be two years before $235 million worth of upgrades deliver a class 2 rail line in the North East, but passengers do not want to keep putting up with "rough riding" until then.
Border Rail Action Group has called on the federal and Victorian governments to allocate extra money for short-term repairs.
Executive committee member John Dunstan said more than 20 temporary speed restrictions had been in place on the line for many months, leading to a "poor" ride for passengers.
"Consequently, V/Line are struggling to maintain the punctuality of their services," he said. "BRAG is hearing from many people saying that the rough riding they are currently experiencing is some of the worst they can remember."
But both Victorian Transport Infrastructure Minister Jacinta Allan and the Australian Rail Track Corporation remain focused on using the money already allocated to do the job.
"Whilst the Border Rail Action Group is a tad confused, my concern is getting on with the North East rail line upgrade," Ms Allan said.
"Planning work, including on the new standard gauge bogies, continues to ensure we can deliver new trains once the track is complete."
The concerns come a day after the ARTC opened its expressions of interest for the $100 million-plus contract.
Ms Allan said the move was welcomed, but the Victorian government "would have hoped to receive more certainty around completion times, so that the community knows when these works will be complete and what disruption to expect".
The ARTC revealed on Tuesday the expressions of interest would remain open until May 3 before contractors are shortlisted, with the contract to be awarded by the end of the year.
ARTC projects general manager Ed Walker said four of the speed restrictions were removed recently after level crossing works at Gravel Pit Road in South Wangaratta and repairs to 500 metres of mud holes.
Workers were also halfway through a 20-day "tamping program" to target rough areas of the track and compact the ballast, which aims to create a smoother journey.
"These early works, completed by ARTC's corridor maintenance team, will continue until the main construction contractor begins by the end of the year," Mr Walker said.
"Once main works start, improvements on the line will continue to be progressively felt by passengers."
But he refuted claims by BRAG that $235 million in works were "unlikely to be completed until well after the 'mid-2021' date announcement", saying the statement was unfounded and the project remained on track.
MORE POLITICS NEWS:
Receive our daily newsletter straight to your inbox each morning from The Border Mail. Sign up here