We won't lose Longman: Labor frontbencher