Campaigners gathered outside the council office to object the plans A town's derelict speedway track has reached a "crucial moment" as plans to transform the site into housing have been rejected.