Factoring is one of the few quantum algorithms that is odd man out in terms of noise/implementation/scaling/etc. This should be indicating something about the structure of factoring that we don't understand.

Out of all the "classical" problems that we might find another algorithm for, "factoring" would be my bet for the one that we are missing a better algorithm.