I often connect to my remote servers because they have a problem. If a machine has a problem I can ssh into it and inspect it without worrying about my localhost. If I used vscode, I have to also be lucky, which is not a good security assumption.