On my machine, I just removed the 'ssh-pkcs11-helper' binary, given that I'll definitely have no need for it in the foreseeable future.

Then I remembered I don't use 'ssh-agent' as my SSH Agent anyway; I use gpg-agent and I'm pretty confident in its security posture. Then I verified it: gpg-agent simply doesn't support any operations except signing with preloaded keys.