> Basically: This attack is only "worth it" if your target is so valuable that you can target them in person. At that point, I'd think the target would use something a little more secure than a Yubikey

Absolutely.

In practice, the Yubikey is almost never going to be the weakest link in the chain. They could target your devices, intercept your communications, or serve warrants on/covertly exploit the services that host your data.