What if you have two things? You'd then need two buttons.
The push button is a perfectly viable option, it just needs to be in a form factor that's works. Could be as simple as a tiny low-energy Bluetooth board with a coin battery that will last several months.
Yeah, a touch-sensitive com-badge mounted on your chest would work.
Actually, I would think a small coin-sized button & transmitter that did nothing but emit a signal that your assistant (or phone) interprets as 'start listening' would be pretty useful. In your pocket, on a watch band, etc.
The push button is a perfectly viable option, it just needs to be in a form factor that's works. Could be as simple as a tiny low-energy Bluetooth board with a coin battery that will last several months.