A major con of the enterprisey ilom systems (such as the idrac) is their atrocious security track record. You are basically giving up your "the network is untrusted, I can survive its compromise" badge if you plug in one of those.
Well those ports should never face the internet anyway. Most servers will have a dedicated (physical) port you use for IPMI or whatever -- vlan that and only allow access from your VPN. If you're extra secure you can full on disable the switchport until you need it.
Make sure in the BIOS to disable fallback to one of the other ethernet ports. Quite a few IPMIs will listen on eth0 if it looses the dedicated IPMI port link by default.
This fail-open "should" is bad besides for the obvious reasons, also because it'll be extra ops complexity compared to a secure kvm widget that you don't have to handle with kid gloves.
(And thirdly because of the sibling comment noted footgun.. or silent foot-boobytrap more properly)
The problem is BMC has an astounding array of features[1] that are worth the operational complexity. This isn't just KVM like in OP's post... being able to remote mount images is a godsend when you're provisioning a server or diagnosing hardware issues or doing a BIOS update on the other side of the globe (with your other alternative being shipping a flash drive[2], then paying $200/hr for DC remote hands to plug it in for you).
