SSRF and RCE are different things. Being able to throw requests to localhost and accessing the host filesystem don't necessarily have to be the same vulnerabilities, but I'll concede that SSRF vulns are uncommon.
I'm just worried that Caddy will be the source of "security misconfiguration" (1) findings in penetration test reports. It's my opinion that we as software engineers should strive not to leave our software insecure by default, is what I'm saying, and that's how I see Caddy 2's admin API.
If the host is popped, I'm not sure what Caddy can do to save you. Even authentication has to be stored on the machine... that was popped.