

Yes, you can just use a reverse proxy for IPv4 only and point it to the IPv6 upstream. That is what I do, with a separate DNS record which then combines the two. See the DNS records for id.knifepoint.net (CNAME), http.vineta.knifepoint.net (AAAA, A) and vineta.knifepoint.net (AAAA).
The reverse proxy config and certificate management is set up with NixOS, if it helps: https://git.dblsaiko.net/systems/tree/nixos/defaults/v4proxy.nix https://git.dblsaiko.net/systems/tree/nixos/modules/sys2x/v4proxy.nix
Tbf, technically data is still decrypted at the reverse proxy and then re-encrypted. So if someone manages to reconfigure the proxy or read its memory somehow they could read traffic in plain text.
However then since they have to control the VPS, they could also get a new cert for that domain (at least the way I’ve configured it) even if it was passed as is to the real host via a tunnel and read the plaintext data that way, so I don’t think a tunnel protects against anything.