Is it possible to completely hide all reverse proxy traffic from a VPS provider?

I run a self-hosted server at home on which I have run a bunch of personal stuff (like nextcloud etc.). To prevent pointing DNS servers at my home router, I run a reverse proxy on a VPS that I rent (from Scaleway FWIW).

Today I was trying to figure to what extent that exposes my data to my VPS provider and whether I can do something about it. Disclaimer: this is just a hobby exercise. I'm not paranoid, I just want to learn for my own self how to improve security of my setup.

My reverse proxy terminates the SSL connection and then proxies the connection over a wireguard connection to my home server. This means that (a) data is decrypted in the RAM of the VPS and (b) the certificates live unencrypted in the storage of the VPS. This means that the VPS provider, if they want to, can read all the traffic unencrypted to and from my home server.

I was thinking that I can solve both problems by using Nginx's SSL pass-through feature. This would allow me to not terminate SSL on the VPS solving (a) and to move the certificates to my home server solving (b).

But just as I was playing around with it, I realised that SSL pass-through would not solve the problem of trying to protect my data from the VPS provider. As long as my DNS records point at the VPS provider's servers, the VPS provider can always get their own certificates for my domains and do a MitM attack. Therefore, I might as well keep the certificates on the VPS since I still have to trust them not to make their own behind my back.

In the end I concluded that as long as I use a VPS provider to route my traffic to my home server, there is no fool-proof way to secure my data from them. Intuitively it makes sense, the data crosses their hardware physically and thus they will have access to it. The only way to stop it would be to update the DNS records to point directly at my home server which I don't want to do.

Is this correct thinking or is there some way to prevent the VPS provider from seeing my data?

Again, I'm trying to solve this problem as a hobby exercise. The most sensitive data that I have is stored encrypted at the filesystem level and I only decrypt it locally on my own machine to work on it. Therefore, the actually sensitive data that would be cost me a lot if compromised is never available unencrypted on the VPS. Due to the overhead of this encryption and other complications, I don't do this for all my files.

pz303,

@dr_robot

Something else to consider is Cloudflare tunnels. They are free and quite handy.

Essentially, it creates a secure wiregiard connection between you and cf so you don't need to open any ports or have a reverse proxy on your side. Then, cf becomes the end point and where dns points.

Cf essentially hosts your reverse proxy for you and you so all the config on their site once you get the tunnel setup. The Cloudflared docker is all that I had to set up.

Its essentially what you are doing with the vps, but purpose built and free.

solidgrue,
@solidgrue@lemmy.world avatar

DNS is very leaky no matter where you run it, unless you run DNS over HTTPS (DoH). Full stop.

I’m no fan of DoH because it scales poorly. Nevertheless, a combination of Tailscale (or tailscale-like securort overlay mesh network) and an in-mesh DoH DNS relay going to be more secure than most other setups. Relay the DNS out through Tor at your own (performance) peril, but that’s going to he very secure.

I’m not a practitioner of this method, but it’s how I would approach it if I needed to.

eskuero,
@eskuero@lemmy.fromshado.ws avatar

Terminate the ssl connection at a reverse proxy hosted in your home server, and instead useiptables to redirect the traffic through the wireguard interface

dr_robot,
@dr_robot@kbin.social avatar

Thanks for the suggestion! That is also doable with Nginx's SSL pass-through. However, that is still vulnerable to the VPS provider obtaining a certificate. But indeed, it does appear that a combination of redirecting encrypted traffic (SSL passthrough or iptables) with cert monitoring appears to be emerging as a solution.

BTW, I prefer SSL pass-through over iptables, because I do keep one endpoint on the VPS and that's my static website which also needs a cert. With SSL pass-through I can terminate connections to the static website while redirecting all other connections as it can pre-read the destination domain. With iptables I would need two IP addresses to distinguish the connections.

taladar,

the VPS provider can always get their own certificates for my domains and do a MitM attack.

You can limit which CA’s will offer certificates for your domain with the CAA record in DNS. You can also at least detect if someone else creates a certificate for your domain if you watch the certificate transparency logs.

dr_robot,
@dr_robot@kbin.social avatar

You can limit which CA’s will offer certificates for your domain with the CAA record in DNS.

Yea, I already have that.

You can also at least detect if someone else creates a certificate for your domain if you watch the certificate transparency logs.

Did not know this before today, but now I have it switched on!

MigratingtoLemmy,

Thank you, this is most helpful

dinosaurdynasty,

CAA can also be used to disable http verification, meaning you would have to have control of DNS to be able to get a certificate (which the VPS ideally wouldn’t have).

taladar,

I didn’t know about that extensions. Thanks for mentioning it. Apparently you can also select which CA account should be the only one allowed to issue certificates for a domain via DNS too.

MigratingtoLemmy,

I am sad, and ashamed, that you had to continuously point towards it being “a hobby exercise” - without which the only answers you would get are “change your VPS provider bro” or “you’re too scared bro”. Paranoid or not, these questions are important to understand and answer (network security is not easy when you get into such concepts), regardless where they are coming from. I am positively dismayed; aggravated even, that even in such a community where people know so much, the first thought that would come to their mind is “just trust them bro”.

That said, you are correct. The VPS can absolutely inspect your storage + RAM and scrape the keys/certificates. Considering that Cloudflare tunnels are much worse, I’d rather stick with a VPS, but the problems remain.

I wonder if LUKS can be used for the underlying storage hosting these certificates. Although, will that help if the RAM of the device is compromised?

Cheers

dr_robot,
@dr_robot@kbin.social avatar

If it was just storage/RAM scraping then that could be solved with SSL pass-through though. That way the reverse proxy would not decrypt the traffic and would forward the encrypted traffic further to the home server. I was actually setting that up a few hours ago. However, since the VPS provider owns the IP address of the VPS, they can simply obtain their own certificate for the domain. After all, Let's Encrypt verifies your ownership of the domain by your ability to control the DNS entries. Therefore, even if the certificates weren't on the VPS, the fact that I am redirecting traffic via their IP address makes me vulnerable to a malicious provider.

The "hobby exercise" was just to indicate that this is not for work and that I'm interested in an answer beyond "you need to trust your provider" which I do :) I agree, these are important questions! And they're also interesting!

MigratingtoLemmy,

Is there a way to be notified when certificates for a particular domain are created?

dr_robot,
@dr_robot@kbin.social avatar

Apparently yes! Based on another comment in this thread: https://certificate.transparency.dev/monitors/.

CAPSLOCKFTW,

I’m using a SSH tunnel to connect a port on my vps to a port on my home server. I have rhevssl certificates both on the vps and the home server (I trust the vps provider), but I’m pretty sure (correct me if I’m wrong!) that this would work with the certificates only on the home server. Could the vps provider do a mitm then? I’m not sure, the packets go in one port and are directly forwarded to my home server.

Can the vps provider get their own certificates? That’s a good question. I guess you could check the certificate when connecting to prevent tampering. Datetime of issue alone should be enough since vps providers can’t fake that. Unless you don’t trust CAs either :)

muddybulldog,

If you’re concerned that you VPS provider is replacing your certificates you need to find another provider.

You should also look in to certificate transparency monitoring. I get notified anytime a certificate gets issued for one of my domains.

dr_robot,
@dr_robot@kbin.social avatar

No, I'm not concerned. This is just a theoretical exercise so that I can understand the trade-offs I'm making.

Edit: The certificate transparency monitoring sounds interesting. Did not know about that.

brombek,
@brombek@lemmy.ml avatar

You could try destination NAT with netfilter/iptables (DNAT) and terminate TLS on your home server.

This way packets will be forwarded to the home server without beign decrypted on the VSP.

ablackcatstail,
@ablackcatstail@lemmy.goblackcat.com avatar

That’s an interesting solution. But could the cloud VPS provider still see the metadata?

taladar,

They will always be able to see the peer’s IP and port, the time packets are sent and the amount of data of course. They will also be able to see the SNI field of the client hello TLS packet, telling them which domains the client is trying to reach.

ablackcatstail,
@ablackcatstail@lemmy.goblackcat.com avatar

It sounds like you’re doing something very similar to me. I run my Lemmy and Mastodon server out of my home. I have a wireguard tunnel between that server and my cloud VPS. The cloud VPS handles reverse proxying. The information that I am most likely leaking is metadata. Metadata is surprisingly useful. In an ideal world we could secure and obfuscate everything. For the most part though, your traffic is secure and your cloud provider won’t be able to really get more than your metadata.

dr_robot,
@dr_robot@kbin.social avatar

I don't think it's just metadata that's leaking though. I would say it's the entire content of the connection. If the reverse proxy terminates the secure connection it will decrypt the data which will be available unencrypted in the VPS. Outside of the VPS instance the traffic remains entirely encrypted.

Admittedly this decrypted data is not easy to access - you would need to have root access and be able to capture the traffic from within the VPS. But a VPS provider has this kind of access - as they run the hypervisor, they have direct access to the RAM (and possibly even a much easier way to just log in as root into the VPS itself). I think you do have to trust the VPS provider not to peek into the VPS itself. As long as you're paying for the service, that's probably a safe assumption.

taladar,

There is no easy way to just log into the VPS as root just because you run the host. There are probably exploits that could do it but there is no simple included tool to do so.

It would probably be easier to just hook into the (likely known) VPS kernel doing something similar to the sslsniff tool from the bpfcc-tools package (hook into the openssl, gnutls,… ssl decryption functions basically).

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • random
  • uselessserver093
  • Food
  • aaaaaaacccccccce
  • [email protected]
  • test
  • CafeMeta
  • testmag
  • MUD
  • RhythmGameZone
  • RSS
  • dabs
  • Socialism
  • KbinCafe
  • TheResearchGuardian
  • Ask_kbincafe
  • oklahoma
  • feritale
  • SuperSentai
  • KamenRider
  • All magazines
    •