Transparently forwarding SSL traffic to a filtering proxy without breaking encryption with Squid

The Problem

"Bring Your Own Device" (BYOD), the term schools are using to describe "let your students and staff use the devices they already have in their pockets", rather than trying to ban them over (usually) over-inflated fears surrounding internet use and behaviour (which as anyone who spends time in schools over break and lunch times will know is only ever a very superficial ban), is gaining a lot of natural traction now that smartphone ownership for secondary-aged children is near ubiquitous. A smartphone or personal laptop is a great resource to have in lessons for a quick google, video search or Kahoot quiz after all.

To help facilitate this on a technical level, we've provided a wireless network available to all students and staff for some years now. It's a totally separate layer 2 VLAN, isolated from the "main" LAN all our managed computers sit on so we can let people's potentially virus-ridden personally owned devices go nuts without concern for our own infrastructure or data. We authenticate users via 802.1x, so users can just use their normal Active Directory username and password to join the WiFi and WiFi access therefore requires a valid account (so when you leave, or end up with your account suspended for any reason, your wireless devices will lose access too); as well as giving us a nice audit trail for which person is behind each device (people relying on pre-shared key networks for this kind of thing, you are doing it wrong!).

Two things we do NOT do across our network however are:

1. We don't want to require ANY settings to be altered on users own devices for them to start working beyond actually authenticating to the network itself. This includes network settings, proxy settings, support for largely abandoned protocols like WPAD, anything like that. Devices need to 'just work' as they do when you join a wireless network at home - we don't want queues of people asking us to poke at settings on their devices at the IT office every day and we don't want to frustrate users by making them change a load of random settings on their devices!
2. We will not break into our users SSL traffic. This applies equally on our "managed" and "BYOD" networks. SSL interception is bad for a great many number of reasons, including but not limited to:

• Appliances that offer to do SSL interception often do so very badly and do all kinds of things that weaken the security and integrity of traffic - they ignore CRLs, they ignore validation of origin site certificates from their trusted roots, they will sometimes happily accept malicious / self-signed certs from origin sites as genuine - they are 'bad' for web security and make your users far more vulnerable to fraud / phishing / malicious websites than they otherwise would be. A real example of SSL interception holding back browser security was only recently in the news.
• They require their own "trusted root" certificate to be added to all client machines. Not only is this impractical when users are using their own devices, but trying to do so is training users to blindly click through what is quite a major change to the security of their own device "because we say so, don't worry about it, just press 'Accept'". This is very poor training to be exposing young people to. Finally, i'm not sure I even have the "right" to require my users to make such changes to their devices and by extension their data - they aren't mine, they were bought with families own money rather than school funds, so I don't really consider that I have an automatic right to demand they weaken their security for me.
• SSL interception is morally wrong! It hugely decreases the security of communication data that users believe is private. Just because people don't really understand what you're *really* doing when you say "We monitor your traffic" isn't necessarily grounds to do so. If what you actually said to people was "Our computers are going to pretend to be your bank, even though you see the padlock icon in your browser it's really us you're talking to, we could totally capture your banking password, personal e-mails, Facebook password, instant messaging chats and any other piece of information about you - please install this software to allow us to do so" then I think more people might (rightly) get upset about networks doing this!

This presents a problem for BYOD in schools. We use a filtering service to provide some assistance to our teachers as a first line of defence against offensive, illegal, time wasting, age-inappropriate or security-risk websites in school. This service is provided to us as part of the network and internet connectivity network we buy from the Local Authority, is fully hosted by them and works well. We are expected to point our devices at their proxy farm, which is backed by an expensive commercially purchase database of websites -> categories, of which we can choose which ones we'd like to Allow or Block. Their proxies will then either forward the request or deny it depending on the policies we've defined. When we configure a web browser and tell it "Use the following proxy server" as we do on our "managed" computers, the browser will send traffic to the configured proxy server rather than the website directly and for HTTPS/SSL websites, will ask the proxy server to create a TLS encrypted tunnel to the destination site (usually with a CONNECT method) and return whatever traffic comes back. This has the advantage that while the proxy CAN'T see inside a HTTPS/SSL request and modify or record the data, it CAN see "I was asked to create this tunnel to www.badsite.com, which I have categorised as Pornography, so instead of forwarding this request i'm just going to drop it and send a TCP RST back". This is great - the filtering service can "filter" websites based on their hostname regardless of whether they are HTTP or HTTPS, but we don't need to break into the content of HTTPS connections at all. On BYOD / "unmanaged" devices, we can easily achieve the exact same thing for HTTP by transparently intercepting the traffic in the network, passing it to a local proxy server on site, which then uses the filtering proxy in our LAs network as a "Parent". HTTPS traffic from BYOD devices however, has historically been a lot more of a challenge. It's encrypted (that's the point), so it's difficult to even see what website the user was trying to connect to in the network, let alone transparently intercept it. The original solution to this problem was simple - much less of the internet was delivered over HTTPS, so we just passed this traffic directly out to the internet and bodged blocks for the tiny number of services that caused us a problem locally as required.

Over the years however, much more of the internet is now delivered over HTTPS. This is generally a good thing - we send our entire lives over the internet now as a matter of routine - having everything encrypted by default is very sensible. From a schools point of view however, we'd started to see larger amounts of suspect traffic over our BYOD network that we couldn't filter very easily - VPN (often from very suspicious services in app stores) use, all kinds of different social networks, behaviour we didn't really want to carry over our network. What we really wanted, was a way to transparently pass the HTTPS traffic from unmanaged devices to our LAs proxy servers, so they could do the whole "the user is trying to connect to some-god-awful-vpn-service.com - send an RST and do not forward" thing, but without having to change settings on every device on the network and without actually trying to break into the content of the traffic itself.

The Solution:

It turns out, this is now achievable! Thanks to the fact that pretty much every modern web browser now supports SNI, the destination hostname the user is requesting is now sent as a header in the clear. Therefore, it should be possible to look at this header as the traffic passes through our network, generate our own CONNECT request on behalf of the user and then pass this to our LAs proxy farm as if it was any other request. Newer versions of the free and open-source Squid proxy server support a feature they call 'SSLBump Peek and Splice'. This seems primarily designed as an SSL Interception feature, but we can use what squid calls "Peek" to look at the SNI header and work out the intended destination of an HTTPS request and then we can "Splice" to "Become a TCP tunnel without decoding the connection", which includes the ability to pass the request to a "Parent" proxy just as if it had come from a browser configured to do so.

There's not much documentation out there for this scenario. I presume because, in most environments, it's largely pointless! For most networks, this would have the same effect as just passing SSL traffic direct to the internet - but we have the unusual situation of having an upstream proxy server we want to "run the traffic by" so it can consult it's database of hostnames and decide whether it's going to allow the tunnel or not.

Let's go:

I'll cover my implementation of squid in this way below. The pre-requisites for this assume:

1. You already have squid installed and functional as a "normal" proxy server sitting on a seperate network to your BYOD clients. This is fairly standard stuff and i'm not going to cover installing it from scratch
2. I'm using FreeBSD (10.3 at the time of writing) as the operating system for my proxy. Other unix-like OSs (e.g. Linux) should work fine, but might have slightly different paths or installation procedures in some cases)
3. I'm using the fantastic PFSense as the router between my various internal networks. Again, other routers will likely work, but need to be configured in their own way.

So:

1. Make sure you are running the latest version of squid3. At the time of writing, this is Squid 3.5.24
2. Generate a self-signed root certificate. You won't ever be using it for anything as we're only going to ask squid to "splice" and never to "bump", but squids sslbump functionality is designed to do MITM stuff and, even if you don't actually ask it to do that, it still seems to insist you give it the ability to do so:
   openssl req -new -newkey rsa:2048 -sha256 -days 3650 -nodes -x509 -extensions v3_ca -keyout dummyssl.pem  -out dummyssl.pem
   
   ... You can provide whatever info you want for the CA - although it's probably sensible to give it names you'll recognise in the future just in case a misconfiguration causes Squid to ever try and offer this certificate to a client. Move the certificate somewhere sensible (/usr/local/etc/squid/ssl_cert seems sensible to me) and make sure the squid user can read it.
   
   

   

   
   
3. Configure sslbump to "peek and splice" in squid.conf. This assumes the proxy will listen for "normal" connections from browsers on port 3128, for transparently intercepted HTTP on port 3129 and for transparently intercepted HTTPS on port 3130:
   
   http_port 3128
   http_port 3129 transparent
   https_port 3130 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=0KB sslflags=NO_DEFAULT_CA cert=/usr/local/etc/squid/ssl_cert/dummyssl.pem
   
   
   acl step1 at_step SslBump1

   acl step2 at_step SslBump2

   acl step3 at_step SslBump3

   acl nosslsplicesites ssl::server_name "/usr/local/etc/squid/acl/sslbadsites"

   
   

   ssl_bump peek all step1

   ssl_bump terminate nosslsplicesites

   ssl_bump splice all
   
   ... The nosslsplicesites ACL can be used to add any sites you want to just drop SSL traffic for. At the time of writing, the 'NO_DEFAULT_CA' setting is also very important as it seems squid has some pretty huge memory leaks. Running without this initially, our traffic volumes were causing squid to eat through ~8G of system memory in a little over an hour before first chewing through the page file and then crashing both itself and any other processes on the system that required a little memory. Nice.
4. If you don't already, you can configure squid to send all it's traffic to another 'parent' proxy (in our case, the filtering one) with:
   cache_peer my.upstream.proxy.com parent 3128 0 no-query default
   

   acl godirect dstdomain "/usr/local/etc/squid/acl/godirect" # A list of sites we DO NOT want to pass to our parent proxy

   always_direct allow godirect

   never_direct allow all # Require that everything else is sent via the parent
5. You need to initialise the little filesystem the Squid SSL helper uses to generate 'fake' certificates, or Squid just won't start and will just die after throwing angry errors in cache.log. Again, we aren't actually looking to make any fake certificates - but the SSL Helpers squid uses get angry if they can't find what they want so to keep them happy:
   mkdir /var/lib
   /usr/local/squid/libexec/ssl_crtd -c -s /var/lib/ssl_db -M 4MB
   chown -R squid:squid /var/lib/ssl_crtd
6. Squid should now successfully start:
   /usr/local/etc/rc.d/squid start
7. Squid is listening for incoming connections on TCP/3129 and TCP/3130 (both unprivileged ports), but the traffic from our users devices is going to be HTTP to TCP/80 and HTTPS to TCP/443, so we'll need to use a firewall locally to forward traffic to the right ports. ipfw works just fine for this on FreeBSD. You can just create a firewall config, e.g. /usr/local/etc/rc.firewall.conf with something like this in it, adjusting interface names and the IP addresses of your BYOD network as appropriate:
   
   fwcmd="/sbin/ipfw"
   
   
   iif="em0"
   
   
   $fwcmd -f flush
   
   
   $fwcmd add fwd 127.0.0.1,3129 tcp from 10.11.0.0/22 to any 80 recv $iif
   $fwcmd add fwd 127.0.0.1,3130 tcp from 10.11.0.0/22 to any 443 recv $iif
   $fwcmd add allow ip from any to any
   
   ... and then edit /etc/rc.conf adding the following lines to enable the firewall and define the script you just created as the one to use:
   

   firewall_enable="YES"

   firewall_script="/usr/local/etc/rc.firewall.conf"
   
   ... after a reboot, 'ipfw show' should show the rules you just created.
   
   

   
8. Finally, you just need to configure the firewall / router between your BYOD network and the network your squid proxy sits on to set the 'Next Hop' of both HTTP and HTTPS traffic as the squid proxy.
   
   In PFSense you just need to define a 'Gateway' under System -> Routing of your Squid proxy:
   
   
   
   And then create firewall rules matching HTTP and HTTPS traffic, with the 'Gateway' option under 'Advanced' set:
   
   
   
   
   
   
   
   

You should now start to see traffic going via your proxy server from BYOD devices.

• squid.log should show HTTP requests from your clients as normal:
  10.11.1.98 - - [08/Mar/2017:22:19:55 +0000] "GET http://init-p01st.push.apple.com/bag HTTP/1.1" 200 7388 TCP_MEM_HIT:HIER_NONE
• squid.log should also show HTTPS traffic as a CONNECT type and that it's being tunnelled to the parent proxy:
  10.11.1.54 - - [08/Mar/2017:22:20:15 +0000] "CONNECT configuration.apple.com:443 HTTP/1.1" 200 4984 TCP_TUNNEL:FIRSTUP_PARENT
• 'Blocked' HTTP requests should show whatever content you return to users when you block a request
• 'Blocked' HTTPS requests should show a browser error:
  
  
  

So far, this has worked well - i'm not aware of any devices that have "broken" as a result of putting squid in the way of HTTPS traffic and filtering logs show that 'Deny' responses are certainly increased.