TLS in simple terms

When your browser verifies a TLS certificate , it checks for expiration , domains , sans etc… but the most important thing it does is it verifies that the certificate has been signed by a CA (Certificate Authority ) it (the browser) trusts. These CAs are a bunch of arbitrary organisations that are allowed to sign certificate requests etc.

A self-signed certificate , in general terms , is as valid as a certificate singed by a CA , the main difference is that is not signed by a CA that is trusted by the browser.

This topic is huge , and its got a lot of ramifications , but for now , CAs are signers we trust , we trust that whatever they sign is legit if you go to google.com and you are presented with a certificate signed by a trusted CA , then your browser assume is legitimate.

One of the issues to note here , is that as long as the certificate is signed by any of the trusted CAs , its considered to be legitimate. The big issue with this is that if a CA was compromised it could sign certificates for domains that are not supposed to be in its domain , but because the browser is not doing an extensive check , eg check the right CA signed the right cert, the browser then says the certificate is legit.

There’s this story of a CA that got broken into once called DigiNotar , and the attackers did effectively that, signed certificates for Gmail then they poisoned some dns servers to redirect traffic to some bogus proxy , did the TLS handshake and boom , they could decrypt the traffic.

To wrap this up , its important to know that browsers trust a bunch of CAs and of course you can add CAs to the list which is what we gonna do in this article

The trust chain

A little diagram to clear out what’s to come:

Let’s pick this website called www.com.ar (they’re domain hosters etc , good guys) we gonna use them to demonstrate the trust chain:

$ echo | openssl s_client -showcerts -connect www.com.ar:443 | grep BEGIN
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = www.com.ar
verify return:1
DONE
-----BEGIN CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----BEGIN CERTIFICATE-----

As you can see when i tell openssl to -showcerts i get 3 certficates:

  • www.com.ar’s one (signed by LetsEncrypt)
  • LetsEncrypt one (signed by ISRG)
  • ISRG the root CA (Signed by themselves)

At this point my browser doesn’t directly trust LetsEncrypt , but it does trust ISRG(image below) and because my browser trust ISRG then it trust LetsEncrypt intermmediate certificate

at x1 in /etc/ssl/certs
$ ls | grep ISRG
lrwxrwxrwx 1 root root   16 Sep 12  2020 4042bcee.0 -> ISRG_Root_X1.pem
lrwxrwxrwx 1 root root   51 Sep 12  2020 ISRG_Root_X1.pem -> /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

If you split the certificates that openssl commend returns (echo | openssl s_client -showcerts -connect www.com.ar:443) , you can verify the whole chain individually:

$ openssl verify -verbose -CAfile <( cat root.crt le.crt) www.com.ar.crt
www.com.ar.crt: OK
  • Verify root against intermediate
$ openssl verify -verbose -CAfile root.crt le.crt
le.crt: OK

$ openssl verify -verbose -CAfile root.crt www.com.ar.crt
CN = www.com.ar
error 20 at 0 depth lookup: unable to get local issuer certificate
error www.com.ar.crt: verification failed

Cilium

Cilium is an alternative dataplane for kubernetes , it uses eBPF for most of its loadbalancing,routing,firewalling and it is super fast. Its a supported dataplane in GKE (dataplane v2) and its got a ton of benefits over iptables (your default dataplane for firewall and loadbalancing) https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine https://cilium.io/

Cilium uses envoy , a very fast proxy written in c++ , so keep that in mind because we gonna be capturing egress tls traffic , and envoy will do most of the work here.

Cilium installs a bunch of CRDs of course for you to define traffic relationships between pods , deployments or services.

Installing Cilium

You can install cilium with helm or straight with their binary

  • binary (if you’re not usign gke , NATIVE_CIDR will be the address space where pods/svcs will be handed ip address , eg 10.2.2.0/24)
NATIVE_CIDR="$(gcloud container clusters describe demo-gke --zone europe-west2-a --project demo-1 --format 'value(clusterIpv4Cidr)')"
cilium install --version v1.10.1 --native-routing-cidr $NATIVE_CIDR
  • It will install a bunch of crds and pods and a number of cnis on your nodes:
$ k get crds | grep cilium
ciliumclusterwidenetworkpolicies.cilium.io       2021-07-07T09:14:53Z
ciliumegressnatpolicies.cilium.io                2021-07-07T09:14:52Z
ciliumendpoints.cilium.io                        2021-07-07T09:14:53Z
ciliumexternalworkloads.cilium.io                2021-07-07T09:14:52Z
ciliumidentities.cilium.io                       2021-07-07T09:14:52Z
ciliumlocalredirectpolicies.cilium.io            2021-07-07T09:14:52Z
ciliumnetworkpolicies.cilium.io                  2021-07-07T09:14:53Z
ciliumnodes.cilium.io                            2021-07-07T09:14:53Z

jgarcia at x1 in ~
$ k get pods -n kube-system | grep cilium
cilium-gke-node-init-5msck                                       1/1     Running   0          12h
cilium-gke-node-init-sjs5t                                       1/1     Running   0          12h
cilium-operator-d64b8bb76-c22vz                                  1/1     Running   0          12h
cilium-tpg84                                                     1/1     Running   0          12h
cilium-vs9f9                                                     1/1     Running   0          12h
$ insidenode# /etc/cni/net.d # cat 05-cilium.conf
{
  "cniVersion": "0.3.1",
  "name": "cilium",
  "type": "cilium-cni",
  "enable-debug": false
}
  • you can verify the status of cilium by running
jgarcia at x1 in ~
$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         OK
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium             Running: 2
                  cilium-operator    Running: 1
                  hubble-relay       Running: 1
Image versions    cilium             quay.io/cilium/cilium:v1.10.1: 2
                  cilium-operator    quay.io/cilium/operator-generic:v1.10.1: 1
                  hubble-relay       quay.io/cilium/hubble-relay:v1.9.6: 1

Looking at some stuff with Hubble

Hubble is another binary that you have to have in your workspace , basically it connects to a pod and it reads data that is being passed from cilium into hubble and you read through a port-forward

$ cilium hubble port-forward
Forwarding from 0.0.0.0:4245 -> 4245
Forwarding from [::]:4245 -> 4245

Once that’s running , you need to connect there with hubble

hubble --server localhost:4245 observe --pod nginx -f --output json | jq

hubble is reading all traffic that cilium is sending to server side hubble , and i tell it to only observe traffic involving a pod with a name of nginx , so if i run curls i can see some really nice output

{
  "time": "2021-09-12T18:44:13.437040789Z",
  "verdict": "FORWARDED",
  "ethernet": {
    "source": "46:cd:5d:ab:5b:ed",
    "destination": "7a:30:6a:19:4e:2c"
  },
  "IP": {
    "source": "109.73.209.117",
    "destination": "157.240.18.35",
    "ipVersion": "IPv4"
  },
  "l4": {
    "TCP": {
      "source_port": 34260,
      "destination_port": 80,
      "flags": {
        "ACK": true
      }
    }
  },

There you can see a snap at layer4 , so my pod nginx initiated a connection to www.facebook.com (157.240.18.35 ) on port 80 , neat but not super useful. You can also find a lot of dns action , but also a layer 3.

Why is we want to inspect at layer 7?

At layer 3/4 you use some fields to make flow decisions , like IP addresses, ports , packet sums , sizes , windows … not super useful, especially these days that you find a million services under the same IP address. We want to make decisions based on more attributes and for that we need a firewall/proxy something that can de-capsulate more complex protocols and this is done at layer 7

For example http:

One way to get layer 7 inspection

You can annotate your pod , to let cilium to capture these traffic (and pass it through envoy)

$ kubectl annotate pod nginx io.cilium.proxy-visibility="<Egress/53/UDP/DNS>,<Egress/80/TCP/HTTP>"

Let’s check if we see something to tell us now our curl are being proxied?

$ k exec -it nginx -- curl http://www.facebook.com/ -vv | grep envoy
< x-envoy-upstream-service-time: 34
< server: envoy

Cool , so now we should be able to inspect at layer7 within hubble:

{
    "destination_names": [
    "www.facebook.com"
  ],
  "l7": {
    "type": "REQUEST",
    "http": {
      "method": "GET",
      "url": "http://www.facebook.com/cucumber",
      "protocol": "HTTP/1.1",
      "headers": [
        {
          "key": "Accept",
          "value": "*/*"
        },
        {
          "key": "User-Agent",
          "value": "curl/7.64.0"
        },
        {
          "key": "X-Header",
          "value": "imatest"
        },
        {
          "key": "X-Request-Id",
          "value": "46ac8011-bd44-4d56-b86e-fb36b93ccf4f"
        }
      ]
    }
  },
  "event_type": {
    "type": 129
  },
  "traffic_direction": "EGRESS",
  "is_reply": false,
  "Summary": "HTTP/1.1 GET http://www.facebook.com/cucumber"
}

You can see that now we see layer7 stuff like headers X-Header ,*cucumber* in the path ,method and also protocol version , these are all visible because of envoy is doing inspection on the egress traffic sourcing from the pod called nginx.

Can cilium do TLS inspection then?

Short Answer is yes , but it’s not straight fwd. The reason is that if when envoy proxies the traffic , envoy won’t be able to impersonate facebook , as it doesn’t have the right certificate , that’s why it went about ssl and trust in the first part of this article.

The only way we can get this to work is by injecting a trusted CA into our pods and get them to trust the certificate that envoy presents to the clients, luckily this isn’t super complex to implement in cilium.

IMPORTANT NOTE Browsers don’t really care what root CA signs a specific certificate for a specific domain , as long as the certificate is signed by an intermediate signed by a root they trust that would be enough , this is a thing that will need to be fixed at some point as when CAs get hacked a world of hell is unleashed and the only thing we users can hope is a fast release from our favourite browser.

So the idea is:

  • We create a CA crt
  • We bake it into our pods
  • We tell Envoy to present a “random” www.facebook.com certificate signed by that CA
  • And curl will be happy

If this all works envoy will be able to decrypt traffic and we will be able to see it in hubble , ok lets tackle it

Create some certificates:

  • CA
openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.crt
  • Let’s create a facebook CSR and sign it with the fake CA
openssl req -new -key internal-facebook.key -out internal-facebook.csr
 ....
openssl x509 -req -days 360 -in internal-facebook.csr -CA myCA.crt -CAkey myCA.key -CAcreateserial -out internal-facebook.crt -sha256

Make some kube secrets

These certs need to be visible to cilium so we need to move them to secrets:

kubectl create secret tls facebook-tls-data -n kube-system --cert=internal-facebook.crt --key=internal-facebook.key

IMPORTANT NOTE You need to tell cilium what root certificates do we trust of the real connection, that will be from envoy -> facebook , this your normal CA bundle in ubuntu you can find it in (etc/ssl/certs/ca-certificates.crt)

kubectl create secret generic tls-orig-data -n kube-system --from-file=ca.crt=./ca-certificates.crt

Wire it all up with a Cilium policy

We gonna create a really simple policy that not only allows traffic over 443 tcp to facebook.com but also tells envoy what certificates to use:

apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "facebook-tls"
spec:
  endpointSelector:
    matchLabels:
      name: nginx
  egress:
  - toFQDNs:
    - matchName: "www.facebook.com"
    toPorts:
    - ports:
      - port: "443"
        protocol: "TCP"
      terminatingTLS:
        secret:
          namespace: "kube-system"
          name: "facebook-tls-data"
      originatingTLS:
        secret:
          namespace: "kube-system"
          name: "tls-orig-data"
      rules:
        http:
        - method: GET
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
          - matchPattern: "*"

Some important things to note:

  • matchLabels –> applying to my nginx pod
  • terminatingTLS –> my fake cert
  • originatingTLS –> trusted real bundle
  • last toPorts rule allowing dns traffic (we need this)
  • rules allowing method GET

Let’s apply the rule and check what the situation is:

k apply -f facebook-tls.yaml
ciliumclusterwidenetworkpolicy.cilium.io/l7-visibility-tls created

Some testing

At this point envoy is showing my fake facebook cert to my nginx pod , lets check that:

$ k exec -it nginx -- curl https://www.facebook.com/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
command terminated with exit code 60

Curl is not trusting the rubbish cert envoy is presenting , but why it should? Remember that if the CA is not trusted curl/browsers/others will flag it and not trust it , lets have a closer look:

$ k exec -it nginx -- bash -c "openssl s_client -connect www.facebook.com:443| openssl x509 -text -noout -subject"
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = www.facebook.com

You can already tell the cert that is being presented is not the real cert , Internet Widgits …Some-State ? it’s def the certificate we created and signed by our fake CA .

Final step , let’s bake our fake CA

We need to copy the CA for curl to trust and re-do the bundle:

jgarcia at x1 in ~/Projects/si/terraform/cilium
$ kubectl cp myCA.crt default/nginx:/usr/local/share/ca-certificates

jgarcia at x1 in ~/Projects/si/terraform/cilium
$ k exec -it nginx -- update-ca-certificates
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

1 added , that looks good , lets try that curl again , it should trust automatically!

$ k exec -it nginx -- curl https://www.facebook.com/ -w %{response_code} --output /dev/null -s
200

It works!

Let’s check on hubble!

Now that my pod trusts everything we are feeding it , hubble will be able to see everything so let’s check!

{
         "destination_names": [
       "www.facebook.com"
     ],
     "l7": {
       "type": "REQUEST",
       "http": {
         "method": "GET",
         "url": "https://www.facebook.com/cucumber",
         "protocol": "HTTP/1.1",
         "headers": [
           {
             "key": "Accept",
             "value": "*/*"
           },
           {
             "key": "Dada",
             "value": "dada"
           },
           {
             "key": "User-Agent",
             "value": "curl/7.64.0"
           },
           {
             "key": "X-Request-Id",
             "value": "42bca2eb-6ced-41ff-9599-96b1dffb3a23"
           }
         ]
       }
     },
     "event_type": {
       "type": 129
     },
     "traffic_direction": "EGRESS",
     "is_reply": false,
     "Summary": "HTTP/1.1 GET https://www.facebook.com/cucumber"
   }

Great you can see that this is an https request and we were able to decrypt traffic on the go , and capture everyting about it beyond the SNI

  • paths
  • headers
  • methods

But also , as envoy is mitm everything , the response:

"l7": {
"type": "RESPONSE",
"latency_ns": "97422433",
"http": {
  "code": 404,
  "method": "GET",
  "url": "https://www.facebook.com/cucumber",
  "protocol": "HTTP/1.1",
  "headers": [
    {
      "key": "Alt-Svc",
      "value": "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600,h3-27=\":443\"; ma=3600"
    },
    {
      "key": "Cache-Control",
      "value": "private, no-cache, no-store, must-revalidate"
    },
    {
      "key": "Connection",
      "value": "keep-alive"
    },
    {
      "key": "Content-Type",
      "value": "text/html; charset=\"utf-8\""
    },
    {
      "key": "Date",
      "value": "Sun, 12 Sep 2021 19:49:09 GMT"
    },
    {
      "key": "Expires",
      "value": "Sat, 01 Jan 2000 00:00:00 GMT"
    },
    {
      "key": "Pragma",
      "value": "no-cache"
    },

Now that we have this level of inspection we can make a lot more complex rules for example we can allow certain headers or methods :

  http:
- method: GET
  path: "/cucumber"
  headers:
    - 'dada: dada'

So unless i send a GET request to https://www.facebook.com with a header dada:dada and to the path /cucumber it will not allow me out:

jgarcia at x1 in ~/Projects/si/terraform/cilium
$ k exec -it nginx -- curl https://www.facebook.com/cucumber -w %{response_code} --output /dev/null -s -H dada:dada
404
jgarcia at x1 in ~/Projects/si/terraform/cilium
$ k exec -it nginx -- curl https://www.facebook.com/cucumber -w %{response_code} --output /dev/null -s -H dada:dado
403

The big difference is the 404 comes from facebook , saying they don’t have a /cucumber path , but the 403 comes really from envoy saying unless the headers are dada:dada and the path cucumber it won’t let me through , notice that in the second request i changed the header to force a failure