Date:

Share:

Go HTTPS servers with TLS

Related Articles

This post is a basic introduction to running HTTPS servers and clients in Go using TLS. This assumes some familiarity with the crypto of a public developer. Feel free to check out my previous posts on RSA and the Diffie-Hellman Key Exchange; TLS uses the elliptical version of Dippy-Hellman. I will not deal here in detail with how the protocol itself works, but if you are interested I recommend reading on the subject.

All code for this post is available In this database.

A brief (very) introduction to TLS

TLS (Transport Layer Security) is a protocol designed to enable client-server communication over the Internet in a way that prevents eavesdropping, jamming and forgery of messages. It is described in RFC 8446.

TLS relies on advanced cryptography; This is also the reason why it is recommended to use the newest version of the available TLS, which is 1.3 (as of early 2021). TLS protocol patches clean up potentially unsafe corners, remove weak encryption algorithms, and generally try to make the protocol more secure.

When a client connects to a server with standard HTTP, it starts sending standard text data wrapped in TCP packets immediately after completing the standard TCP handshake (SYN -> SYN-ACK -> ACK). Using TLS, the situation is a little more complicated:

After completing the TCP handshake, the server and client perform a TLS handshake to agree on a shared secret that is unique only to them (and to that specific session). This shared secret is then used to securely encrypt all the data exchanged between them. While a lot is happening here, it is something that the TLS layer is implementing for us. We just need to configure the TLS (or client) server properly; The actual difference between HTTP and HTTPS server in Go is minimal.

TLS certificates

Before we jump to code that shows how to set up HTTPS server in Go using TLS, let’s talk about Certificates. In the diagram above, you will notice that the server sends a certificate to the client as part of its first
Hello server message. Formally these are known as X.509 certifications, described by RFC 5280.

Public key encryption plays a key role in TLS. A certificate is a standard way to wrap the public key of the server, along with the identity and signature of a trusted authority (usually Permission Authority). Suppose you want to talk https://bigbank.com; How do you know it’s really a Big Bank name asking for your password? What if someone sits on your cable connection, intercepts all traffic and pretends to be a Big Bank (classic MITM – mid-man attack).

The approval process is designed to prevent this scenario. When implementing your customer’s basic TLS you approach https://bigbank.com, He is expecting a Big Bank certificate with a public key, signed by a trusted certification authority (CA). Certificate signatures can be a tree (a bank key signed by A, which is signed by B, which is signed by C, etc.), but at the end of the chain he must have some credentials trusted by your customer. Modern browsers have a built-in list of pre-trusted CAs (along with their own credentials). Because prying your cables can not forge a credible certificate signature, they can not impersonate Big Bank.

Generating self-signed certificates in Go

For local testing, it is very often useful to be able to work with self-signed certificates. A self-signed certificate is a certificate for entity E with public key P, but the key is signed not by a known certification authority, but by P itself. While self-signed certificates have several other legitimate uses, we will focus on using them for testing here.

Go’s standard library has excellent support for everything related to crypto, TLS and certificates. Let’s see how to create a self-signed certificate in Go!

privateKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil 
  log.Fatalf("Failed to generate private key: %v", err)

This code uses crypto / ecdsa, Crypto / Elliptical and Crypto / Rand
Packages for creating a new pair of keys, using the elliptical curve P-256, which is one of the allowed curves in TLS 1.3.

Next, we will create a Certificate template:

serialNumberLimit := new(big.Int).Lsh(big.NewInt(1), 128)
serialNumber, err := rand.Int(rand.Reader, serialNumberLimit)
if err != nil 
  log.Fatalf("Failed to generate serial number: %v", err)


template := x509.Certificate
  SerialNumber: serialNumber,
  Subject: pkix.Name
    Organization: []string"My Corp",
  ,
  DNSNames:  []string"localhost",
  NotBefore: time.Now(),
  NotAfter:  time.Now().Add(3 * time.Hour),

  KeyUsage:              x509.KeyUsageDigitalSignature,
  ExtKeyUsage:           []x509.ExtKeyUsagex509.ExtKeyUsageServerAuth,
  BasicConstraintsValid: true,

Each certificate needs a unique serial number; Normally, the certification authorities will store them in some database, but for our local needs, a random 128-bit number will fit. This is what the first lines of the passage do.

Next comes the x509.Certificate pattern. For more information on the meaning of the fields, see Crypto / x509 package documents, In addition to RFC 5280. Please note that the permit is valid for 3 hours, and is only valid for Local host domain.

the next:

derBytes, err := x509.CreateCertificate(rand.Reader, &template, &template, &privateKey.PublicKey, privateKey)
if err != nil 
  log.Fatalf("Failed to create certificate: %v", err)

The certificate is created from the template, and is signed in the private key we created earlier. It noted &pattern Delivered in both for the
pattern and parent Parameters of Create a certificate. The latter is what makes this certificate Signed himself.

That is, we have the private key to our server and its certificate (which contains, among other things, the public key). All that is left now is to arrange them into files. First, the certificate:

pemCert := pem.EncodeToMemory(&pem.BlockType: "CERTIFICATE", Bytes: derBytes)
if pemCert == nil 
  log.Fatal("Failed to encode certificate to PEM")

if err := os.WriteFile("cert.pem", pemCert, 0644); err != nil 
  log.Fatal(err)

log.Print("wrote cert.pemn")

Then, the private key:

privBytes, err := x509.MarshalPKCS8PrivateKey(privateKey)
if err != nil 
  log.Fatalf("Unable to marshal private key: %v", err)

pemKey := pem.EncodeToMemory(&pem.BlockType: "PRIVATE KEY", Bytes: privBytes)
if pemKey == nil 
  log.Fatal("Failed to encode key to PEM")

if err := os.WriteFile("key.pem", pemKey, 0600); err != nil 
  log.Fatal(err)

log.Print("wrote key.pemn")

We arrange the certificate and key into PEM files, Which looks like this (for the certificate):

-----BEGIN CERTIFICATE-----
MIIBbjCCARSgAwIBAgIRALBCBgLhD1I/4S0fRZv6yfcwCgYIKoZIzj0EAwIwEjEQ
MA4GA1UEChMHTXkgQ29ycDAeFw0yMTAzMjcxNDI1NDlaFw0yMTAzMjcxNzI1NDla
MBIxEDAOBgNVBAoTB015IENvcnAwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASf
wNSifB2LWDeb6xUAWbwnBQ2raSQTqqpaR1C1eEiy6cgqUiiOlr4jUDDiFCly+AS9
pNNe8o63/Gab/98dwFNQo0swSTAOBgNVHQ8BAf8EBAMCB4AwEwYDVR0lBAwwCgYI
KwYBBQUHAwEwDAYDVR0TAQH/BAIwADAUBgNVHREEDTALgglsb2NhbGhvc3QwCgYI
KoZIzj0EAwIDSAAwRQIgYlJYGIwSvA+AmsHe8P34B5+hlfWEK4+kBmydJ65XJZMC
IQCzg5aihUXh7Rm0L1K3JrG7eRuTuFSkHoAhzk4cy6FqfQ==
-----END CERTIFICATE-----

If you have ever set SSH keys, the format should look familiar. We can use openssl Command line tool to view its contents:

$ openssl x509 -in cert.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            b0:42:06:02:e1:0f:52:3f:e1:2d:1f:45:9b:fa:c9:f7
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O = My Corp
        Validity
            Not Before: Mar 27 14:25:49 2021 GMT
            Not After : Mar 27 17:25:49 2021 GMT
        Subject: O = My Corp
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:9f:c0:d4:a2:7c:1d:8b:58:37:9b:eb:15:00:59:
                    bc:27:05:0d:ab:69:24:13:aa:aa:5a:47:50:b5:78:
                    48:b2:e9:c8:2a:52:28:8e:96:be:23:50:30:e2:14:
                    29:72:f8:04:bd:a4:d3:5e:f2:8e:b7:fc:66:9b:ff:
                    df:1d:c0:53:50
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name:
                DNS:localhost
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:20:62:52:58:18:8c:12:bc:0f:80:9a:c1:de:f0:fd:
         f8:07:9f:a1:95:f5:84:2b:8f:a4:06:6c:9d:27:ae:57:25:93:
         02:21:00:b3:83:96:a2:85:45:e1:ed:19:b4:2f:52:b7:26:b1:
         bb:79:1b:93:b8:54:a4:1e:80:21:ce:4e:1c:cb:a1:6a:7d

HTTPS server in Go

Now that we have the certificate and private key in hand, we are ready to run an HTTPS server! Again, the standard library makes it very easy, though it is important to mention that security is a very complicated issue. Before exposing your server to the public Internet, consider consulting a security engineer about best practices and what configuration options you should be aware of.

Here is a basic HTTPS server in Go:

func main() 
  addr := flag.String("addr", ":4000", "HTTPS network address")
  certFile := flag.String("certfile", "cert.pem", "certificate PEM file")
  keyFile := flag.String("keyfile", "key.pem", "key PEM file")
  flag.Parse()

  mux := http.NewServeMux()
  mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) 
    if req.URL.Path != "/" 
      http.NotFound(w, req)
      return
    
    fmt.Fprintf(w, "Proudly served with Go and HTTPS!")
  )

  srv := &http.Server
    Addr:    *addr,
    Handler: mux,
    TLSConfig: &tls.Config
      MinVersion:               tls.VersionTLS13,
      PreferServerCipherSuites: true,
    ,
  

  log.Printf("Starting server on %s", *addr)
  err := srv.ListenAndServeTLS(*certFile, *keyFile)
  log.Fatal(err)

It serves as a single therapist in the root path. The interesting part is the TLS configuration as well ListenAndServeTLS call, which takes the paths to a certificate file and a private key file (in PEM format, just as we created them earlier). The TLS configuration has many possible fields; Here, I have chosen a relatively strict protocol of at least TLS 1.3 coercion. TLS 1.3 comes with strong security out of the box, so this is a good option if you can ensure that all your customers understand this version (and in 2021, they should!)

The difference from a standard HTTP server is less than 10 lines of code! Most of the server’s code (handler for specific routes) is not fully aware of the basic protocol and will not change.

When this server is running locally (and serving in port 4000 by default), Chrome will initially crash while accessing it:

This is because a web browser will not receive, by default, a self-signed certificate. As mentioned above, browsers come with an encrypted list of CAs they trust, and our self-signed certificate is of course not one of them. We can still continue to serve by clicking Advanced and then allow Chrome to continue, accepting the risk explicitly. He will then show us the site, albeit reluctantly (with a “unsecured” red sign in the address bar).

If we try curl For server, we also get error:

$ curl -Lv  https://localhost:4000

*   Trying 127.0.0.1:4000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

By reading the documents, we can find this curl You can make our server trust by giving the server permission on --cacert flag. If we try it:

$ curl -Lv --cacert <path/to/cert.pem>  https://localhost:4000

*   Trying 127.0.0.1:4000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /home/eliben/eli/private-code-for-blog/2021/tls/cert.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=My Corp
*  start date: Mar 29 13:30:25 2021 GMT
*  expire date: Mar 29 16:30:25 2021 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: O=My Corp
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x557103006e10)
> GET / HTTP/2
> Host: localhost:4000
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< content-type: text/plain; charset=utf-8
< content-length: 33
< date: Mon, 29 Mar 2021 13:31:34 GMT
<
* Connection #0 to host localhost left intact
Proudly served with Go and HTTPS!

success!

We can also talk to our server using a custom HTTPS client written in Go. Here is the code:

func main() 
  addr := flag.String("addr", "localhost:4000", "HTTPS server address")
  certFile := flag.String("certfile", "cert.pem", "trusted CA certificate")
  flag.Parse()

  cert, err := os.ReadFile(*certFile)
  if err != nil 
    log.Fatal(err)
  
  certPool := x509.NewCertPool()
  if ok := certPool.AppendCertsFromPEM(cert); !ok 
    log.Fatalf("unable to parse cert from %s", *certFile)
  

  client := &http.Client
    Transport: &http.Transport
      TLSClientConfig: &tls.Config
        RootCAs: certPool,
      ,
    ,
  

  r, err := client.Get("https://" + *addr)
  if err != nil 
    log.Fatal(err)
  
  defer r.Body.Close()

  html, err := io.ReadAll(r.Body)
  if err != nil 
    log.Fatal(err)
  
  fmt.Printf("%vn", r.Status)
  fmt.Printf(string(html))

The only part that differentiates this from a standard HTTP client is the TLS setup. The important part is defining the RootCAs Field of
tls.Config a building. This tells Go what credentials the customer can trust.

Additional options for producing certificates

You may not be aware that Go comes with a self-signed TLS certificate generation tool, right in the standard installation. If you installed Go on / usr / local / go, You can run this tool with:

$ go run /usr/local/go/src/crypto/tls/generate_cert.go -help

In general, it achieves the same goal as the first code snippet in this post, but while my code snippet makes some opinionated decisions about what to create, gener_cert Can be configured with flags and supports a number of different options.

As we have seen, while self-signed certificates can work for testing, they are not ideal for all scenarios. For example, it is difficult to make browsers trust them, and even then the user experience does not fully replicate that of the “real world”.

Another option for producing local certificates for testing is
Mkcert tool. It creates a local CA (CA), and adds it to your system’s trusted certificate list. It then produces for you certificates signed by this authority, so that as far as the browser is concerned, they are completely reliable.

If we run our simple HTTPS server with a certificate / key generated by
mkcertChrome will happily access it without any warnings; We can also see the details in the Developer Tools Security tab:

Chrome is happy with the HTTPS server

curl Also successfully contact the server without requiring a
cacert Flag, because it already tests the system’s trusted CAs.

If you are looking True Certificates, Let’s Encrypt is of course a natural option, using the certbot Customer or something similar. In Go, directories like certmagic Can automate interaction with Let’s Encrypt for servers.

Client Authentication (mTLS)

So far, the examples we have seen are that the server provides its certificate (signed CA) to the customer to prove that the server is legitimate who he claims to be (e.g. your bank website, before you agree to provide your password).

It is easy to expand this idea Mutual authentication, When the customer also has a signed certificate to prove his identity. In the world of TLS, it’s called mTLS (To mutual TLS), and can be useful in many settings where internal services need to communicate with each other securely. Public key crypto is generally considered more secure than passwords.

Here is a simple HTTPS server with client authentication. The lines that have changed from the previous HTTPS server are marked:

func main() 
  addr := flag.String("addr", ":4000", "HTTPS network address")
  certFile := flag.String("certfile", "cert.pem", "certificate PEM file")
  keyFile := flag.String("keyfile", "key.pem", "key PEM file")
  clientCertFile := flag.String("clientcert", "clientcert.pem", "certificate PEM for client authentication")
  flag.Parse()

  mux := http.NewServeMux()
  mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) 
    if req.URL.Path != "/" 
      http.NotFound(w, req)
      return
    
    fmt.Fprintf(w, "Proudly served with Go and HTTPS!")
  )

  // Trusted client certificate.
  clientCert, err := os.ReadFile(*clientCertFile)
  if err != nil 
    log.Fatal(err)
  
  clientCertPool := x509.NewCertPool()
  clientCertPool.AppendCertsFromPEM(clientCert)

  srv := &http.Server
    Addr:    *addr,
    Handler: mux,
    TLSConfig: &tls.Config
      MinVersion:               tls.VersionTLS13,
      PreferServerCipherSuites: true,
      ClientCAs:                clientCertPool,
      ClientAuth:               tls.RequireAndVerifyClientCert,
    ,
  

  log.Printf("Starting server on %s", *addr)
  err = srv.ListenAndServeTLS(*certFile, *keyFile)
  log.Fatal(err)

The changes are pretty much what you would expect; In addition to setting up its own certificate, key and TLS configuration, the server loads a client certificate and configures TLSConfig Trust it. Of course, this can also be the approval of a trusted local CA that signs customer certificates.

And this is an HTTPS client that authenticates itself when connected to a server; Again, the lines that changed from the previous client (other than mTLS) are checked:

func main() {
  addr := flag.String("addr", "localhost:4000", "HTTPS server address")
  certFile := flag.String("certfile", "cert.pem", "trusted CA certificate")
  clientCertFile := flag.String("clientcert", "clientcert.pem", "certificate PEM for client")
  clientKeyFile := flag.String("clientkey", "clientkey.pem", "key PEM for client")
  flag.Parse()

  // Load our client certificate and key.
  clientCert, err := tls.LoadX509KeyPair(*clientCertFile, *clientKeyFile)
  if err != nil 
    log.Fatal(err)
  

  // Trusted server certificate.
  cert, err := os.ReadFile(*certFile)
  if err != nil 
    log.Fatal(err)
  
  certPool := x509.NewCertPool()
  if ok := certPool.AppendCertsFromPEM(cert); !ok 
    log.Fatalf("unable to parse cert from %s", *certFile)
  

  client := &http.Client
    Transport: &http.Transport
      TLSClientConfig: &tls.Config
        RootCAs:      certPool,
        Certificates: []tls.CertificateclientCert,
      ,
    ,
  

  r, err := client.Get("https://" + *addr)
  if err != nil 
    log.Fatal(err)
  
  defer r.Body.Close()

  html, err := io.ReadAll(r.Body)
  if err != nil 
    log.Fatal(err)
  
  fmt.Printf("%vn", r.Status)
  fmt.Printf(string(html))
}

Before we try this, we’ll need to modify the certificate creation script to create certificates that are appropriate for customers as well. The change is in this line:

ExtKeyUsage:           []x509.ExtKeyUsagex509.ExtKeyUsageServerAuth,

What changes to:

ExtKeyUsage:           []x509.ExtKeyUsagex509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth,

Now let’s do a test run. Start by creating separate credentials / keys for the client and server:

# client cert

$ go run tls-self-signed-cert.go
2021/04/03 05:51:25 wrote cert.pem
2021/04/03 05:51:25 wrote key.pem
$ mv cert.pem clientcert.pem
$ mv key.pem clientkey.pem

# server cert

$ go run tls-self-signed-cert.go
2021/04/03 05:51:42 wrote cert.pem
2021/04/03 05:51:42 wrote key.pem

Running an mTLS server, it should collect the correct files based on the flag’s defaults:

$ go run https-server-mtls.go
2021/04/03 05:54:51 Starting server on :4000

In a separate window, if we run the older client (other than mTLS), we get an error:

$ go run https-client.go
2021/04/03 05:55:24 Get "https://localhost:4000": remote error: tls: bad certificate
exit status 1

And the server log will show that “the client did not provide a certificate”. However, if we run the new mTLS client, it works:

$ go run https-client-mtls.go
200 OK
Proudly served with Go and HTTPS!

While this demonstrates the mechanics of running mTLS servers and clients, in reality there will be much more to do, especially managing credentials, renewing and revoking credentials and trusted credentials. This is called Public Key Infrastructure (PKI), and it’s a big topic outside the scope of this modest post.


Source

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Popular Articles