Chris Bao's Blog

Background

In this article, I’ll write about one topic: how to implement the HTTP protocol. I keep planning to write about this topic for a long time. In my previous articles, I already wrote several articles about HTTP protocol:

• How to write a Golang HTTP server with Linux system calls
• Understand how HTTP/1.1 persistent connection works based on Golang: part one - sequential requests
• Understand how HTTP/1.1 persistent connection works based on Golang: part two - concurrent requests

I recommend you to read these articles above before this one.

As you know, HTTP protocol is in the application layer, which is the closest one to the end-user in the protocol stack.

So relatively speaking, HTTP protocol is not as mysterious as other protocols in the lower layers of this stack. Software engineers use HTTP every day and take it for granted. Have you ever thought about how we can implement a fully functional HTTP protocol library?

It turns out to be a very complex and big work in terms of software engineering. Frankly speaking, I can’t work it out by myself in a short period. So in this article, we’ll try to understand how to do it by investigating Golang net/http package as an example. We’ll read a lot of source code and draw diagrams to help your understanding of the source code.

Note HTTP protocol itself has evolved a lot from HTTP1.1 to HTTP2 and HTTP3, not to mention HTTPS. In this article, we’ll focus on the mechanism of HTTP1.1, but what you learned here can help you understand other new versions of HTTP protocol.

Note HTTP protocol is on the basis of client-server model. This article will focus on the client-side. For the HTTP server part, I’ll write another article next.

Main workflow of http.Client

HTTP client’s request starts from the application’s call to Get method of net/http package, and ends by writing the HTTP message to the TCP socket. The whole workflow can be simplified to the following diagram:

First, the public Get method calls Get method of DefaultClient, which is a global variable of type Client,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// Get method
func Get(url string) (resp *Response, err error) {
	return DefaultClient.Get(url)
}

// DefaultClient is a global variable in net/http package
var DefaultClient = &Client{}

// struct type Client
type Client struct {
	Transport RoundTripper

	CheckRedirect func(req *Request, via []*Request) error

	Jar CookieJar

	Timeout time.Duration
}

Then, NewRequest method is used to construct a new request of type Request:

1
2
3
4
5
6
7
8
9
10
11
12

func (c *Client) Get(url string) (resp *Response, err error) {
	req, err := NewRequest("GET", url, nil)
	if err != nil {
		return nil, err
	}
	return c.Do(req)
}


func NewRequest(method, url string, body io.Reader) (*Request, error) {
	return NewRequestWithContext(context.Background(), method, url, body)
}

I’ll not show the function body of NewRequestWithContext, since it’s very long. But only paste the block of code for actually building the Request object as follows:

1
2
3
4
5
6
7

req := &Request{
    // omit some code 
    Proto:      "HTTP/1.1", // the default HTTP protocol version is set to 1.1
    ProtoMajor: 1,
    ProtoMinor: 1,
    // omit some code
}

Note that by default the HTTP protocol version is set to 1.1. If you want to send HTTP2 request, then you need other solutions, and I’ll write about it in other articles.

Next, Do method is called, which delegates the work to the private do method.

1
2
3

func (c *Client) Do(req *Request) (*Response, error) {
	return c.do(req)
}

do method handles the HTTP redirect behavior, which is very interesting. But since the code block is too long, I’ll not show its function body here. You can refer to the source code of it here.

Next, send method of Client is called which goes as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

func (c *Client) send(req *Request, deadline time.Time) (resp *Response, didTimeout func() bool, err error) {
	if c.Jar != nil {
		for _, cookie := range c.Jar.Cookies(req.URL) {
			req.AddCookie(cookie)
		}
	}
    // call send method here
	resp, didTimeout, err = send(req, c.transport(), deadline)
	if err != nil {
		return nil, didTimeout, err
	}
	if c.Jar != nil {
		if rc := resp.Cookies(); len(rc) > 0 {
			c.Jar.SetCookies(req.URL, rc)
		}
	}
	return resp, nil, nil
}

It handles cookies for the request, then calls the private method send with three parameters.

We already talked about the first parameter above. Let’s take a look at the second parameter c.transport() as follows:

1
2
3
4
5
6

func (c *Client) transport() RoundTripper {
	if c.Transport != nil {
		return c.Transport
	}
	return DefaultTransport
}

Transport is extremely important for HTTP client workflow. Let’s examine how it works bit by bit. First of all, it’s type of RoundTripper interface.

1
2
3
4
5
6
7

// this interface is defined inside client.go file 

type RoundTripper interface {
	// RoundTrip executes a single HTTP transaction, returning
	// a Response for the provided Request.
	RoundTrip(*Request) (*Response, error)
}

RoundTripper interface only defines one method RoundTrip, all right.

If you don’t have any special settings, the DefaultTransport will be used for c.Transport above.

The DefaultTransport is going as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

// defined in transport.go
var DefaultTransport RoundTripper = &Transport{
	Proxy: ProxyFromEnvironment,
	DialContext: (&net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
		DualStack: true,
	}).DialContext,
	ForceAttemptHTTP2:     true,
	MaxIdleConns:          100,
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   10 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
}

Note that its actual type is Transport as below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

type Transport struct {
	idleMu       sync.Mutex
	closeIdle    bool                                // user has requested to close all idle conns
	idleConn     map[connectMethodKey][]*persistConn // most recently used at end
	idleConnWait map[connectMethodKey]wantConnQueue  // waiting getConns
	idleLRU      connLRU
	reqMu       sync.Mutex
	reqCanceler map[cancelKey]func(error)
	altMu    sync.Mutex   // guards changing altProto only
	altProto atomic.Value // of nil or map[string]RoundTripper, key is URI scheme
	connsPerHostMu   sync.Mutex
	connsPerHost     map[connectMethodKey]int
	connsPerHostWait map[connectMethodKey]wantConnQueue // waiting getConns
	Proxy func(*Request) (*url.URL, error)
	DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
	Dial func(network, addr string) (net.Conn, error)
	DialTLSContext func(ctx context.Context, network, addr string) (net.Conn, error)
	DialTLS func(network, addr string) (net.Conn, error)
	TLSClientConfig *tls.Config
	TLSHandshakeTimeout time.Duration
	DisableKeepAlives bool
	DisableCompression bool
	MaxIdleConns int
	MaxIdleConnsPerHost int
	MaxConnsPerHost int
	IdleConnTimeout time.Duration
	ResponseHeaderTimeout time.Duration
	ExpectContinueTimeout time.Duration
	TLSNextProto map[string]func(authority string, c *tls.Conn) RoundTripper
	ProxyConnectHeader Header
	MaxResponseHeaderBytes int64
	WriteBufferSize int
	ReadBufferSize int
	nextProtoOnce      sync.Once
	h2transport        h2Transport // non-nil if http2 wired up
	tlsNextProtoWasNil bool        // whether TLSNextProto was nil when the Once fired
	ForceAttemptHTTP2 bool
}

I list the full content of Transport struct here, although it contains many fields, and many of them will not be discussed in this article.

As we just mentioned, Transport is type of RoundTripper interface, it must implement the method RoundTrip, right?

You can find the RoundTrip method implementation of Transport struct type in roundtrip.go file as follows:

1
2
3
4

// RoundTrip method in roundtrip.go
func (t *Transport) RoundTrip(req *Request) (*Response, error) {
	return t.roundTrip(req)
}

In the beginning, I thought this method should be included inside transport.go file, but it is defined inside another file.

Let’s back to the send method which takes c.Transport as the second argument:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

// send method in client.go 
func send(ireq *Request, rt RoundTripper, deadline time.Time) (resp *Response, didTimeout func() bool, err error) {
	req := ireq // req is either the original request, or a modified fork

	if rt == nil {
		req.closeBody()
		return nil, alwaysFalse, errors.New("http: no Client.Transport or DefaultTransport")
	}

	if req.URL == nil {
		req.closeBody()
		return nil, alwaysFalse, errors.New("http: nil Request.URL")
	}

	if req.RequestURI != "" {
		req.closeBody()
		return nil, alwaysFalse, errors.New("http: Request.RequestURI can't be set in client requests")
	}

	// forkReq forks req into a shallow clone of ireq the first
	// time it's called.
	forkReq := func() {
		if ireq == req {
			req = new(Request)
			*req = *ireq // shallow clone
		}
	}

	// Most the callers of send (Get, Post, et al) don't need
	// Headers, leaving it uninitialized. We guarantee to the
	// Transport that this has been initialized, though.
	if req.Header == nil {
		forkReq()
		req.Header = make(Header)
	}

	if u := req.URL.User; u != nil && req.Header.Get("Authorization") == "" {
		username := u.Username()
		password, _ := u.Password()
		forkReq()
		req.Header = cloneOrMakeHeader(ireq.Header)
		req.Header.Set("Authorization", "Basic "+basicAuth(username, password))
	}

	if !deadline.IsZero() {
		forkReq()
	}
	stopTimer, didTimeout := setRequestCancel(req, rt, deadline)

	resp, err = rt.RoundTrip(req)
	if err != nil {
		stopTimer()
		if resp != nil {
			log.Printf("RoundTripper returned a response & error; ignoring response")
		}
		if tlsErr, ok := err.(tls.RecordHeaderError); ok {
			// If we get a bad TLS record header, check to see if the
			// response looks like HTTP and give a more helpful error.
			// See golang.org/issue/11111.
			if string(tlsErr.RecordHeader[:]) == "HTTP/" {
				err = errors.New("http: server gave HTTP response to HTTPS client")
			}
		}
		return nil, didTimeout, err
	}
	if resp == nil {
		return nil, didTimeout, fmt.Errorf("http: RoundTripper implementation (%T) returned a nil *Response with a nil error", rt)
	}
	if resp.Body == nil {
		// The documentation on the Body field says “The http Client and Transport
		// guarantee that Body is always non-nil, even on responses without a body
		// or responses with a zero-length body.” Unfortunately, we didn't document
		// that same constraint for arbitrary RoundTripper implementations, and
		// RoundTripper implementations in the wild (mostly in tests) assume that
		// they can use a nil Body to mean an empty one (similar to Request.Body).
		// (See https://golang.org/issue/38095.)
		//
		// If the ContentLength allows the Body to be empty, fill in an empty one
		// here to ensure that it is non-nil.
		if resp.ContentLength > 0 && req.Method != "HEAD" {
			return nil, didTimeout, fmt.Errorf("http: RoundTripper implementation (%T) returned a *Response with content length %d but a nil Body", rt, resp.ContentLength)
		}
		resp.Body = ioutil.NopCloser(strings.NewReader(""))
	}
	if !deadline.IsZero() {
		resp.Body = &cancelTimerBody{
			stop:          stopTimer,
			rc:            resp.Body,
			reqDidTimeout: didTimeout,
		}
	}
	return resp, nil, nil
}

At line 50 of send method above:

1

resp, err = rt.RoundTrip(req)

RoundTrip method is called to send the request. Based on the comments in the source code, you can understand it in the following way:

• RoundTripper is an interface representing the ability to execute a single HTTP transaction, obtaining the Response for a given Request.

Next, let’s go to roundTrip method of Transport:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122

// roundTrip method in transport.go, which is called by RoundTrip method internally 

// roundTrip implements a RoundTripper over HTTP.
func (t *Transport) roundTrip(req *Request) (*Response, error) {
	t.nextProtoOnce.Do(t.onceSetNextProtoDefaults)
	ctx := req.Context()
	trace := httptrace.ContextClientTrace(ctx)

	if req.URL == nil {
		req.closeBody()
		return nil, errors.New("http: nil Request.URL")
	}
	if req.Header == nil {
		req.closeBody()
		return nil, errors.New("http: nil Request.Header")
	}
	scheme := req.URL.Scheme
	isHTTP := scheme == "http" || scheme == "https"
	if isHTTP {
		for k, vv := range req.Header {
			if !httpguts.ValidHeaderFieldName(k) {
				req.closeBody()
				return nil, fmt.Errorf("net/http: invalid header field name %q", k)
			}
			for _, v := range vv {
				if !httpguts.ValidHeaderFieldValue(v) {
					req.closeBody()
					return nil, fmt.Errorf("net/http: invalid header field value %q for key %v", v, k)
				}
			}
		}
	}

	origReq := req
	cancelKey := cancelKey{origReq}
	req = setupRewindBody(req)

	if altRT := t.alternateRoundTripper(req); altRT != nil {
		if resp, err := altRT.RoundTrip(req); err != ErrSkipAltProtocol {
			return resp, err
		}
		var err error
		req, err = rewindBody(req)
		if err != nil {
			return nil, err
		}
	}
	if !isHTTP {
		req.closeBody()
		return nil, badStringError("unsupported protocol scheme", scheme)
	}
	if req.Method != "" && !validMethod(req.Method) {
		req.closeBody()
		return nil, fmt.Errorf("net/http: invalid method %q", req.Method)
	}
	if req.URL.Host == "" {
		req.closeBody()
		return nil, errors.New("http: no Host in request URL")
	}

	for {
		select {
		case <-ctx.Done():
			req.closeBody()
			return nil, ctx.Err()
		default:
		}

		// treq gets modified by roundTrip, so we need to recreate for each retry.
		treq := &transportRequest{Request: req, trace: trace, cancelKey: cancelKey}
		cm, err := t.connectMethodForRequest(treq)
		if err != nil {
			req.closeBody()
			return nil, err
		}

		// Get the cached or newly-created connection to either the
		// host (for http or https), the http proxy, or the http proxy
		// pre-CONNECTed to https server. In any case, we'll be ready
		// to send it requests.
		pconn, err := t.getConn(treq, cm)
		if err != nil {
			t.setReqCanceler(cancelKey, nil)
			req.closeBody()
			return nil, err
		}

		var resp *Response
		if pconn.alt != nil {
			// HTTP/2 path.
			t.setReqCanceler(cancelKey, nil) // not cancelable with CancelRequest
			resp, err = pconn.alt.RoundTrip(req)
		} else {
			resp, err = pconn.roundTrip(treq)
		}
		if err == nil {
			resp.Request = origReq
			return resp, nil
		}

		// Failed. Clean up and determine whether to retry.
		if http2isNoCachedConnError(err) {
			if t.removeIdleConn(pconn) {
				t.decConnsPerHost(pconn.cacheKey)
			}
		} else if !pconn.shouldRetryRequest(req, err) {
			// Issue 16465: return underlying net.Conn.Read error from peek,
			// as we've historically done.
			if e, ok := err.(transportReadFromServerError); ok {
				err = e.err
			}
			return nil, err
		}
		testHookRoundTripRetried()

		// Rewind the body if we're able to.
		req, err = rewindBody(req)
		if err != nil {
			return nil, err
		}
	}
}

There are three key points:

• at line 70, a new variable of type transportRequest, which embeds Request, is created.
• at line 81, getConn method is called, which implements the cached connection pool to support the persistent connection mode. Of course, if no cached connection is available, a new connection will be created and added to the connection pool. I will explain this behavior in detail next section.
• from line 89 to line 95, pconn.roundTrip is called. The name of variable pconn is self-explaining which means it is type of persistConn.

transportRequest is passed as parameter to getConn method, which returns pconn. pconn.roundTrip is called to execute the HTTP request. we have covered all the steps in the above workflow diagram.

Summary

In this first article of this series, we talked about the workflow of sending an HTTP request step by step. And I’ll discuss how to send the HTTP message to the TCP stack in the second article.

Background

In the last post, I show you how HTTP/1.1 persistent connection works in a simple demo app, which sends sequential requests.

We observe the underlying TCP connection behavior based on the network analysis tool: netstat and tcpdump.

In this article, I will modify the demo app and make it send concurrent requests. In this way, we can have more understanding about HTTP/1.1’s persistent connection.

Concurrent requests

The demo code goes as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"net/http"
	"sync"
	"time"
)

func startHTTPserver() {

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(time.Duration(50) * time.Microsecond)
		fmt.Fprintf(w, "Hello world")
	})

	go func() {
		http.ListenAndServe(":8080", nil)
	}()

}

func startHTTPRequest(index int, wg *sync.WaitGroup) {
	counter := 0
	for i := 0; i < 10; i++ {
		resp, err := http.Get("http://localhost:8080/")
		if err != nil {
			panic(fmt.Sprintf("Error: %v", err))
		}
		io.Copy(ioutil.Discard, resp.Body) // fully read the response body
		resp.Body.Close()                  // close the response body
		log.Printf("HTTP request #%v in Goroutine #%v", counter, index)
		counter += 1
		time.Sleep(time.Duration(1) * time.Second)
	}
	wg.Done()
}

func main() {
	startHTTPserver()
	var wg sync.WaitGroup
	for i := 0; i < 10; i++ {
		wg.Add(1)
		go startHTTPRequest(i, &wg)
	}
	wg.Wait()
}

We create 10 goroutines, and each goroutine sends 10 sequential requests concurrently.

Note: In HTTP/1.1 protocol, concurrent requests will establish multiple TCP connections. That’s the restriction of HTTP/1.1, the way to enhance it is using HTTP/2 which can multiplex one TCP connection for multiple parallel HTTP connections. HTTP/2 is not in the scope of this post. I will talk about it in another article.

Note that in the above demo, we have fully read the response body and closed it, and based on the discussion in last article, the HTTP requests should work in the persistent connection model.

Before we use the network tool to analyze the behavior, let’s imagine how many TCP connections will be established. As there are 10 concurrent goroutines, 10 TCP connections should be established, and all the HTTP requests should re-use these 10 TCP connections, right? That’s our expectation.

Next, let’s verify our expectation with netstat as follows:

It shows that the number of TCP connections is much more than 10. The persistent connection does not work as we expect.

After reading the source code of net/http package, I find the following hints:

The Client is defined inside client.go which is the type for HTTP client, and Transport is one of the properties.

1
2
3
4
5
6
7
8
9

type Client struct {
	Transport RoundTripper

	CheckRedirect func(req *Request, via []*Request) error

	Jar CookieJar

	Timeout time.Duration
}

Transport is defined in transport.go like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// DefaultTransport is the default implementation of Transport and is
// used by DefaultClient. It establishes network connections as needed
// and caches them for reuse by subsequent calls. It uses HTTP proxies
// as directed by the $HTTP_PROXY and $NO_PROXY (or $http_proxy and
// $no_proxy) environment variables.
var DefaultTransport RoundTripper = &Transport{
	Proxy: ProxyFromEnvironment,
	DialContext: (&net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
	}).DialContext,
	ForceAttemptHTTP2:     true,
	MaxIdleConns:          100,
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   10 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
}

// DefaultMaxIdleConnsPerHost is the default value of Transport's
// MaxIdleConnsPerHost.
const DefaultMaxIdleConnsPerHost = 2

Transport is type of RoundTripper, which is an interface representing the ability to execute a single HTTP transaction, obtaining the Response for a given Request. RoundTripper is a very important structure in net/http package, we’ll review (and analyze) the source code in the next article. In this article, we’ll not discuss the details.

Note that there are two parameters of Transport:

• MaxIdleConns: controls the maximum number of idle (keep-alive) connections across all hosts.
• MaxIdleConnsPerHost: controls the maximum idle (keep-alive) connections to keep per-host. If zero, DefaultMaxIdleConnsPerHost is used.

By default, MaxIdleConns is 100 and MaxIdleConnsPerHost is 2.

In our demo case, ten goroutines send requests to the same host (which is localhost:8080). Although MaxIdleConns is 100, but only 2 idle connections can be cached for this host because MaxIdleConnsPerHost is 2. That’s why you saw much more TCP connections are established.

Based on this analysis, let’s refactor the code as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"net/http"
	"sync"
	"time"
)

var (
	httpClient *http.Client
)

func init() {
	httpClient = &http.Client{
		Transport: &http.Transport{
			MaxIdleConnsPerHost: 10, // set connection pool size for each host
			MaxIdleConns:        100,
		},
	}
}

func startHTTPserver() {

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(time.Duration(50) * time.Microsecond)
		fmt.Fprintf(w, "Hello world")
	})

	go func() {
		http.ListenAndServe(":8080", nil)
	}()

}

func startHTTPRequest(index int, wg *sync.WaitGroup) {
	counter := 0
	for i := 0; i < 10; i++ {
		resp, err := httpClient.Get("http://localhost:8080/")
		if err != nil {
			panic(fmt.Sprintf("Error: %v", err))
		}
		io.Copy(ioutil.Discard, resp.Body) // fully read the response body
		resp.Body.Close()                  // close the response body
		log.Printf("HTTP request #%v in Goroutine #%v", counter, index)
		counter += 1
		time.Sleep(time.Duration(1) * time.Second)
	}
	wg.Done()
}

func main() {
	startHTTPserver()
	var wg sync.WaitGroup
	for i := 0; i < 10; i++ {
		wg.Add(1)
		go startHTTPRequest(i, &wg)
	}
	wg.Wait()
}

This time we don’t use the default httpClient, instead we create a customized client which sets MaxIdleConnsPerHost to be 10. This means the size of the connection pool is changed to 10, which can cache 10 idle TCP connections for each host.

Verify the behavior with netstat again:

Now the result is what we expect.

Summary

In this article, we discussed how to make HTTP/1.1 persistent connection work in a concurrent case by tunning the parameters for the connection pool. In the next article, let’s review the source code to study how to implement HTTP client.

Background

Initially, HTTP was a single request-and-response model. An HTTP client opens the TCP connection, requests a resource, gets the response, and the connection is closed. And establishing and terminating each TCP connection is a resource-consuming operation (in detail, you can refer to my previous article). As the web application becomes more and more complex, displaying a single page may require several HTTP requests, too many TCP connection operations will have a bad impact on the performance.

So persistent-connection (which is also called keep-alive) model is created in HTTP/1.1 protocol. In this model, TCP connections keep open between several successive requests, and in this way, the time needed to open new connections will be reduced.

In this article, I will show you how persistent connection works based on a Golang application. We will do some experiments based on the demo app, and verify the TCP connection behavior with some popular network packet analysis tools. In short, After reading this article, you will learn:

• Golang http.Client usage (and a little bit source code analysis)
• network analysis with netstat and tcpdump

You can find the demo Golang application in this Github repo.

Sequential requests

Let’s start from the simple case where the client keeps sending sequential requests to the server. The code goes as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"
)

func startHTTPserver() {

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(time.Duration(50) * time.Microsecond)
		fmt.Fprintf(w, "Hello world")
	})

	go func() {
		http.ListenAndServe(":8080", nil)
	}()

}

func startHTTPRequest() {
	counter := 0
	for i := 0; i < 10; i++ {
		_, err := http.Get("http://localhost:8080/")
		if err != nil {
			panic(fmt.Sprintf("Error: %v", err))
		}
		log.Printf("HTTP request #%v", counter)
		counter += 1
		time.Sleep(time.Duration(1) * time.Second)
	}
}

func main() {
	startHTTPserver()

	startHTTPRequest()
}

We start an HTTP server in a Goroutine, and keep sending ten sequential requests to it. Right? Let’s run the application and check the numbers and status of TCP connections.

After running the above code, you can see the following output:

When the application stops running, we can run the following netstat command:

1

netstat -n  | grep 8080

The TCP connections are listed as follows:

Obviously, the 10 HTTP requests are not persistent since 10 TCP connections are opened.

Note: the last column of netstat shows the state of TCP connection. The state of TCP connection termination process can be explained with the following image:

I will not cover the details in this article. But we need to understand the meaning of TIME-WAIT.

In the four-way handshake process, the client will send the ACK packet to terminate the connection, but the state of TCP can’t immediately go to CLOSED. The client has to wait for some time and the state in this waiting process is called TIME-WAIT. The TCP connection needs this TIME-WAIT state for two main reasons.

• The first is to provide enough time that the ACK is received by the other peer.
• The second is to provide a buffer period between the end of current connection and any subsequent ones. If not for this period, it’s possible that packets from different connections could be mixed. In detail, you can refer to this book.

In our demo application case, if you wait for a while after the program stops, and run the netstat command again then no TCP connection will be listed in the output since they’re all closed.

Another tool to verify the TCP connections is tcpdump, which can capture every network packet send to your machine. In our case, you can run the following tcpdump command:

1

sudo tcpdump -i any -n host localhost

It will capture all the network packets send from or to the localhost (we’re running the server in localhost, right?). tcpdump is a great tool to help you understand the network, you can refer to its document for more help.

Note: in our demo code above, we send 10 HTTP requests in sequence, which will make the capture result from tcpdump too long. So I modified the for loop to only send 2 sequential requests, which is enough to verify the behavior of persistent connection. The result goes as follows:

In tcpdump output, the Flag [S] represents SYN flag, which is used to establish the TCP connection. The above snapshot contains two Flag [S] packets. The first Flag [S] is triggered by the first HTTP call, and the following packets are HTTP request and response. Then you can see the second Flag [S] packet to open a new TCP connection, which means the second HTTP request is not persistent connection as we hope.

Next step, let’s see how to make HTTP work as a persistent connection in Golang.

In fact,this is a well known issue in Golang ecosystem, you can find the information in the official document:

• If the returned error is nil, the Response will contain a non-nil Body which the user is expected to close. If the Body is not both read to EOF and closed, the Client’s underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent “keep-alive” request.

The fix will be straightforward by just adding two more lines of code as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"net/http"
	"time"
)

func startHTTPServer() {

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(time.Duration(50) * time.Microsecond)
		fmt.Fprintf(w, "Hello world")
	})

	go func() {
		http.ListenAndServe(":8080", nil)
	}()

}

func startHTTPRequest() {
	counter := 0
	for i := 0; i < 10; i++ {
		resp, err := http.Get("http://localhost:8080/")
		if err != nil {
			panic(fmt.Sprintf("Error: %v", err))
		}
		io.Copy(ioutil.Discard, resp.Body) // read the response body
		resp.Body.Close() // close the response body
		log.Printf("HTTP request #%v", counter)
		counter += 1
		time.Sleep(time.Duration(1) * time.Second)
	}
}

func main() {
	startHTTPServer()

	startHTTPRequest()
}

let’s verify by running netstat command, the result goes as follows:

This time 10 sequential HTTP requests establish only one TCP connection. This behavior is just what we hope: persistent connection.

We can double verify it by doing the same experiment as above: run two HTTP requests in sequence and capture packets with tcpdump:

This time, only one Flag [S] packet is there! The two sequential HTTP request re-use the same underlying TCP connection.

Summary

In this article, we showed how HTTP persistent connection works in the case of sequential requests. In the next article, we can show you the case of concurrent requests.

Background

I like system programming which can allow you to touch more software development skills in the bottom level.

Linux is the perfect platform when you want to do system programming. But if you’re using a computer running Windows on it, then you have to spend some time to set up the Linux development environment. Generally speaking there are two traditional ways to do that: virtual machine and dualboot, both need some effort. Or you can try to do that with container technology, for example, I once shared one article about how to do it with Docker.

In this article, I will introduce a new and easier way to do this without too much overhead.

Windows Subsystem for Linux

The new way is Windows Subsystem for Linux (WSL). I have to admit that the operating system is complex and difficult so for now I don’t know how Microsoft make WSL works. In details, you can refer to this article to learn how WSL allows Windows to access Linux files. In this article let’s focus on how to set it up and what kind of benefits it can provide to developers.

Based on the official document, with WSL you can

• Run common command-line tools such as grep, sed, awk.
• Run Bash shell scripts and GNU/Linux command-line applications including:
  • Tools: vim, emacs, tmux.
  • Languages: NodeJS, Javascript, Python, Ruby, C/C++, C# & F#, Rust, Go, etc.
  • Services: SSHD, MySQL, Apache, lighttpd, MongoDB, PostgreSQL.
• Install additional software using your own GNU/Linux distribution package manager.

With these conditions, you can set up a completed Linux development environment.

Install WSL

For detail steps to install WSL, you can find it on the official document. Based on my experience, I follow the document to download and install Linux Ubuntu distribution smoothly, which is much easier than settig the virtual machine.

File mount

By default, you can also access your local machine’s file system from within the Linux Bash shell. Since your local drives are mounted under the /mnt folder of the subsystem.

In this way, you can develop the code with the productivity tools in Windows and build it in Linux environment.

Network

This is another convenient point. WSL shares the IP address of Windows, as it is running on Windows.
As such you can access any ports on localhost e.g. if you had a web server running on port 8080, you could access it just by visiting http://localhost:8080 into your Windows browser.

Set up the development environment

After install the Ubuntu system, I also install tools to prepare the development environment. For example, GCC to develop C language program as below.

Summary

Based on my testing and experience, WSL can save developers’ time to set up Linux environment.

Background

HTTP is everywhere. As a software engineer, you’re using the HTTP protocol every day. Starting an HTTP server will be an easy task if you’re using any modern language or framework. For example, in Golang you can do that with the following lines of code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/hi", func(w http.ResponseWriter, r *http.Request){
        fmt.Fprintf(w, "Hi")
    })

    log.Fatal(http.ListenAndServe(":8081", nil))
}

You can finish the job easily because net/http package implements the HTTP protocol completely. How can you do that without net/http package ? That’s the target of this article.

Note: This article is inspired by Joe Schafer’s post a lot. My implementation has something different which totally removes dependency on Golang’s net package, but the idea of using system call in Golang to setup the TCP/IP connetion is the same. Thanks very much for Joe Schafer’s interesting post.

Another thing I need to mention is this article will cover many concepts, but it’s very difficult to discuss all of them in detail. To understand this article smoothly, you need some prerequisite knowledge such as OSI model, TCP/IP stack, socket programming, HTTP protocol and system call. I will add some explanations on these topics to help you understand this article and give some references and links to let you continue exploring more in advanced level.

OSI network model

OSI model partitions the data flow in a communication system into seven abstraction layers. These layers form a protocol stack, with each layer communicating with the layer above and the layer below as follows:

For example, HTTP is in layer 7, TCP is in layer 4 and IP is in layer 3.

OSI is a general model, which was first specified in the early 1980s. But neither traditional nor modern networking protocols fit into this model neatly. For example, TCP/IP stack does not define the three upper layers: session, presentation, and application. In fact, it does not define anything above the transport layer. From the viewpoint of TCP/IP, everything above the transport layer is part of
the application. So the layered network model more consistent with Linux (TCP/IP stack is implemented in Linux kernel) is as follows:

• Application Layer (telnet, ftp, http)
• Host-to-Host Transport Layer (TCP, UDP)
• Internet Layer (IP and routing)
• Network Access Layer (Ethernet, wi-fi)

Once again, it is important to point out that the upper layers—Layers 5, 6, and 7—are not part of the TCP/IP stack.

Another critical point to understand is data encapsulation. The data flow goes from the bottom physical level to the highest-level representation of data in an application.

Each layer has administrative information that it has to keep about its own layer. It does this by adding header information to the packet it receives from the layer above, as the packet passes down. Each header contains information regarding the message contents. For example, one HTTP server sends data from one host to another. It uses the TCP protocol on top of the IP protocol, which may be sent over Ethernet. This looks like:

The packet transmitted over ethernet, is the bottom one. On the receiving side, these headers are removed as the packet moves up.

Next let’s see how TCP/IP stack encapsulates HTTP message and send it over the network through socket. The idea can be illustrated with the following image:

I will explain how it works by writing a HTTP server from scratch, you can refer to this Github repo to get all the code.

TCP/IP

TCP/IP stack is originated from ARPANET project, which is integrated into Unix BSD OS as the first implementation of TCP/IP protocols.

Nowadays, TCP/IP is still implemented in the operating system level. For Linux system, you can find the source code inside the kernel. The detailed implementation is outside the scope of this article. You can study it in this Github link.

Socket

As I mentioned in the above sections, HTTP server is running in the application level. How it can work with TCP/IP stack which lives in the kernel? The answer is socket.

The socket interface was originally developed as part of the BSD operating system. Sockets provide an interface between the application level programs and the TCP/IP stack. Linux (or other OS) provides an API and sockets, and applications use this API to access the networking facilities in the kernel.

The socket interface is really TCP/IP’s window on the world. In most modern systems incorporating TCP/IP, the socket interface is the only way that applications make use of the TCP/IP suite of protocols.

One main advantage of sockets in Unix or Linux system is that the socket is treated as a file descriptor, and all the standard I/O functions work on sockets in the same way they work on a local file. File descriptor is simply an integer associated with an open file.

You may heard everything in Unix is a file. The file can be a network connection, a pipe, a real file in the disk, a device or anything else. So when you want to send data to another program over the Interent you will do it through a file descriptor.

In our HTTP server case, it will get the request by reading data from the socket and send the response by writing data to the socket.

Next, let’s review the source code to see how the HTTP server is implemented.

First, we need setup the TCP connection through socket, the process can be described in the following image:

In Golang, net package provides all the socket related functionalities. Since this article’s purpose is writing a HTTP server from scratch, so I create a package named simplenet to provide the very basic implementation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

package simplenet

import (
	"os"
	"syscall"
)

type netSocket struct {
	fd int
}

func NewNetSocket(ip IP, port int) (*netSocket, error) {
	// ForkLock docs state that socket syscall requires the lock.
	syscall.ForkLock.Lock()
	// AF_INET = Address Family for IPv4
	// SOCK_STREAM = virtual circuit service
	// 0: the protocol for SOCK_STREAM, there's only 1.
	fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM, 0)
	if err != nil {
		return nil, os.NewSyscallError("socket", err)
	}
	syscall.ForkLock.Unlock()

	// Allow reuse of recently-used addresses.
	if err = syscall.SetsockoptInt(fd, syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1); err != nil {
		syscall.Close(fd)
		return nil, os.NewSyscallError("setsockopt", err)
	}

	// Bind the socket to a port
	sa := &syscall.SockaddrInet4{Port: port}
	copy(sa.Addr[:], ip)
	if err = syscall.Bind(fd, sa); err != nil {
		return nil, os.NewSyscallError("bind", err)
	}

	// Listen for incoming connections.
	if err = syscall.Listen(fd, syscall.SOMAXCONN); err != nil {
		return nil, os.NewSyscallError("listen", err)
	}

	return &netSocket{fd: fd}, nil
}

func (ns netSocket) Read(p []byte) (int, error) {
	if len(p) == 0 {
		return 0, nil
	}
	n, err := syscall.Read(ns.fd, p) // read from socket file descriptor
	if err != nil {
		n = 0
	}
	return n, err
}

func (ns netSocket) Write(p []byte) (int, error) {
	n, err := syscall.Write(ns.fd, p) // write to socket file descriptor
	if err != nil {
		n = 0
	}
	return n, err
}

// Creates a new netSocket for the next pending connection request.
func (ns *netSocket) Accept() (*netSocket, error) {
	// syscall.ForkLock doc states lock not needed for blocking accept.
	nfd, _, err := syscall.Accept(ns.fd)
	if err == nil {
		syscall.CloseOnExec(nfd)
	}
	if err != nil {
		return nil, err
	}
	return &netSocket{nfd}, nil
}

func (ns *netSocket) Close() error {
	return syscall.Close(ns.fd)
}

netSocket data model is created to represent the socket, which contains only one field fd means file descriptor. And all the socket related APIs: Read, Write, Accept and Close, are defined. The usage of socket API is not in this article’s scope, you can easily find a lot of great documents about it online.

The logic of netSocket is not complicated, because it delegates the job to the kernel by system call. A system call is a programmatic way a program requests a service from the kernel, in detail you can refer to this article. In Golang, all the system calls are wrapped inside the syscall standard package.

One thing need to mention is different platform have different syscall usages, so the demo code shown in this article can only be compiled and build on Linux system.

Now we setup the TCP server and wait for connection request from client side. Next, let’s see how to read or write HTTP request and response through socket.

HTTP

The main workflow is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

import (
	"flag"
	"http-server-scratch/simplenet"
	"log"
)

func main() {
	ipFlag := flag.String("ip_addr", "127.0.0.1", "The IP address to use")
	portFlag := flag.Int("port", 8080, "The port to use.")
	flag.Parse()

	ip := simplenet.ParseIP(*ipFlag)
	port := *portFlag
	socket, err := simplenet.NewNetSocket(ip, port)
	defer socket.Close()
	if err != nil {
		panic(err)
	}

	log.Print("===============")
	log.Print("Server Started!")
	log.Print("===============")
	log.Print()
	log.Printf("addr: http://%s:%d", ip, port)

	for {
		// Block until incoming connection
		rw, e := socket.Accept()
		log.Print()
		log.Print()
		log.Printf("Incoming connection")
		if e != nil {
			panic(e)
		}

		// Read request
		log.Print("Reading request")
		req, err := simplenet.ParseRequest(rw)
		log.Print("request: ", req)
		if err != nil {
			panic(err)
		}

		// Write response
		log.Print("Writing response")
		simplenet.WriteString(rw, "HTTP/1.1 200 OK\r\n"+
			"Content-Type: text/html; charset=utf-8\r\n"+
			"Content-Length: 20\r\n"+
			"\r\n"+
			"<h1>hello world</h1>")
		if err != nil {
			log.Print(err.Error())
			continue
		}
	}
}

As you can see, the HTTP request parsing logic is defined in the ParseRequest method in simplenet package.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

package simplenet

import (
	"bufio"
	"errors"
	"http-server-scratch/simplenet/simpleTextProto"
	"io"
	"strconv"
	"strings"
)

type request struct {
	method string // GET, POST, etc.
	header simpleTextProto.MIMEHeader
	body   []byte
	uri    string // The raw URI from the request
	proto  string // "HTTP/1.1"
}

func ParseRequest(c *netSocket) (*request, error) {
	b := bufio.NewReader(*c)
	tp := simpleTextProto.NewReader(b) // need replace
	req := new(request)

	// Parse request line: parse "GET /index.html HTTP/1.0"
	var s string
	s, _ = tp.ReadLine() // need replace
	sp := strings.Split(s, " ")
	req.method, req.uri, req.proto = sp[0], sp[1], sp[2]

	// Parse request headers
	mimeHeader, _ := tp.ReadMIMEHeader() // need replace
	req.header = mimeHeader

	// Parse request body
	if req.method == "GET" || req.method == "HEAD" {
		return req, nil
	}
	if len(req.header["Content-Length"]) == 0 {
		return nil, errors.New("no content length")
	}
	length, err := strconv.Atoi(req.header["Content-Length"][0])
	if err != nil {
		return nil, err
	}
	body := make([]byte, length)
	if _, err = io.ReadFull(b, body); err != nil {
		return nil, err
	}
	req.body = body
	return req, nil
}

The HTTP request message can be divided into three parts request line, request headers and request body as follows:

The logic inside ParseRequest handles these 3 parts step by step. You can refer to the comments in the demo code.

One thing need to emphasis is that ParseRequest method doesn’t depends on net package. Because I want to show how HTTP server works in the bottom level, so I copy the request parsing logics from net package into my simplenet package. The parsing for request header part is kind of complex, but it doesn’t influence your understanding about the main concept of HTTP server. If you want to know the details, you can refer to the simplenet/simpleTextProto package. The important thing to understand is HTTP server reads the request message with Read method of netSocket . And the Read method makes socket read system call to get network data from TCP stack:

1

syscall.Read(ns.fd, p)

On the other side, HTTP response is sent back by calling WriteString method of simplenet package

1
2
3

func WriteString(c *netSocket, s string) (n int, err error) {
	return c.Write([]byte(s))
}

WriteString simply calls Write method of netsocket, which makes socket write system call to send data over Interent with TCP stack:

1

syscall.Write(ns.fd, p)

That’s all for the code part. Next let’s try to run this simple HTTP server we build from scratch.

Demo

Build (need Linux platform) and run this HTTP server with default options setting and send request to it with curl. The result goes as follows:

the server works as expected.