In this article, I will share what I learned about .NET memory management by troubleshooting a memory leak issue for a .NET service running in a real production environment. After reading this post, you can understand more about the following items:

• .NET Memory Management and .NET Garbage Collection
• Diagnose memory usage with WinDbg

Previously I once published articles about memory allocation of native applications written in C on Linux systems. I highly recommend you read both articles and compare what’s the difference, this can deepen your understanding of memory-related techniques.

Background

Firstly, let me explain the background: the problematic .NET service is running on the Azure Service Fabric, which consists of a cluster of Windows Server. If you are not familiar with Service Fabric, please refer to my previous article, which explains how to develop microservices based on it. The problem is that the memory usage of this .NET service reaches roughly 1.5GB, which is 4 times higher than the average usage. Generally speaking, on the cloud more resource usage means higher cost. Next, let’s diagnose what is happening here. But before we roll up our sleeves to debug this issue, let’s first review how .NET memory works at a theoretical level.

.NET Memory Management

.NET Runtime

Unlike native software written directly to the operating system’s APIs, programs written with .NET are called managed applications because they depend on a runtime and framework that manages many of their vital tasks and ensures a basic safe operating environment. In .NET, runtime refers to the Common Language Runtime(CLR) and the framework is .NET Framework or .NET Core. And the .NET application is built on top of them.

CLR works as a virtual execution engine for .NET applications, and it is a very complex topic that is out of the scope of this article, so we can’t dive into the details. But one of the crucial functionalities it provides is just: memory management.

.NET GC

When you’re working on system programming with C or C++, you need to manage the memory by calling standard library’s APIs like malloc and free to allocate and release the memory. But on the .NET framework, your life will be easier since the .NET memory manager does this task for you. When the program creates a new object, the memory manager will allocate the memory for you. Easy, right? But that’s only a trivial part of the memory manager, the complicated task is how and when the memory manager can decide to collect the memory. Because of this, the .NET memory manager has another name: .NET Garbage Collector.

The .NET GC uses two Win32 functions VirtualAlloc and VirtualFree for allocating a segment of memory and releasing segments back to the operating system respectively. The entire process can be divided into the following three phases:

• Marking Phase: A list of live objects is created during the marking phase. This is done by starting from a set of references called the GC roots. GC marks these root objects as live and then looks at any object that they reference and marks these as being live too. And the GC continues the search iteratively. This process is also called reachability analysis. All of the objects that are not on the reachable list are potentially released from the memory.

• Relocating Phase: The references of all the reachable objects are updated so that they point to the new location. The purpose of the relocating phase is to optimize memory usage by compacting the live objects closer together. This helps reduce memory fragmentation and improves memory allocation efficiency.

• Compacting Phase: The memory space occupied by the dead objects is released and the live objects are moved to the new location.

GC algorithm is a complex topic, the above description only touches the surface of it. You can explore more by yourself. In the following sections, I will add extra information when necessary.

WinDbg

To diagnose .NET applications’ memory issues, there are many modern tools, like dotMemory. But in this article, I will show you a low-level tool: WinDbg. WinDbg is a multipurpose debugger for the Windows OS, which can be used to debug both user mode and kernel mode. Originally WinDbg was used to debug the native applications, but WinDbg allows the loading of extensions which can augment the debugger’s supported commands. For example, to debug .NET applications running on CLR, we need the SOS extension.

Now that we have a powerful debugger in hand, the next question is how to use it. Generally speaking, you can attach the debugger to the problematic service, but since the target service is running in the production environment, we can’t do that easily. So we need to troubleshoot this issue in the offline style as follows:

We need to use ProcDump to dump the memory information into a file, called dump file. Then use WinDbg to analyze the dump file. I will not cover the usage of ProcDump in this article and leave it to you.

Next, let’s play with the dump file, all right? Let’s go!

Diagnose the memory leak

First, let’s use the WinDbg !address extension to display information about the memory that the target process or target computer uses as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    338     7ffe`d06bb000 ( 127.995 TB)          100.00%
<unknown>                              4715        0`feb0f000 (   3.980 GB)  83.90%    0.00%
Stack                                   276        0`16880000 ( 360.500 MB)   7.42%    0.00%
Image                                  1241        0`0e989000 ( 233.535 MB)   4.81%    0.00%
Heap                                    519        0`0b997000 ( 185.590 MB)   3.82%    0.00%
Other                                    15        0`001cd000 (   1.801 MB)   0.04%    0.00%
TEB                                      92        0`000b8000 ( 736.000 kB)   0.01%    0.00%
PEB                                       1        0`00001000 (   4.000 kB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            5051        1`1e9d4000 (   4.478 GB)  94.41%    0.00%
MEM_IMAGE                              1753        0`1013b000 ( 257.230 MB)   5.30%    0.00%
MEM_MAPPED                               55        0`00e26000 (  14.148 MB)   0.29%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                338     7ffe`d06bb000 ( 127.995 TB)          100.00%
MEM_RESERVE                            2503        0`cdc3c000 (   3.215 GB)  67.78%    0.00%
MEM_COMMIT                             4356        0`61cf9000 (   1.528 GB)  32.22%    0.00%

Let’s examine the output, which provides so much information!

The first section shows the memory Usage Summary:

• Free: is the entire virtual memory that can potentially be claimed from the operating system. This may include swap space, not only physical RAM. For a 64-bit process, the virtual memory is 128TB.
• Heap: what you see as Heap is the memory that was allocated through the Windows Heap Manager, we generally call it the native heap.
• Unknown: any other heap managers will implement their own memory management. Basically, they all work similarly: they get large blocks of memory from VirtualAlloc and then try to do a better management of the small block within that large block. Since WinDbg doesn’t know any of these memory managers, that memory is declared as Unknown. It includes but is not limited to the managed heap of .NET. In my case, the value is 3.98GB which is much bigger than the 1.5GB memory usage reported by the monitor tool. I will explain why it goes like this in the following sections.
• Image: is the memory space allocated for the binary assemblies.
• Stack: is straightforward. If you don’t know, please refer to my previous article.

The second section shows the usage by memory Type: MEM_PRIVATE, MEM_IMAGE and MEM_MAPPED. MEM_PRIVATE is private to one process and MEM_IMAGE and MEM_MAPPED can be shared among multiple processes.

The third section shows the usage by memory State.

• MEM_COMMIT: committed memory refers to the portion of the virtual address space that has been allocated and backed by physical memory or the paging file. Committed memory is actively used and accessible by the process. It includes memory regions that have been allocated and are in use, such as code, data, and heap allocations.
• MEM_RESERVE: corresponds to reserved memory. Reserved memory refers to memory that has been reserved in the virtual address space but has not yet been committed. When memory is reserved, the address space is allocated, but physical memory or paging file resources are not immediately allocated. The process reserves the address space to ensure that it will be available when needed. Reserved memory can later be committed to make it usable by the process.
• MEM_FREE: represents free memory. This category includes all memory that has not been reserved or committed.

In my case, the committed memory is 1.528GB, which exactly matches the monitored memory usage. This can perfectly explain the confusing point mentioned about the 3.98GB unknown memory segment. It turns out the majority of these memories are only reserved. So next step is to get the actual committed memory usage for various types. How to achieve that?

Thanks for the powerful address extension, I can do that by adding some filters like this:

1

!address -f:MEM_COMMIT -c:".echo %1 %3 %5 %7"

This command will output all the committed memory regions, and for each region prints its base address(%1), region size(%3), state(%5) and type(%7). The output goes as follows:

1
2
3
4
5
6
7
8
9

0x5f3a0000 0x1000 MEM_COMMIT Image
0x5f3a1000 0x2ab000 MEM_COMMIT Image
0x5f64c000 0xb000 MEM_COMMIT Image
0x5f657000 0x1b000 MEM_COMMIT Image
0x5f672000 0x2000 MEM_COMMIT Image
0x5f674000 0x2d000 MEM_COMMIT Image
0x7ffe0000 0x1000 MEM_COMMIT Other
0x4580ca7000 0x5000 MEM_COMMIT Stack
...thousands of lines omitted...

I wrote a simple script to parse the above output and finally got the following result:

1
2
3
4
5
6
7
8

Memory by Type:
type Image:  233.52344MB
type Other:  1.8007812MB
type Stack:  3.796875MB
type TEB  :  0.71875MB
type PEB  :  0.00390625MB
type Heap :  84.88281MB
type <unknown> :  1240.2461MB

Based on this result, you can see the majority comes from the unknown segment. I suspect that the memory leak issue occurs in this problematic service. Various WinDbg commands can diagnose the memory leak problem. And in this specific case, I find the command !finalizequeue is super helpful!

Before I show you the output of the command, let’s examine what the finalizer queue is. In C#, the finalizer(also referred to as destructor) is used to perform any necessary final clean-up when a class instance is being collected by the GC.

1
2
3
4
5
6
7

class Car
{
    ~Car()  // finalizer
    {
        // cleanup statements...
    }
}

When your application encapsulates unmanaged resources, such as windows, files and network connections, you should use the finalizer to free those resources.

Objects of classes with finalizers can’t be removed immediately: they instead go to the finalizer queue and are removed from memory once the finalizer has been run.

Based on this, let’s examine the output of command !finalizequeue:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

0:000> !FinalizeQueue
SyncBlocks to be cleaned up: 0
Free-Threaded Interfaces to be released: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
----------------------------------
generation 0 has 66 finalizable objects (0000021c729c47b0->0000021c729c49c0)
generation 1 has 4 finalizable objects (0000021c729c4790->0000021c729c47b0)
generation 2 has 4372 finalizable objects (0000021c729bbef0->0000021c729c4790)
Ready for finalization 0 objects (0000021c729c49c0->0000021c729c49c0)
Statistics for all finalizable objects (including all objects ready for finalization):
              MT    Count    TotalSize Class Name
00007ffcde443f78        1           32 Microsoft.Win32.SafeHandles.SafePEFileHandle
00007ffcde4407f8        1           32 System.Security.Cryptography.X509Certificates.SafeCertContextHandle

--- many lines of output omitted ---

00007ffcde447db0       97         3104 Microsoft.Win32.SafeHandles.SafeWaitHandle
00007ffcdd272468       28         4928 System.Diagnostics.PerformanceCounter
00007ffcde45d6a8       64         6144 System.Threading.Thread
00007ffcde4297c0     3639       262008 System.Reflection.Emit.DynamicResolver
Total 4442 objects

You can notice that there are 3639 objects of type System.Reflection.Emit.DynamicResolver. As we know, the objects in the finalizer queue can’t be removed until the finalizer runs. This also means that any object they reference and any object referenced by those, and so on has to be kept in memory.

This is the potential reason for the memory leak problem. System.Reflection.Emit is a low-level library used to generate the Microsoft Intermediate Language (MSIL) dynamically, my application does not rely on it. Finally, it turns out the issue is from the Service Fabric SDK and upgrading to new versions can fix it.

Summary

In this article, we examined how .NET GC works and how to diagnose the memory leak issue with WinDbg.

I plan and hope to set up a bright new career in the open-source software field, that’s the motivation to join the program!

Thanks to the award from Linux Foundation, I got the opportunity to select one training course and certification exam freely. Big thanks.

And the one I choose is CKAD (Certified Kubernetes Application Developer), stay tuned, I will update the status later.

Recently, the engineering team at Amazon Prime Video posted an article that becomes super popular in the software community. Many discussions arose concerning how to design a scalable application in the modern cloud computing era.

Scalability is one critical metric for designing and developing an online application. Technically it is a challenging task, but the cloud made it easy; because the public cloud services providers like AWS and Azure do the job for you. Virtual machine, Container , and Serverless, you have so many powerful technologies to scale your application and business.

But the problem is you need to choose the one suitable for your application. So in this article, let’s examine the details of why Aamzon refactors the application and how they do it.

Since Amazon doesn’t expose the code implementation of this project, our analysis is based on the original post and the technical understanding of AWS. All right?

Case study

The product refactored is called Amazon Prime Video Monitoring Service, which monitors the quality of thousands of live video streams. This monitoring tool runs the real-time analysis of the streams and detects quality issues. The entire process consists of 3 steps: media converter, defect detector and real-time notification.

• media converter: converting input audio/video streams to frames.
• defect detector: executing machine learning algorithms that analyze frames to detect defects.
• real-time notification: sending real-time notifications whenever a defect is found.

The old architecture goes as follows:

In this microservices architecture, each step was implemented by AWS Lambda serverless service (I assume you already know what is serverless; if not, please refer to other online documents). And the entire workflow was orchestrated by AWS Step Functions, which is a serverless orchestration service.

AWS Step Functions State Transition

The first bottleneck is just coming from AWS Step Functions. But before discussing the issues of this architecture, we need to understand what is AWS Step Functions and how it works basically. This knowledge is very critical to understand the performance bottlenecks later.

AWS Step Functions is a serverless service that allows you to coordinate and orchestrate multiple AWS serverless functions using a state machine. You can define the workflow of serverless applications as a state machine, which represents the different states and transitions of the application’s execution.

The State machine can be thought of as a directed graph, where each node represents a state, and each edge represents a transition between states, which is used to model complex systems. The topic of state machine isn’t in the scope of this article, I will write a post about it in the future. So far, you only need to know each state in the state machine represents a specific step in the workflow, while transitions represent the events that trigger the transition from one step to another. You can find many examples of AWS Step Functions in this repo, please take a look at what problems you can solve with it.

In theory, this serverless-based microservices architecture can scale out easily. However, as the original post mentioned it “hit a hard scaling limit at around 5% of the expected load. Also, the overall cost of all the building blocks was too high to accept the solution at a large scale.“

So the bottleneck is the cost! Because AWS Step Functions charges users per state transition, and in this monitoring service case, it performed multiple state transitions for every second of the video stream, it quickly hit the account limits!

As a software engineer working in a team, maybe you can just focus on writing great codes, but when you need to select the tech stack, you have to consider the cost, especially when your application is running on the cloud.

AWS S3 Buckets Tier-1 requests

As mentioned above, the media converter service splits videos into frames, and the defect detector service loads and analyzes the frames later. So in the original architect, the frame images are stored in the Amazon S3 bucket. As the original post mentioned “Defect detectors then download images and processed them concurrently using AWS Lambda. However, the high number of Tier-1 calls to the S3 bucket was expensive.”

So the second bottleneck is also a cost issue. But what is a Tier-1 call or request in the context of AWS S3?

A Tier-1 request refers to an API request that retrieves or lists objects in an S3 bucket, and is charged at a higher rate than other types of API requests. AWS S3 API requests are classified into two categories: standard requests and Tier-1 requests.

• standard requests: including API requests such as PUT, COPY, DELETE and HEAD requests.
• Tier-1 requests: including API requests such as GET and LIST requests.

Tier-1 requests are expensive because they involve retrieving and listing objects in an AWS S3 bucket, which is more resource-intensive. Because when you retrieve or list objects, S3 needs to scan through the entire bucket to find the targets. Additionally, S3 needs to transfer the data for each retrieved object over the network. So basically, it consumes more storage and network resources on the cloud.

Monolith Architecture

Based on these two bottlenecks, they refactored this monitoring tool and returned to the monolith architecture as follows:

In the new design, everything is running inside a single process host in Amazon Elastic Container Service(ECS). In this monolith architecture, the frames are stored in the memory instead of S3 buckets. It doesn’t need serverless orchestration service either.

How does this new architecture run at a high scale? They directly scale out and partition the Amazon ECS cluster. In this way, they get a scalable monitoring application with a 90% cost reduction.

Summary

There is no perfect application architecture that can fit all cases. In the cloud computing era, you need to understand the service you used better than before and make a wise decision.

In this article, I want to introduce a distributed systems platform: Service Fabric, that is used to build microservices.

First things first, what is Service Fabric? In the software world, Service Fabric is in the scope of the orchestrator. So it is the competitor of Kubernetes, Docker Swarm, etc.

I know what you’re thinking about, at the time of writing, Kubernetes already won the competition. Why am I still writing about Service Fabric? My motivation to write this post is as follows:

• Firstly, I once used Service Fabric to build microservices and learned something valuable about it. I want to summarize all my learnings and share them with you here!
• Secondly, we can (simply) examine both Service Fabric and Kubernetes to understand why the cloud-native solution is better and what problems it can solve.
• Finally, Service Fabric is still widely used in some enterprise applications, in detail, you can refer here. So the skills you learned about service fabric are valuable.

Hands-on Project

To learn the Service Fabric platform, we will examine an open-source project: HealthMetrics. This project is originally developed by Microsoft itself to demonstrate the power of service fabric and I added some enhancements to it, in detail please refer to this GitHub repo.

At the business requirement level, this application goes like this: each patient keeps reporting the heart rate data to the doctor via the wearable device(the band in this case). And the doctor aggregates the collected health data and reports to the county service. Each county service instance runs some statistics on doctors’ data and sends the report to the country service. Finally, you can get something as follows:

Before we explore the detailed implementation, you can pause here for a few minutes. How will you design the architecture if this application is assigned to you? How to make it both reliable and scalable?

Service Fabric Programming Model

Service Fabric provides multiple ways to write and manage your services:

• Reliable Services: is the most popular choice in the context of service fabric, we’ll examine much more about it later.
• Containers: service fabric can also deploy services in containers. But if you choose to use containers, why not directly use Kubernetes? I’ll not cover it in this post.
• Guest executables: You can run any type of code or script like Node.js as guest executables in the service fabric. Note that the service fabric platform doesn’t include the runtime for the guest executables, instead, the operating system hosting the service fabric must provide the corresponding runtime for the given language. It provides a flexible way to run legacy applications within microservices. If you have an interest in it, please refer to this article, I will not examine the details in this post.

Next, let’s examine what are Reliable Services.

Reliable Services

As I mentioned above, service fabric is an orchestrator which provides infrastructure-level functionalities like cluster management, service discovery, and scaling as Kubernetes does. But service fabric goes further by also providing a reliable services model to guide the development on the application level, this’s a special point compared to Kubernetes. Simply speaking, the application code can call the service fabric runtime APIs to query the system for building reliable applications. In the following sections, I’ll show you what that means with some code blocks.

Reliable Services can be classified into two types as follows:

• Stateless service: when we mentioned stateless service in the context of service fabric, we are not saying that the service doesn’t have any state to store, but it means that the service doesn’t store the data inside the cluster of service fabric. Indeed, the stateless service can store states in the external database. The naming of stateless is relative to the cluster.
• Stateful service: similarly, the stateful service keeps its state locally in the service fabric cluster, which means the data and service are in the same virtual or physical machine. And this is called Reliable Collections. Compared with external data storage, Reliable Collections implies low latency and high performance. In the following section, I will introduce more about Reliable Collections, which is a very interesting topic. Please hold on!

Besides stateless and stateful services listed above, there is the third type of service provided by Service Fabric: Actor Service.

• Actor Service: is a special type of stateful service, which applies the actor model theory. The actor model is a general computer science theory handling concurrent computation. It is a very big topic worthy of a separate article. I’ll not cover the details in this post, instead, let’s go through this model in the context of service fabric.

The actor model is proposed many years ago in 1973 to simplify the process of writing concurrent programs with the following several advantages:

• Encapsulation: in the actor model, each actor is a self-contained unit that encapsulates its own state and behavior. And each actor runs only one thread, so you don’t have to worry about complex multi-threading programming issues. But how does it support high concurrency? Just allocate more actor instances as the load goes up.
• Message passing: different from the traditional multi-threading programming techniques, actors do not share memories, instead, they communicate with async message-passing communication, which can reduce the complexity of concurrent programs.
• Location transparency: as a self-contained computation unit, each actor can be located on different machines, which makes it the perfect solution for building distributed applications as service fabric does.

In the future, I will write an article on the actor model to examine more details about it. Next, let’s take a look at the demo app: HealthMetrics and analyze how it was built based on the programming models we discussed above.

Architecture of HealthMetrics

There are several services included in this HealthMetrics demo application. They are:

• BandCreationService: this stateless service read the input CSV file and creates the individual band actors and doctor actors. For example, as the above snapshot shows, the application creates roughly 300 band actors and doctor actors.
• BandActor: this actor service is the host for the band actors. Each BandActor represents an individual wearable device, which generates heart rate data and then sends it to the designated DoctorActor every 5 seconds.
• DoctorActor: the doctor actor service aggregates all of the data it receives from each band actor and generates an overall view for that doctor and then pushes it into the CountyService every 5 seconds.
• CountyService: this stateful service aggregates the information provided by the doctor actor further and also pushes the data into the NationalService.
• NationalService: this stateful service maintains the total aggerated data for the entire county and is used to serve data requested by WebService.
• WebService: this stateless service just hosts a simple web API to query information from NationalService and render the data on the web UI.

As you can see, this demo application consists of all three types of services provided by service fabric, which is a perfect case to learn service fabric, right?

In the next sections, let’s examine what kinds of techniques of service fabric are applied to build this highly scalable and available microservices application.

Naming Perspective

Naming plays an important role in the distributed system. They are used to share resources, to uniquely identify entities, to refer to locations, and so on. In the distributed system, each name should be resolved to the entity it refers to. To resolve names, it is necessary to implement a naming system.

In Service Fabric, naming is used to identify and locate services and actors within a cluster. And Service Fabric uses a structured naming system, where services and actors are identified using a hierarchical naming scheme. The naming system is based on the concept of a namespace. A namespace is a logical grouping of related applications and services. In the Service Fabric namespace, the first level is the application, which represents a logical grouping of services, and the second level is the service, which represents a specific unit of functionality that is part of the application. And the default namespace is fabric:/, so the service URI in the Service Fabric cluster is in the following format:

1

fabric:/applicationName/ServiceName

And in our HealthMetrics demo app, the URL will be something like fabric:/HealthMetrics/HealthMetrics.BandActor or fabric:/HealthMetrics/HealthMetrics.CountyService(the application name is HealthMetrics and the service name is HealthMetrics.BandActor or HealthMetrics.CountyService).

In this demo app, we build a helper class ServiceUriBuilder to generate URI as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public Uri ToUri()
{
    string applicationInstance = this.ApplicationInstance;

    if (String.IsNullOrEmpty(applicationInstance))
    {
        try
        {
            // the ApplicationName property here automatically prepends "fabric:/" for us
            applicationInstance = FabricRuntime.GetActivationContext().ApplicationName.Replace("fabric:/", String.Empty);
        }
        catch (InvalidOperationException)
        {
            // FabricRuntime is not available. 
            // This indicates that this is being called from somewhere outside the Service Fabric cluster.
        }
    }

    return new Uri("fabric:/" + applicationInstance + "/" + this.ServiceInstance);
}

For detail, please refer to this source code file.

Since the behavior of the name resolver is quite similar to the internet domain name resolver, the Service Fabric Name Service can be regarded as an internal DNS service. This type of internal DNS service is a common requirement for all distributed systems, including Kubernetes. In the modern cloud-native ecosystem, the popular choice is CoreDNS, which runs inside K8S. In the future, I will write an article about DNS. Please keep watching my blog’s update.

Besides this default DNS solution mentioned above, you can use other service registry and service discovery solutions. For example, in my previous article, I once examined how to do this based on Consul. Imagine your large-scale application consists of hundreds of microservices, with the default DNS solutions you have to hardcode so many names in each service. That’s just where Consul can help. In detail, I’ll not repeat it here, please refer to my previous article!

Now that we understand how the naming system works, let’s examine how to do inter-service communication based on that.

Communication Perspective

Inter-service communication is at the heart of all distributed systems. For nondistributed platforms, communication between processes can be done by sharing memories. But the communication between processes on different machines has always been based on the low-level message passing as offered by the underlying network. There are two widely used models for communications in distributed systems: Remote Procedure Call(RPC) and Message-Oriented Middleware(MOM).

• Remote Procedure Call(RPC): is ideal for client-server applications and aims at hiding message-passing details from the client. The HTTP protocol is a typical client-server model, so HTTP requests can be thought of as a type of RPC. In the client-server model, the client and server must be online at the same time to exchange information, so it’s often called synchronous communication.

• Message-Oriented Middleware(MOM): is suitable for distributed applications, where the communication does not follow the strict pattern of a client-server model. In this model, there are three parties involved: the message producer, the message consumer, and the message broker. In this case, when the message producer publishes the message, the message consumer doesn’t have to be online, the message broker will store the message until the consumer becomes available to receive it, so this model is also called asynchronous communication.

The Service Fabric supports RPC out of the box. In our HealthMetrics demo app, you can find the RPC calls and HTTP requests easily. For example, when the BandActor and the DoctorActor are created, the RPC call is used in the BandCreationService as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13

private async Task CreateBandActorTask(BandActorGenerator bag, CancellationToken cancellationToken)
{
    // omit some code for simplicity
    ActorId bandActorId;
    ActorId doctorActorId;
    bandActorId = new ActorId(Guid.NewGuid()); 
    doctorActorId = new ActorId(bandActorInfo.DoctorId);
    IDoctorActor docActor = ActorProxy.Create<IDoctorActor>(doctorActorId, this.DoctorServiceUri);
    await docActor.NewAsync(doctorName, randomCountyRecord);
    IBandActor bandActor = ActorProxy.Create<IBandActor>(bandActorId, this.ActorServiceUri);
    await bandActor.NewAsync(bandActorInfo);
    // omit some code for simplicity
}

You can see the ActorProxy instance is created as the RPC client, and the NewAsync method of BandActor and DoctorActor service is called. For example, the NewAsync method of BandActor goes like this:

1
2
3
4
5
6
7
8
9
10
11

public async Task NewAsync(BandInfo info)
{
    await this.StateManager.SetStateAsync<CountyRecord>("CountyInfo", info.CountyInfo);
    await this.StateManager.SetStateAsync<Guid>("DoctorId", info.DoctorId);
    await this.StateManager.SetStateAsync<HealthIndex>("HealthIndex", info.HealthIndex);
    await this.StateManager.SetStateAsync<string>("PatientName", info.PersonName);
    await this.StateManager.SetStateAsync<List<HeartRateRecord>>("HeartRateRecords", new List<HeartRateRecord>()); // initially the heart rate records are empty list
    await this.RegisterReminders();

    ActorEventSource.Current.ActorMessage(this, "Band created. ID: {0}, Name: {1}, Doctor ID: {2}", this.Id, info.PersonName, info.DoctorId);
}

You can ignore the detailed content of this method for now, I will explain in a later section.

And when the DoctorActor service reports its status, it calls the CountyService endpoint with HTTP requests:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

public async Task SendHealthReportToCountyAsync()
{
    // omit some code
    await servicePartitionClient.InvokeWithRetryAsync(
        client =>
        {
            Uri serviceAddress = new Uri(
                client.BaseAddress,
                string.Format(
                    "county/health/{0}/{1}",
                    partitionKey.Value.ToString(),
                    id));

            HttpWebRequest request = WebRequest.CreateHttp(serviceAddress);
            request.Method = "POST";
            request.ContentType = "application/json";
            request.KeepAlive = false;
            request.Timeout = (int) client.OperationTimeout.TotalMilliseconds;
            request.ReadWriteTimeout = (int) client.ReadWriteTimeout.TotalMilliseconds;

            using (Stream requestStream = request.GetRequestStream())
            {
                using (BufferedStream buffer = new BufferedStream(requestStream))
                {
                    using (StreamWriter writer = new StreamWriter(buffer))
                    {
                        JsonSerializer serializer = new JsonSerializer();
                        serializer.Serialize(writer, payload);
                        buffer.Flush();
                    }

                    using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
                    {
                        ActorEventSource.Current.Message("Doctor Sent Data to County: {0}", serviceAddress);
                        return Task.FromResult(true);
                    }
                }
            }
        }
    );
    // omit some code
}

In the demo app, no asynchronous communication is used. But you can easily integrate the message broker middleware with the Service Fabric. This kind of microservice design is called Event-Driven Architecture. In the future, I will write another article about it, please keep watching my blog!

Scalability Perspective

Scalability has become one of the most important design goals for developers of distributed systems. A system can be scalable, meaning that we can easily add more users and resources to the system without any noticeable loss of performance. In most cases, scalability problems in distributed systems appear as performance problems caused by the limited capacity of servers and networks.

Simply speaking, there are two types of scaling techniques: scaling up and scaling out:

• scaling up: is the process by which a machine is equipped with more and often more powerful resources(e.g., by increasing memory, upgrading CPUs, or replacing network modules) so that it can better accommodate performance-demanding applications. However, there are limits to how much you can scale up a single machine, and at some point, it may become more cost-effective to scale out.
• scaling out: is all about extending a networked computer system with more computers and subsequently distributing workloads across the extended set of computers. There are basically only three techniques we can apply: asynchronous communication, replication and Partitioning.

We already examined asynchronous communication in the last section. Next, let’s take a deep look at the other two:

• Replication: is a technique, which replicates more components or resources, etc., across a distributed system. Replication not only increases availability; but also helps to balance the load between components, leading to better performance. In Service Fabric, no matter whether it is a stateless or stateful service, you can replicate the service across multiple nodes. As the workload of your application increases, Service Fabric will automatically distribute the load. But replication can only boost performance for read requests (which don’t change data); if you need to optimize the performance for write requests (which change data), you need Partitioning.

• Partitioning: is an important scaling technique, which involves taking a component or other resource, splitting it into smaller parts, and subsequently spreading those parts across the system. Each partition only contains a subset of the entire dataset. This can help reduce the amount of data that needs to be processed and accessed by each partition, which can lead to faster processing times and improved performance. In addition to reducing the size of the data set, partitioning can also improve concurrency and reduce contention.

In Service Fabric, each partition consists of a replica set with a single primary replica and multiple active secondary replicas. Service Fabric makes sure to distribute replicas of partitions across nodes so that secondary replicas of a partition do not end up on the same node as the primary replica, which can increase the availability.

The difference between a primary replica and a secondary replica is that the primary replica can handle both read and write requests, while the secondary replica can only handle read requests. Moreover, by default, the read requests are only handled by the primary replica, if you want to balance the read requests among all the secondary replicas, you need to set ListenOnSecondary at the application code level. You can set the number of replicas with MinReplicaSetSize and TargetReplicaSetSize

And when the client needs to call the service with multiple partitions, the client needs to generate a ServicePartitionKey; and send requests to the service with this partition key. For example, the DoctorActor sends the health report to the county service with multiple partitions as follows:

1
2
3
4
5
6
7
8
9
10
11

public async Task SendHealthReportToCountyAsync() {
    // omit some code 
    ServicePartitionKey partitionKey = new ServicePartitionKey(countyRecord.CountyId);
    ServicePartitionClient<HttpCommunicationClient> servicePartitionClient =
    new ServicePartitionClient<HttpCommunicationClient>(
        this.clientFactory,
        this.countyServiceInstanceUri,
        partitionKey);
    await servicePartitionClient.InvokeWithRetryAsync();
    // omit some code
}

You can see the partition key is generated based on the county id field. Service Fabric provides several different partition schemas. In this case, ranged partitioning is used, where the partition key is an integer.

The county service specifies an integer range by a Low Key ( set to 0) and High Key (set to 57000). It also defines the number of partitions as PartitionCount (set to 3). All the integer keys are evenly distributed among the partitions. So the partitions of the county service go as follows:

As I mentioned above, the county id is unique, we could then generate a hash code based on the id field, then modulus the key range, to finally get the partition key. Service Fabric runtime will direct the requests to the target node based on that partition key.

Summary

In this post, we quickly examined some interesting topics about Service Fabric. I have to admit that Service Fabric is a big project, what I examined here only covers a small portion of the entire system. Feel free to explore more!

In this series of articles, I want to examine an important data structure: the red-black tree. The red-black tree is an advanced data structure that is difficult to fully understand. Maybe you have some wonders and confusion about it as follows:

• What is the meaning of red and black here?
• The red-black tree is known as a self-balancing binary search tree. But what’s the difference between it and others?
• Where is it used or applied?

If you want to know the answer to the above questions, then this article is just for you. I will cover the following topics:

• Why do we need the red-black tree and what is its advantage?
• How does a red-black tree work?
• How to write a red-black tree from scratch?

Background of Red-Black Tree

In this section, let’s review the history of the red-black tree. During this process, I will show you why it was invented and what kind of advantages it can provide compared with other tree data structures.

First thing first, let’s define the red-black tree as follows:

• Red-Black Tree is a self-balancing binary search tree.

Let’s analyze this definition step by step:

• Tree: A tree is a nonlinear data structure, compared to arrays, linked lists, stacks and queues which are linear data structures. A tree is a data structure consisting of one node called the root and zero or one or more subtrees. One disadvantage of linear data structures is the time required to search a linear list is proportional to the size of the data set, which means that the time complexity is O(n). That’s why more efficient data structures like trees are invented to store and search data.

• General Tree: A general tree is a tree where each node may have zero or more children.

• Binary Tree: A binary tree is a specialized case of a general tree, where each node can have no more than two children.

• Binary Search Tree: A binary tree satisfying the binary search property is called a binary search tree(BST). To build a BST, the node with a key greater than any particular node is stored on the right sub-trees and the one equal to or less than is stored on the left sub-tree.

The average search time complexity of BST is O(logN), but in the worst case, it will be degraded to O(N). It happens when we insert the nodes one by one in order. For example, if we insert the elements in array [1, 2, 3, 4, 5, 6] into RST in order, what we get is as follows:

This kind of unbalanced BST is degraded to a single linked list. So the BST needs to keep balanced during the insert and delete of the node. That’s where the self-balancing binary search tree comes from.

• Self-Balancing Binary Search Tree: it’s a binary search tree that automatically keeps its height small in the face of arbitrary item insertions and deletions. And the red-black tree is just one type of it.

Summary

As the first post of this series, I examined the background of the red-black tree bit by bit. I hope you can understand what’s self-balancing binary search tree and why we need it. In the next post, I will start to examine the behavior and property of the red-black tree.

Today, in this article I want to examine one concrete topic: how to delete a node from a binary search tree. This question is important for understanding other tree data structures, like the AVL tree, and the Red-Black tree.

As is common with many data structures, deletion is always the hardest operation. For example, to delete a specific element from the array, we need to shift all the elements after it by one index. And to delete one node from the linked list, we need to reset the pointer of the previous node (if it’s a doubled linked list, we need to reset more points, which is a more complex case). This is the same for the binary search tree as well. Let’s see in the following section.

Note: the idea of this article is inspired by the book Data Structures and Algorithm Analysis in C written by Mark Allen Weiss. The demo code shown in the following section is from this book. And I made some changes based on it. I highly recommend this great book to readers.

Deletion of the binary search tree

Before we can examine the deletion operation in depth, let’s quickly review the concept of the binary search tree as follows:

• Binary search tree: is a binary tree holding the following property that for every node, X, in the tree, the values of all the keys in its left subtree are smaller than the key value in X, and the values of all the keys in its right subtree are larger than the key value in X.

Simply speaking, the binary search tree is a binary tree satisfying the binary search property. So each node of a binary search tree can have two children subtrees at most. When we need to delete one node from the binary search tree, then it’s necessary to consider the following 3 cases.

If the node is a leaf, then it can be deleted immediately. For example, to delete node 1 from the following binary search tree:

I will show you how to implement it at the code level later, which summarizes all the cases.

If the node has only one child subtree, the node can be deleted after we reset its parent’s pointer to bypass the node and point to its child node. For example, to delete node 4 as follows:

In the above example, node 4 has only one left child node 3. Let’s consider the symmetric scenario, imagine what will happen when it has only one right child node. The answer is it doesn’t influence how we handle the deleted node here and the result is the same. I will not draw the diagram here and leave it for you to explore.

The complicated case is how to deal with a node with two children. Before we introduce the solution, let’s clarify one concept about the binary search tree: successor:

• Successor: is the node with the minimum value in the right subtree of any node.

Let’s take node 16 in the above binary search tree as an example, the successor is node 17. All right! Based on this concept, the solution to delete a node with two children is straightforward:

• Replace the data of the deleted node with the value of the successor and recursively delete the successor from the right subtree.

Let’s try to analyze this solution. Firstly, replacing the data of the node with the value of the successor can keep the binary search property after the deletion operation. Secondly, since the successor has the minimum value of the subtree, which means it cannot have a left child. So the successor is either a leaf node without any child node or a node with only one right child node. So recursively deleting the successor can be resolved by the two simple cases we discussed above, it’s perfect, right? For instance, the deletion of node 16 goes as follows:

As we mentioned above, node 16‘s successor is node 17. So replace the value with 17 and delete node 17 from the right subtree, where node 17 is just a leaf node. Next, let’s implement the node deletion operation.

Code implementation

In this article, I will only show the codes related to the deletion operation rather than the complete implementation of a binary search tree. If you want to know how to write a BST from scratch, please refer to this GitHub repo.

Firstly, let’s examine the header file, which contains the data type definitions and function declarations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#include "utility.h"
#include <stdio.h>
#include <stdlib.h>

#ifndef _BST_H
#define _BST_H

typedef int ET;

struct TreeNode;
typedef struct TreeNode *Position;
typedef Position BST;

BST delete(ET, BST);
Position findMinBST(BST);

// code omitted here
#endif

You can notice that besides the delete function, we also define a helper function findMinBST which is used to find the successor node.

Next, let’s examine the function definitions as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

BST
delete(ET elem, BST T)
{
    Position tmpCell;
  
    if (T == NULL)
        fatal("Element not found");
    else if (elem < T->Element)
        T->Left = delete(elem, T->Left);
    else if (elem > T->Element)
        T->Right = delete(elem, T->Right);
    else // we found the element to be deleted
    if (T->Left != NULL && T->Right != NULL) // two children
    {
        tmpCell = findMinBST(T->Right);
        T->Element = tmpCell->Element;
        T->Right = delete(T->Element, T->Right);
    }
    else // one or zero children, change the pointer T pointing to new address(NULl or the child node)
    {
        tmpCell = T;
        // for leaf node, T will be reset to null
        if (T->Right == NULL)
            T = T->Left;
        else if (T->Left == NULL)
            T = T->Right;
        free(tmpCell);
    }
    return T;
}

Position
findMinBST(BST T)
{
  if (T == NULL)
    return NULL;
  else if (T->Left == NULL)
    return T;
  else
    return findMinBST(T->Left);
}

I add some comments in the above code block which can help your understanding of this recursive algorithm. Please go ahead and think hard about it.

In the next section, we’ll have some open discussions about this solution. Let’s see whether there is any other solution. And what’re the potential issues of the current solution?

Open discussion

successor vs predecessor

Firstly, in the above solution, we delete the node with two children based on the successor. And there is the other concept called predecessor:

• Predecessor: is the node with the maximum value in the left subtree of any node.

So similarly, the alternative solution to delete the node with two children is:

• Replace the data of the deleted node with the value of the predecessor and recursively delete the predecessor from the left subtree.

We can do this by writing another helper function findMaxBST as follows:

1
2
3
4
5
6
7
8

Position
findMaxBST(BST T)
{
  if (T != NULL)
    while (T->Right != NULL)
      T = T->Right;
  return T;
}

Does it work? The answer is yes. But the performance of the solution based on the predecessor is worse than the one based on the successor. Because the predecessor has the maximum value of the left subtree, it means that the predecessor can have two children. Then when we delete the predecessor recursively, the worst-case time complexity can reach O(logN) while the solution based on the successor only requires constant(O(1)) time. That’s the difference.

balanced vs unbalanced

Although the above solution can work, it exposes a serious performance issue. The reason why people invent binary search tree data structures is that we can get O(logN) level performance for searching operations. But imagine what will happen if we keep inserting and deleting nodes in the binary search tree in the way we mentioned here. The depth of the tree will become unbalanced. The left subtree grows deeper than the right subtree because we are always replacing a deleted node with the successor(which is from the right subtree, right?)

The following image is borrowed from Mark’s great book I mentioned above, which clearly shows that the tree becomes unbalanced after many rounds of insertion and deletion. If you want to know about it in theory, please refer to the book.

For an unbalanced BST, the worst-case time complexity can be degraded to O(n). To keep the desired performance, people invent a more advanced data structure self-balancing binary search tree, like the AVL tree and Red-black tree. I will share about them in the coming articles, please keep watching my blog.

Summary

In this article, we examined various solutions to delete a node from the binary search tree and evaluated their performance. We also discussed some open questions about BST, which prove why we need more advanced data structures like the red-black tree.

In this last article, we examined how external mergesort works in theory, let’s implement it now.

First, you can find all the source codes of this implementation in this Github repo. In this repo, I implemented both two-way and multi-way solutions, which are tracked in different branches, please be aware of this. But for the sake of simplicity, I will focus on the generalized multi-way mergesort solution in this article.

Data Preparation

Before diving into the code, let’s define the problem we need to solve here. I will generate an input file containing several millions of seven digits random numbers, from 1,000,000 to 9,999,999. The random numbers can be duplicated, and each number is stored in one new line of the input file. The input file can be prepared with the following Bash script which calls GNU shuf :

1
2
3
4
5
6
7

#!/usr/bin/bash 

# Author: Chris Bao
# Generate millions of seven digits random integers
# based on shuf utility

shuf -i 1000000-9999999 -n 7777777 > ./input.txt

The generated input file is roughly 60 MB in size. For modern computers, it can be loaded to memory easily. But since we are working external memory algorithm, so let’s assume we are running the algorithm on an old computer, which has only 100000 Byte memory. Based on this assumed restriction, we need to sort the numbers in the input file and save the result in a new file.

Implementation

Let’s define some global constants in this algorithm:

1
2
3
4
5
6
7
8

#ifndef CONSTANT_H
#define CONSTANT_H

#define MEMORY_LIMIT 100000
#define RECORD_SIZE 4
#define MULTI_WAY_NUMBER 2

#endif 

MEMORY_LIMIT denotes the 100000 bytes memory limit; In C, we can use the type unsigned int to store an integer in the range from 1,000,000 to 9,999,999. So RECORD_SIZE means each record(or integer) will take up 4 bytes of memory.

And by default, the algorithm will use the two-way merge, but the user can pass an argument to run a multi-way merge too.

Sort phase

The sort phase is implemented inside the separationSort function as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

/*
 * Goes through a given file and separates that file into sorted 1MB files using (internal) mergeSort algorithm
 * */
void separationSort(FILE *input) {
    FILE *fp;
    unsigned int *buffer = malloc(sizeof(unsigned int)*(MEMORY_LIMIT/RECORD_SIZE));
    char *line = NULL;
    size_t len = 0;
    ssize_t nread;
    int count = 0;
    printf("Sort phase start.\n");
    while((nread = getline(&line, &len, input)) != -1) {
        if (count < MEMORY_LIMIT/RECORD_SIZE) {
            buffer[count++] = (unsigned int)strtoul(line, NULL, 10); 
        } else {
            mergeSort(buffer, count); // sort records
            
            // output sorted to file
            if (fileNum == 1) { // create the dir
                int status;
                // create tmp directory
                if ((status = mkdir("./tmp", S_IRWXU | S_IRWXU | S_IROTH | S_IXOTH)) == -1) {
                    fprintf(stderr, "Failed to create tmp directory.\n");
                    exit(EXIT_FAILURE);
                }
                // create pass0 directory for sort phase
                if ((status = mkdir("./tmp/pass0", S_IRWXU | S_IRWXU | S_IROTH | S_IXOTH)) == -1) {
                    fprintf(stderr, "Failed to create pass0 directory.\n");
                    exit(EXIT_FAILURE);
                }
 
            }

            char fileName[20];
            sprintf(fileName, "./tmp/pass0/%d.txt", fileNum);
            if ((fp = fopen(fileName, "w+")) == NULL) {
                fprintf(stderr, "Failed to create file: %s.\n", fileName);
                exit(EXIT_FAILURE);
            }

            outputToFile(buffer, count, fp);

            // Reset memory buffer(zero-out the entire array)
            memset(buffer, 0, sizeof(unsigned int)*(MEMORY_LIMIT/RECORD_SIZE));
            count = 0;
            fileNum++;
            buffer[count++] = (unsigned int)strtoul(line, NULL, 10); // add the current record into new buffer's as first element
            
        }
    }

    // sort the last and final file
    mergeSort(buffer, count);
    char fileName[20];
    sprintf(fileName, "./tmp/pass0/%d.txt", fileNum);
    if ((fp = fopen(fileName, "w+")) == NULL) {
        fprintf(stderr, "Failed to create file: %s.\n", fileName);
        exit(EXIT_FAILURE);
    }
    outputToFile(buffer, count, fp);
    
    free(buffer);
    free(line);
    printf("Sort phase done. %d tmp sorted files are produced.\n", fileNum);
}

The logic is not difficult. The function takes the input file descriptor as a parameter and reads each line(via the getline method) in a loop until reaches the end of the file. The numbers will be read into the memory buffer before hitting the memory limit. When the memory buffer is full(100000 bytes), the numbers are sorted with the function mergeSort.

The function mergeSort is defined inside the file internal_sort.c, which implements the classic internal merge sorting algorithm. I will not cover it in this article, since you can find many documents about it online. If you don’t know about it, please spend some time learning about it. Of course, you can replace it with other sorting algorithms like quick sort too. I leave this for the readers.

After sorting, the numbers are saved in the temporary files in the directory of ./tmp/pass0. The filename is just the run number.

1
2
3
4
5
6
7
8
9
10
11

/*
 * Output sorted record to given file(of)
 * */

void outputToFile(unsigned int *buffer, int size, FILE *of) {
    int i;
    for (i = 0; i < size; i++) {
        fprintf(of, "%u\n", buffer[i]);
    }
    fclose(of);
}

We can verify the result of the sort phase as follows:

You can see each file contains up to 25000 (equal to MEMORY_LIMIT/RECORD_SIZE) numbers and 312 files are created in pass0.

Note that I will not examine the details about how to make a new directory and how to open a new file to read or write. You can learn such Linux file I/O concepts by yourself.

Merge phase

The exMerge function controls the passes in the merge phase starting from pass1.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

void exMerge() {
    /* some code omitted ... */
    int pass = 1;
    while (fileNum > 1) {
        exMergeSort(pass, fileNum);
        int remainer = fileNum % ways;
        fileNum = fileNum / ways;
        if (remainer > 0) {
            fileNum++;
        } 
        pass++;
    }
    /* some code omitted ... */
}

The variable fileNum stores the run number in each pass. And the variable ways denotes the number of multi-way. Thus, the run number of the next pass should be calculated as fileNum / ways.

The detailed merging logic is inside the function exMergeSort, which takes two parameters. pass means the current pass number(starting from 1), while nums means how many runs(or sub-files) in the last pass need to be merged.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

void exMergeSort(int pass, int nums) {
    /* some code omitted ... */
    int inputFileNum = 0; 
    int run = 1;
    for (; inputFileNum < nums;) { 
        
        // create the dir for current pass
        if (inputFileNum == 0) {
            int status;
            char dirName[20];
            sprintf(dirName, "./tmp/pass%d", pass);
            if ((status = mkdir(dirName, S_IRWXU | S_IRWXU | S_IROTH | S_IXOTH)) == -1) {
                fprintf(stderr, "Failed to create tmp directory %s.\n", dirName);
                exit(EXIT_FAILURE);
            }
        }
        // open new file to merge in each run
        FILE *fm; 
        char mergedFileName[20];
        sprintf(mergedFileName, "./tmp/pass%d/%d.txt", pass, run);
        if ((fm = fopen(mergedFileName, "w+")) == NULL) {
            fprintf(stderr, "%s\n", strerror(errno));
            fprintf(stderr, "merged file %s: can't create or open.\n", mergedFileName);
        }
        run++;
        /* some code omitted ... */
    }

}

The above code creates a temp directory for each pass and a temp file for each run.

Next, we create an array of the input files for each run. And the input file is inside the temp directory of the last pass. Each run merges multiple files in the for loop. The only trick logic here is that the remaining files in the last run may be less than the number of ways declared, we need to handle that properly (line 5 of the below code block).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// Rewind the sorted files in previous pass, each run merge way_numbers numbers of files
// Merge the sorted files with multi ways in N runs. 
// In the first N - 1 runs, each run merge ways numbers of files
// In the last run, merge the remaining files. 
int way_numbers = run * ways <= nums ? ways : nums - inputFileNum;
FILE *fiarr[way_numbers];
for (int i = 0; i < way_numbers; i++) {
   char inputFileName[20];
   inputFileNum++; // start from 0 to nums
   sprintf(inputFileName, "./tmp/pass%d/%d.txt", pass - 1, inputFileNum);
   if ((fiarr[i] = fopen(inputFileName, "r")) == NULL) {
        fprintf(stderr, "%s\n", strerror(errno));
        fprintf(stderr, "input file %s: can't create or open.\n", inputFileName);
   }
   rewind(fiarr[i]);
}

Next, we need to read one number from every input file until only one file is not run out. Find the smallest one and save it in the temp output run file. And for the last remaining file, remember to put the rest numbers into the output file too.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

// get and compare records until files runs out of records
char *records[way_numbers]; 
for (int i = 0; i < way_numbers; i++) {
    records[i] = getRecord(fiarr[i]);
}
// loop until only one file is not run-out
while(validateRecords(records, way_numbers )) {
    int index = getMinRecordIndex(records, way_numbers);
    fprintf(fm, "%s", records[index]); // print record to new merged file
    free(records[index]); // free the memory allocated by getline in getRecord function
    records[index] = getRecord(fiarr[index]); // Get new record from the file
}
       
// put the rest record in the last remaining file into new file 
int lastIndex = getLastRemainRecordIndex(records, way_numbers);
while(records[lastIndex]) {
    fprintf(fm, "%s", records[lastIndex]);
    free(records[lastIndex]);
    records[lastIndex] = getRecord(fiarr[lastIndex]);
}

The above code bock utilizes several methods like getRecord, validateRecords, getMinRecordIndex and getLastRemainRecordIndex as follows, and these functions are easy to understand.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

/*
 * Returns a copy of the record
 *
 * */
char* getRecord(FILE *ifp) {
    char *line = NULL;
    size_t len = 0;
    ssize_t nread;
    while ((nread = getline(&line, &len, ifp)) != -1) {
        return line;       
    }
    return NULL;
}
/*
 * Validate whether at least two records are non-zero
 * */
bool validateRecords(char **records, int size) {
    int count = 0;
    for (int i = 0; i < size; i++) {
        if (records[i] != NULL) {
            count++;
        }
    }
    return count >= 2;
}

/*
 * Get the min valid record's index
 * */
int getMinRecordIndex(char **records, int size) {
    int index = 0;
    unsigned int min = (int)INFINITY;
    for (int i = 0; i < size; i++) {
        if (records[i] == NULL) { // pass invalid run-out record files
            continue;
        } 
        if (strtoul(records[i], NULL, 10) < min) {
            min = strtoul(records[i], NULL, 10);
            index = i;
        }
    }
    return index;
}
/*
 * Get the last remainer of the records
 * */
int getLastRemainRecordIndex(char **records, int size) {
    for (int i = 0; i < size; i++) {
        if (records[i] != NULL) {
            return i;
        }
    }
}

In detail, you can refer to the source code of this github repo. Next, let’s evaluate the performance of this algorithm by tuning the number of ways for merging.

Performance Evaluation

We’ll use the Linux time utility to measure the running time of this algorithm.

The result of two-way mergesort is:

while six-way mergesort can complete with a shorter runtime.

This article will examine one interesting question which I came across when reading the book Programming Pearls. This question simply goes like this: How to sort a large disk file? The disk file has so much data that it cannot all fit into the main memory. I considered this algorithm question for a while; but noticed that all the classic sorting algorithms, like Quick sort and merge sort, can’t solve it easily. Then I did some research about it, and I will share what I learned in this article. I believe you can solve this problem as well, after reading this article.

Background of External Algorithm

Traditionally, computer scientists analyze the running time of an algorithm by counting the number of executed instructions, which is usually expressed as a function of the input size n, like the well-known Big O notation. This kind of algorithm complexity analysis is based on the Random access machine(RAM) model, which defines the following assumptions:

• A machine with an unbounded amount of available memory;
• Any desired memory location can be accessed in unit time;
• Every instruction takes the same amount of time.

This model works well when the amount of memory that the algorithm needs to use is smaller than the amount of memory in the computer running the code.

But in the real world, some applications need to process data that are too large to fit into a computer’s main memory at once. And the algorithms used to solve such problems are called external memory algorithms or disk-based algorithms; since the input data are stored on an external memory storage device.

Instead of the RAM model, the external memory algorithm is analyzed based on the external memory model. The model consists of a CPU processor with an internal memory of bounded size, connected to an unbounded external memory. One I/O operation consists of moving a block of contiguous elements from external to internal memory(this is called page cache and is managed by the kernel.).

Compared with the main memory, the access speed of external memory is slower by several orders of magnitude, even though modern storage techniques such as SSD are already adopted. Thus, for external algorithms, the bottleneck of the performance is disk IO speed instead of the CPU cycles.

In this section, we examined the computational model of the external memory algorithms. Next, let’s review one typical type of external memory algorithm: external sorting.

External Mergesort

Similar to the traditional internal sorting algorithms, several different solutions are available for external sorting. In this article, I will focus on external mergesort.

External mergesort is a straightforward generalization of internal mergesort. The algorithm starts by repeatedly loading M input items into the memory(since the memory buffer size is limited, can only store M input items at once), sorting them, and writing them back out to disk. This step divides the input file into some(or many, if the input file is very large) sorted runs, each with M items sorted. Then the algorithm repeatedly merges multiple sorted runs, until only a single sorted run containing all the input data remains.

Let’s use the following example model to analyze this algorithm. First of all, assume the memory buffer is limited, and the size is one page. And the input file size is 8 pages. External mergesort can be divided into two phases: sort phase and merge phase.

Sort phase:

• Divide the entire input file into 8 groups, each group size is one page(memory buffer capacity).
• Load each page into the memory, sort the items in the memory buffer(with an internal sorting algorithm), and save the sorted items in a temporary sub-file. Each sub-file is called a run.

At the end of the sort phase, 8 temporary sorted 1-page runs will be created. This step can be marked as pass 0.

Merge phase:

The 8 sorted runs in pass 0 will be merged into a single sorted file with 3 more passes.

• pass 1: Perform 4 runs for the merge.
  • Run 1: Merge the first two small 1-page runs into a big 2-page run. This merging step goes as follows:
    • Read the first two sorted sub-files (one item from each file).
    • Find the smaller item, output it in the new sub-file, and the appropriate input sub-file is advanced. Repeat this cycle until the input sub-file is completed. This routine’s logic is the same as the internal mergesort algorithm.
  • Run 2: Merge the next two 1-page runs into a 2-page run.
  • Run 3 and 4: follow the same process.
  • At the end of pass 1, 4 temporary sorted 2-page runs will be created.
• pass 2: Perform 2 runs for the merge.
  • At the end of pass 2, 2 temporary sorted 4-page runs will be created.
• pass 3: Perform 1 run for the merge.
  • At the end of pass 3, the final 8-page run containing all the sorted items will be created.

Note: the above process may seem complicated at the first sight, but the logic is nearly the same as the internal merge sort. The only difference is internal merging is based on memory buffers, while external merging is based on disk files, and needs reading the item from disk to memory.

Since we keep merging two small sub-files into a big one with doubled size, the above algorithm can be called two-way external merge sorting. We can generalize the idea to multi-way external merge sorting, which merges M runs into one.

Next, let’s analyze its complexity. Suppose the input file has N items and each page consists of B items. And M denotes the number of ways used in the merge phase, thus the number of passes should be: log_M(N/B) + 1, where plus one means the first pass in the sort phase. And each pass, each item is read and written once from and to the disk file. So the total number of disk I/O is: 2N*(log_M(N/B) + 1)

Summary

In this article, we examined the abstract computational model of the external memory algorithm and analyzed the details of the external mergesort algorithm. Next article, let’s implement the code and evaluate the performance.

In the last article, we investigated how the allocated chunks are aligned and stored in the heap. This article continues to examine how to free a chunk of memory and how the freed chunks are stored in the heap.

Hands-on demo

Let’s continue debugging the demo code shown in the last article:

1
2
3
4
5
6
7
8
9
10
11
12
13

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <malloc.h>

int main(int argc, char *argv[]) {
    char *a = (char*)malloc(100);
    strcpy(a, "AAAABBBBCCCCDDDD");
    free(a);
    char *b = (char*)malloc(100);
    free(b);
    return 0;
}

Previously, we allocated a chunk of memory and put data in it. The next line will free this chunk. Before we run the instruction and show the demo result, let’s discuss the theory first.

The freed chunk will not be returned to the kernel immediately after the free is called. Instead, the heap allocator keeps track of the freed chunks in a linked list data structure. So the freed chunks in the linked list can be reused when the application requests new allocations again. This can decrease the performance overhead by avoiding too many system calls.

The allocator could store all the freed chunks together in a long linked list, this would work but the performance would be slow. Instead, the glibc maintains a series of freed linked lists called bins, which can speed up the allocations and frees. We will examine how bins work later.

It is worth noting that each free chunk needs to store pointers to other chunks to form the linked list. That’s what we discussed in the last section, there’re two points in the malloc_chunk structure: fd and bk, right? Since the user data region of the freed chunk is free for use by the allocator, so it repurposes the user data region as the place to store the pointer.

Based on the above description, the following picture illustrates the exact structure of a freed chunk:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

chunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             Size of previous chunk, if freed                  | 
            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             Size of chunk, in bytes                     |A|M|P|
      mem-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             pointer to the next freed chunk                   |
            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             pointer to the previous freed chunk               |
            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            .                                                               .
            .                           ......                              .
nextchunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             Size of chunk                                     |
            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Now step over one line in gdb and check chunks in the heap as follows:

You can see the changes: the allocated chunk is marked as a Free chunk (tcache) and pointer fd is set(which indicates this freed chunk is inserted into a linked list).

The tcache is one kind of bins provided by glibc. The gdb pwndbg plugin allows you to check the content of bins by running command bins as follows:

Note that the freed chunk(at 0x5555555592a0) is inserted into tcache bins as the liked list header.

Note that there 5 types of bins: small bins, large bins, unsorted bins, fast bins and tcache bins. If you don’t know, don’t worry I will examine them in the following section.

According to the definition, after the second malloc(100) is called, the allocator should reuse the freed chunk in the bins. The following image can prove this:

The freed chunk at 0x555555559290 is in use again and all bins are empty after the chunk is removed from the linked list. All right!

Recycling memory with bins

Next, I want to spend a little bit of time examining why we need bins and how bins optimize chunk allocation and free.

If the allocator keeps track of all the freed chunks in a long linked list. The time complexity is O(N) for the allocator to find a freed chunk with fit size by traversing from the head to the tail. If the allocator wants to keep the chunks in order, then at least O(NlogN) time is needed to sort the list by size. This slow process would have a bad impact on the overall performance of programs. That’s the reason why we need bins to optimize this process. In summary, the optimization is done on the following two aspects:

• High-performance data structure
• Per-thread cache without lock contention

High-performance data structure

Take the small bins and large bins as a reference, they are defined as follows:

1
2
3
4
5

#define NBINS             128

typedef struct malloc_chunk* mchunkptr;

mchunkptr bins[NBINS * 2 - 2];

They are defined together in an array of linked lists and each linked list(or bin) stores chunks that are all the same fixed size. From bins[2] to bins[63] are the small bins, which track freed chunks less than 1024 bytes while the large bins are for bigger chunks. small bins and large bins can be represented as a double-linked list shown below:

The glibc provides a function to calculate the index of the corresponding small(or large) bin in the array based on the requested size. Since the index operation of the array is in O(1) time. Moreover, each bin contains chunks of the same size, so it can also take O(1) time to insert or remove one chunk into or from the list. As a result, the entire allocation time is optimized to O(1).

bins are LIFO(Last In First Out) data structure. The insert and remove operations can be illustrated as follows:

Moreover, for small bins and large bins, if the neighbors of the current chunk are free, they are merged into a larger one. That’s the reason we need a double-linked list to allow running fast traverse both forward and backward.

Unlike small bins and large bins, fast bins and tcache bins chunks are never merged with their neighbors. In practice, the glibc allocator doesn’t set the P special flag at the start of the next chunk. This can avoid the overhead of merging chunks so that the freed chunk can be immediately reused if the same size chunk is requested. Moreover, since fast bins and tcache bins are never merged, they are implemented as a single-linked list.

This can be proved by running the second free method in the demo code and checking the chunks in the heap as follows:

First, the top chunk’s size is still 0x20d01 rather than 0x20d00, which indicates the P bit is equal to 1. Second, the Free chunk only has one pointer: fd. If it’s in a double-linked list, both fd and bk should point to a valid address.

Per-thread cache without lock contention

The letter t in tcache bins represents the thread, which is used to optimize the performance of multi-thread programs. In multi-thread programming, the most common way solution to prevent the race condition issue is using the lock or mutex. Similarly, The glibc maintains a lock in the data structure for each heap. But this design comes with a performance cost: lock contention, which happens when one thread attempts to acquire a lock held by another thread. This means the thread can’t do any tasks.

tcache bins are per-thread bins. This means if the thread has a chunk on its tcache bins, it can serve the allocation without waiting for the heap lock!

Summary

In this article, we examined how the userland heap allocaor works by debugging into the heap memory with gdb. The discussion is fully based on the glibc implementation. The design and behavior of the glibc heap allocator are complex but interesting, what we covered here just touches the tip of the iceberg. You can explore more by yourself.

Moreover, I plan to write a simple version of a heap allocator for learning and teaching purpose. Please keep watching my blog!

The previous article gave a general overview of memory management. The story goes on. In this section, let’s break into the heap memory to see how it works basically.

Memory allocator

We need to first understand some terminology in the memory management field:

• mutator: the program that modifies the objects in the heap, which is simply the user application. But I will use the term mutator in this article.
• allocator: the mutator doesn’t allocate memory by itself, it delegates this generic job to the allocator. At the code level, the allocator is generally implemented as a library. The detailed allocation behavior is fully determined by the implementations, in this article I will focus on the memory allocator in the library of glibc.

The relationship between the mutator and allocator is shown in the following diagram:

There is a third component in the memory management field: the garbage collector(GC). GC reclaims memories automatically. Since this article is talking about manual heap memory allocation in system programming, we will ignore GC for now. GC is a very interesting technical challenge, I will examine it in the future. Please keep watching my blog!

Hands-on demo

We will use gdb and pwndbg(which is a gdb plugin) and break into the heap memory to see how it works. The gdb provides the functionality to extend it via Python plugins. pwndbg is the most widely used.

The demo code is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <malloc.h>

int main(int argc, char *argv[]) {
    char *a = (char*)malloc(100);
    strcpy(a, "AAAABBBBCCCCDDDD");
    free(a);
    char *b = (char*)malloc(100);
    free(b);
    return 0;
}

The demo code above just allocates some memory, set the content of the memory and releases it later. And then allocate other chunks of memory again. Very simple, all right?

First, set a breakpoint at line 7(the first malloc call) and run the program in gdb. Then run vmmap command from pwndbg, which can get the process memory layout as follows:

Note that there is no heap segment yet before the first malloc call is made. After step over one line in gdb, check the layout again:

Now the heap segment is created with the size of 132KB(21000 in hexadecimal). As described above, the kernel maps 132KB of physical memory to this process’s virtual memory and marks this 132KB block of physical memory as used to isolate other processes. This mapping routine is done via system calls like brk, sbrk and mmap. Please investigate these system calls yourself.

132KB is much bigger than the 100B(the size passed to malloc). This behavior can answer one question at the beginning of this article. The system calls aren’t necessary to be triggered each time when malloc is called. This design is aimed to decrease performance overhead. Now the 132KB heap memory is maintained by the allocator. Next time the application calls malloc again, the allocator will allocate memory for it.

Next, step one more line in gdb to assign value(“AAAABBBBCCCCDDDD”) to the allocated block. Let’s check the content of this 132KB heap segment with heap command as follows:

There are 3 chunks. Let’s examine these chunks one by one.

The top chunk contains all the remaining memories which have not been allocated yet. In our case, the kernel maps 132KB of physical memory to this process. And 100B memory is allocated by calling malloc(100), so the remaining memories are in the top chunk. The top chunk stays on the border of the heap segment, and it can grow and shrink as the process allocates more memory or release unused memory.

Then let’s look at the chunk with the size of 0x291. The allocator uses this chunk to store heap management structures. It is not important for our analysis, just skip it.

What we care about is the chunk in the middle with a size of 0x71. It should be the block we requested and contains the string “AAAABBBBCCCCDDDD”. We can verify this point by checking its content:

gdb’s x command can display the memory contents at a given address using the specified format. x/40wx 0x555555559290 prints 40 words(each word is 32 bits) of memories starting from 0x555555559290 in the hexadecimal format.

We can see that the string “AAAABBBBCCCCDDDD” is there. So our guess is correct. But the question is why the size of this chunk is 0x71. To understand this, we need to first analyze how the allocator stores chunk. A chunk of memory is represented by the following structure:

1
2
3
4
5
6
7
8

struct malloc_chunk {
  INTERNAL_SIZE_T      prev_size;  /* Size of previous chunk (only if free).  */
  INTERNAL_SIZE_T      size;       /* Size in bytes, including overhead. */
  struct malloc_chunk* fd;                /* double links -- used only if free. */
  struct malloc_chunk* bk;                /* double links -- used only if free. */
};

typedef struct malloc_chunk* mchunkptr;

• prev_size: the size of the previous chunk only when the previous chunk is free, otherwise when the previous chunk is in use it stores the user data of the previous chunk.
• size: the size of the current chunk.
• fd: pointer to the next free chunk only when the current chunk is free, otherwise when the current chunk is in use it stores the user data.
• bk: pointer to the previous free chunk. Behaves in the same way as pointer fd.

Based on the above description, the following picture illustrates the exact structure of an allocated chunk:

1
2
3
4
5
6
7
8
9
10
11
12

    chunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             Size of previous chunk, if freed                  | 
            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             Size of chunk, in bytes                     |A|M|P|
      mem-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             User data starts here...                          .
            .                                                               .
            .                                                               .
            .                                                               |
nextchunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            |             Size of chunk                                     |
            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

• chunk: indicates the real starting address of the object in the heap memory.
• mem: indicates the returned address by malloc.

The memory in between is reserved for the metadata mentioned above: prev_size and size. On a 64-bit system, they’re (type of INTERNAL_SIZE_T) 8 bytes in length.

For the size field, it is worth noting:

• It includes both the size of metadata and the size of the actual user data.
• It is usually aligned to a multiple of 16 bytes. You can investigate the purpose of memory alignment by yourself.
• It contains three special flags(A|M|P) at the three least significant bits. We can ignore the other two bits for now, but the last bit indicates whether the previous chunk is in use(set to 1) or not(set to 0).

According to this, let’s review the content of this chunk again:

I add marks on the image to help you understand. Let’s do some simple calculations. 100 + 8 = 108, 100 is the size of memory we requested, 8 is the size of metadata(for size field). Then 108 is aligned to 112 as a multiple of 16 bytes. Finally, since the special flag P is set to 1, then we get 112 + 1 = 113(0x71)(that’s the reason why the size is 0x71 instead of 0x70).

In this section, we break into the heap segment and see how an allocated chunk works. Next, we’ll check how to free a chunk.

In my eyes, compared with developing applications with high-level programming languages, one of the biggest differences for system programming with low-level languages like C and C++, is you have to manage the memory by yourself. So you call APIs like malloc, and free to allocate the memory based on your need and release the memory when the resource is no longer needed. It is not only one of the most frequent causes of bugs in system programming; but also can lead to many security issues.

It’s not difficult to understand the correct usage of APIs like malloc, and free. But have you ever wondered how they work, for example:

• When you call malloc, does it trigger system calls and delegate the task to the kernel or there are some other mechanisms?
• When you call malloc(10) and try to allocate 10 bytes of heap memory, how many bytes of memory do you get? 10 bytes or more?
• When the memory is allocated, where exactly the heap objects are located?
• When you call free, is the memory directly returned to the kernel?

This article will try to answer these questions.

Note that memory is a super complex topic, so I can’t cover everything about it in one article (In fact, what is covered in this article is very limited). This article will focus on userland memory(heap) allocation.

Process memory management overview

Process virtual memory

Every time we start a program, a memory area for that program is reserved, and that’s process virtual memory as shown in the following image:

You can note that each process has one invisible memory segment containing kernel codes and data structures. This invisible memory segment is important; since it’s directly related to virtual memory, which is employed by the kernel for memory management. Before we dive into the other different segments, let’s understand virtual memory first.

Virtual memory technique

Why do we need virtual memory? Virtual memory is a service provided by the kernel in the form of abstraction. Without virtual memory, applications would need to manage their physical memory space, coordinating with every other process running on the computer. Virtual memory leaves that management to the kernel by creating the maps that allow translation between virtual and physical memory. The kernel creates an illusion that each process occupies the entire physical memory space. We can also realize process isolation based on virtual memory to enhance security.

Virtual memory is out of this article’s scope, if you’re interested, please take a look at the core techniques: paging and swapping.

Static vs Dynamic memory allocation

Next, let’s take a close look at the process memory layout above and understand where they are from. Generally speaking, there are two ways via which memories can be allocated for storing data: static and dynamic. Static memory allocation happens at compile time, while dynamic memory allocation occurs at runtime.

When a program started, the executable file(on the Linux system, it’s called an ELF file) will be loaded into the memory as a Process Image. This ELF file contains the following segments:

• .TEXT: contains the executable part of the program with all the machine codes.
• .DATA: contains initialized static and global variables.
• .BSS: is short for block started by symbol contains uninitialized static and global variables.

The ELF file will be loaded by the kernel and create a process image. And these static data will be mapped into the corresponding segments of the virtual memory. The ELF loader is also an interesting topic, I will write another article about it in the future. Please keep watching my blog!

The memory-mapped region segment is used for storing the shared libraries.

Finally, stack and heap segments are produced at runtime dynamically, which are used to store and operate on temporary variables that are used during the execution of the program. Previously, I once wrote an article about the stack, please refer to it if you want to know the details.

The only remaining segment we didn’t mention yet is the heap, which is this article’s focus!

You can check the memory layout of one process by examining this file /proc/{pid}/maps as below:

Note that the above investigation doesn’t consider multiple threads. The memory layout of the process with multi-threads will be more complex, please refer to other online documents.

In this section, we had a rough overview of memory management from top to bottom. Hope you can see the big picture and know where we are. Next, let’s dig into the heap segment and see how it works.

In this post, I want to introduce my new project: cPacketSniffer. I worked on it for the past two months. Finally, I worked it out and feel very proud of putting it here!

Motivation

Simply speaking, I want to sharpen my techniques in network programming and Linux system programming. Both of these two topics can lead you to the bottom of computers or software. Feynman said “There is plenty of room at the bottom”, I think this physics law can apply to software as well.

Acknowledgement

It’s very lucky for me to come across this site “Network programming in Linux”, which developed a network packet capturing tool with C++. After confirming that the documents and source code on this site is completed and clear, I decided to refactor it with C language. That’s the starting point for my project cPacketSniffer.

Features

As a network packets sniffer, cPacketSniffer provides the following features:

• Integrate with libpcap to support: filtering captured packets, capturing packets offline, capturing packets on specific devices and capturing packets in promiscuous mode.
• Analyze network packets at low layers of TCP/IP stack, including Ethernet, ARP, ICMP, IP(IPv4), TCP, UDP, etc. Also one protocol in the application layer: TFTP.
• Detect network security attacks:
  • ARP spoofing detection.
  • Ping flood detection.
• Analyze and track network traffics:
  • TCP session tracking and traffic analysis.
  • TFTP session tracking and traffic analysis.

The following images demonstrate some typical usages of cPacketSniffer:

Packet Analysis:

ARP Spoofing Detection:

PING Flood Detection:

TCP Session Tracking:

Besides the above network programming-related functionalities, it also covers the following points:

• Develop a generic data structure in C.
• Error handling in C.
• Data encapsulation (object-oriented style programming) in C.
• Manual memory management in C.
• etc.

This article will not cover these points in detail, I will write articles on these topics separately in the future. Please keep watching my blog!

Future work

Now cPacketSniffer can work as a network packet sniffer based on the design. Moreover, it can also serve as a testbed to try experimental features. Next step I plan to try the following ideas:

• Implement the network intrusion detection function.
• Improve the performance with advanced data structures, like binary search trees.
• Memory and cache performance tuning.
• Automatic memory management by Garbage Collection.
• Integrate ncurses for Text-based user interface.

In the previous article, we examined how to write a Kernel module and load it dynamically into a running Linux system. Based on this understanding, let’s continue our journey to write a Netfilter module as our mini-firewall.

Netfilter architecture.

Basics of Netfilter hooks

The Netfilter framework provides a bunch of hooks in the Linux kernel. As network packets pass through the protocol stack in the kernel, they will traverse these hooks as well. And Netfilter allows you to write modules and register callback functions with these hooks. When the hooks are triggered, the callback functions will be called. This is the basic idea behind Netfilter architecture. Not difficult to understand, right?

Currently, Netfilter provides the following 5 hooks for IPv4:

• NF_INET_PRE_ROUTING: is triggered right after the packet has been received on a network card. This hook is triggered before the routing decision was made. Then the kernel determines whether this packet is destined for the current host or not. Based on the condition, the following two hooks will be triggered.
• NF_INET_LOCAL_IN: is triggered for network packets that are destined for the current host.
• NF_INET_FORWARD: is triggered for network packets that should be forwarded.
• NF_INET_POST_ROUTING: is triggered for network packets that have been routed and before being sent out to the network card.
• NF_INET_LOCAL_OUT: is triggered for network packets generated by the processes on the current host.

The hook function you defined in the module can mangle or filter the packets, but it eventually must return a status code to Netfilter. There are several possible values for the code, but for now, you only need to understand two of them:

• NF_ACCEPT: this means the hook function accepts the packet and it can go on the network stack trip.
• NF_DROP: this means the packet is dropped and no further parts of the network stack will be traversed.

Netfilter allows you to register multiple callback functions to the same hook with different priorities. If the first hook function accepts the packet, then the packet will be passed to the next functions with low priority. If the packet is dropped by one callback function, then the next functions(if existing) will not be traversed.

As you see, Netfilter has a big scope and I can’t cover every detail in the articles. So the mini-firewall developed here will work on the hook NF_INET_PRE_ROUTING, which means it works by controlling the inbound network traffic. But the way of registering the hook and handling the packet can be applied to all other hooks.

Note: there is another remarkable question: what’s the difference between Netfilter and eBPF? If you don’t know eBPF, please refer to my previous article. Both of them are important network features in the Linux kernel. The important thing is Netfilter and eBPF hooks are located in different layers of the Kernel. As I drew in the above diagram, eBPF is located in a lower layer.

Kernel code of Netfilter hooks

To have a clear understanding of how the Netfilter framework is implemented inside the protocol stack, let’s dig a little bit deeper and take a look at the kernel source code (Don’t worry, only shows several simple functions). Let’s use the hook NF_INET_PRE_ROUTING as an example; since the mini-firewall will be written based on it.

When an IPv4 packet is received, its handler function ip_rcv will be called as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

//In source code file /kernel-src/net/ipv4/ip_input.c
/*
 * IP receive entry point
 */
int ip_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt,
           struct net_device *orig_dev)
{
        struct net *net = dev_net(dev);

        skb = ip_rcv_core(skb, net);
        if (skb == NULL)
                return NET_RX_DROP;
        // run Netfilter NF_INET_PRE_ROUTING hook's callback function
        return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, 
                       net, NULL, skb, dev, NULL,
                       ip_rcv_finish);
}

In this handler function, you can see the hook is passed to the function NF_HOOK. Based on the name NF_HOOK, you can guess that it is for triggering the Netfilter hooks. Right? Let’s continue to examine how NF_HOOK is implemented as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

//In source code file /kernel-src/include/linux/netfilter.h
static inline int
NF_HOOK(uint8_t pf, unsigned int hook, struct net *net, struct sock *sk, struct sk_buff *skb,
        struct net_device *in, struct net_device *out,
        int (*okfn)(struct net *, struct sock *, struct sk_buff *))
{
        int ret = nf_hook(pf, hook, net, sk, skb, in, out, okfn);
        if (ret == 1)
                ret = okfn(net, sk, skb); // in our case: okfn is ip_rcv_finish
        return ret;
}
/**
 *      nf_hook - call a netfilter hook
 *
 *      Returns 1 if the hook has allowed the packet to pass.  The function
 *      okfn must be invoked by the caller in this case.  Any other return
 *      value indicates the packet has been consumed by the hook.
 */
static inline int nf_hook(u_int8_t pf, unsigned int hook, struct net *net,
                          struct sock *sk, struct sk_buff *skb,
                          struct net_device *indev, struct net_device *outdev,
                          int (*okfn)(struct net *, struct sock *, struct sk_buff *))
{
    // code omit
}

The function NF_HOOK contains two steps:

• First, runs the hook’s callback functions by calling the underlying function nf_hook.
• Second, invokes the function okfn (passed to NF_HOOK as the argument), if the packet passes through the hook functions and doesn’t drop.

For the hook NF_INET_LOCAL_IN, the function ip_rcv_finish will be invoked after the hook functions pass. Its job is to pass the packet on to the next protocol handler(TCP or UDP) in the protocol stack to continue its journey!

The other 4 hooks all use the same function NF_HOOK to trigger the callback functions. The following table shows where the hooks are embedded in the kernel, I leave them to the readers.

Next, Let’s review the Netfilter’s APIs to create and register the hook function.

Netfilter API

It’s straightforward to create a Netfilter module, which involves three steps:

• Define the hook function.
• Register the hook function in the kernel module initialization process.
• Unregister the hook function in the kernel module clean-up process.

Let’s go through them quickly one by one.

Define a hook function

The hook function name can be whatever you want, but it must follow the signature below:

1
2
3
4

//In source code file /kernel-src/include/linux/netfilter.h
typedef unsigned int nf_hookfn(void *priv,
                               struct sk_buff *skb,
                               const struct nf_hook_state *state);

The hook function can mangle or filter the packet whose data is stored in the sk_buff structure (we can ignore the other two parameters; since we don’t use them in our mini-firewall). As we mentioned above, the callback function must return a Netfilter status code which is an integer. For instance, the accepted and dropped status is defined as follows:

1
2
3
4

// In source code file /kernel-src/include/uapi/linux/netfilter.h
/* Responses from hook functions. */
#define NF_DROP 0
#define NF_ACCEPT 1

Register and unregister a hook function

To register a hook function, we should wrap the defined hook function with related information, such as which hook you want to bind to, the protocol family and the priority of the hook function, into a structure struct nf_hook_ops and pass it to the function nf_register_net_hook.

1
2
3
4
5
6
7
8
9
10
11

//In source code file /kernel-src/include/linux/netfilter.h
struct nf_hook_ops {
        /* User fills in from here down. */
        nf_hookfn               *hook;    // callback function
        struct net_device       *dev;     // network device interface
        void                    *priv; 
        u_int8_t                pf;       // protocol
        unsigned int            hooknum;  // Netfilter hook enum
        /* Hooks are ordered in ascending priority. */
        int                     priority; // priority of callback function
};

Most of the fields are very straightforward to understand. The one need to emphasize is the field hooknum, which is just the Netfilter hooks discussed above. They are defined as enumerators as follows:

1
2
3
4
5
6
7
8
9
10

// In source code file /kernel-src/include/uapi/linux/netfilter.h
enum nf_inet_hooks {
NF_INET_PRE_ROUTING,
NF_INET_LOCAL_IN,
NF_INET_FORWARD,
NF_INET_LOCAL_OUT,
NF_INET_POST_ROUTING,
NF_INET_NUMHOOKS,
NF_INET_INGRESS = NF_INET_NUMHOOKS,
};

Next, let’s take a look at the functions to register and unregister hook functions goes as follows:

1
2
3
4

//In source code file /kernel-src/include/linux/netfilter.h
/* Function to register/unregister hook points. */
int nf_register_net_hook(struct net *net, const struct nf_hook_ops *ops);
void nf_unregister_net_hook(struct net *net, const struct nf_hook_ops *ops);

The first parameter struct net is related to the network namespace, we can ignore it for now and use a default value.

Next, let’s implement our mini-firewall based on these APIs. All right?

Implement mini-firewall

First, we need to clarify the requirements for our mini-firewall. We’ll implement two network traffic control rules in the mini-firewall as follows:

• Network protocol rule: drops the ICMP protocol packets.
• IP address rule: drops the packets from one specific IP address.

The completed code implementation is in this Github repo.

Drop ICMP protocol packets

ICMP is a network protocol widely used in the real world. The popular diagnostic tools like ping and traceroute run the ICMP protocol. We can filter out the ICMP packets based on the protocol type in the IP headers with the following hook function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// In mini-firewall.c 
static unsigned int nf_blockicmppkt_handler(void *priv, struct sk_buff *skb, const struct nf_hook_state *state)
{
struct iphdr *iph;   // IP header
struct udphdr *udph; // UDP header
if(!skb)
return NF_ACCEPT;
iph = ip_hdr(skb); // retrieve the IP headers from the packet
if(iph->protocol == IPPROTO_UDP) { 
udph = udp_hdr(skb);
if(ntohs(udph->dest) == 53) {
return NF_ACCEPT; // accept UDP packet
}
}
else if (iph->protocol == IPPROTO_TCP) {
return NF_ACCEPT; // accept TCP packet
}
else if (iph->protocol == IPPROTO_ICMP) {
printk(KERN_INFO "Drop ICMP packet \n");
return NF_DROP;   // drop TCP packet
}
return NF_ACCEPT;
}

The logic in the above hook function is easy to understand. First, we retrieve the IP headers from the network packet. And then according to the protocol type field in the headers, we decided to accept TCP and UDP packets but drop the ICMP packets. The only technique we need to pay attention to is the function ip_hdr, which is the kernel function defined as follows:

1
2
3
4
5
6
7
8
9
10

//In source code file /kernel-src/include/linux/ip.h
static inline struct iphdr *ip_hdr(const struct sk_buff *skb)
{
        return (struct iphdr *)skb_network_header(skb);
}
// In source code file /kernel-src/include/linux/skbuff.h
static inline unsigned char *skb_network_header(const struct sk_buff *skb)
{
        return skb->head + skb->network_header;
}

The function ip_hdr delegates the task to the function skb_network_header. It gets IP headers based on the following two data:

• head: is the pointer to the packet;
• network_header: is the offset between the pointer to the packet and the pointer to the network layer protocol header. In detail, you can refer to this document.

Next, we can register the above hook function as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

// In mini-firewall.c 
static struct nf_hook_ops *nf_blockicmppkt_ops = NULL;

static int __init nf_minifirewall_init(void) {
nf_blockicmppkt_ops = (struct nf_hook_ops*)kcalloc(1,  sizeof(struct nf_hook_ops), GFP_KERNEL);
if (nf_blockicmppkt_ops != NULL) {
nf_blockicmppkt_ops->hook = (nf_hookfn*)nf_blockicmppkt_handler;
nf_blockicmppkt_ops->hooknum = NF_INET_PRE_ROUTING;
nf_blockicmppkt_ops->pf = NFPROTO_IPV4;
nf_blockicmppkt_ops->priority = NF_IP_PRI_FIRST; // set the priority

nf_register_net_hook(&init_net, nf_blockicmppkt_ops);
}
return 0;
}

static void __exit nf_minifirewall_exit(void) {
if(nf_blockicmppkt_ops != NULL) {
nf_unregister_net_hook(&init_net, nf_blockicmppkt_ops);
kfree(nf_blockicmppkt_ops);
}
printk(KERN_INFO "Exit");
}

module_init(nf_minifirewall_init);
module_exit(nf_minifirewall_exit);

The above logic is self-explaining. I will not spend too much time here.

Next, it’s time to demo how our mini-firewall works.

Demo time

Before we load the mini-firewall module, the ping command can work as expected:

1
2
3
4
5
6
7
8
9
10
11

chrisbao@CN0005DOU18129:~$ lsmod | grep mini_firewall
chrisbao@CN0005DOU18129:~$ ping www.google.com
PING www.google.com (142.250.4.103) 56(84) bytes of data.
64 bytes from sm-in-f103.1e100.net (142.250.4.103): icmp_seq=1 ttl=104 time=71.9 ms
64 bytes from sm-in-f103.1e100.net (142.250.4.103): icmp_seq=2 ttl=104 time=71.8 ms
64 bytes from sm-in-f103.1e100.net (142.250.4.103): icmp_seq=3 ttl=104 time=71.9 ms
64 bytes from sm-in-f103.1e100.net (142.250.4.103): icmp_seq=4 ttl=104 time=71.8 ms
^C
--- www.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 71.857/71.902/71.961/0.193 ms

In contrast, after the mini-firewall module is built and loaded (based on the commands we discussed previously):

1
2
3
4
5
6
7

chrisbao@CN0005DOU18129:~$ lsmod | grep mini_firewall
mini_firewall          16384  0
chrisbao@CN0005DOU18129:~$ ping www.google.com
PING www.google.com (142.250.4.105) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5097ms

You can see all the packets are lost; because it is dropped by our mini-firewall. We can verify this by running the command dmesg:

1
2
3
4
5
6

chrisbao@CN0005DOU18129:~$ dmesg | tail -n 5
[ 1260.184712] Drop ICMP packet
[ 1261.208637] Drop ICMP packet
[ 1262.232669] Drop ICMP packet
[ 1263.256757] Drop ICMP packet
[ 1264.280733] Drop ICMP packet

But other protocol packets can still run through the firewall. For instance, the command wget 142.250.4.103 can return normally as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

chrisbao@CN0005DOU18129:~$ wget 142.250.4.103
--2022-06-25 10:12:39--  http://142.250.4.103/
Connecting to 142.250.4.103:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://142.250.4.103:6080/php/urlblock.php?args=AAAAfQAAABAjFEC0HSM7xhfO~a53FMMaAAAAEILI_eaKvZQ2xBfgKEgDtwsAAABNAAAATRPNhqoqFgHJ0ggbKLKcdinR4UvnlhgAR4~YyrY4tAnroOFkE_IsHsOg9~RFPc7nEoj6YdiDgqZImAmb_xw9ZuFLvF91P2HzP5tlu1WX&url=http://142.250.4.103%2f [following]
--2022-06-25 10:12:39--  http://142.250.4.103:6080/php/urlblock.php?args=AAAAfQAAABAjFEC0HSM7xhfO~a53FMMaAAAAEILI_eaKvZQ2xBfgKEgDtwsAAABNAAAATRPNhqoqFgHJ0ggbKLKcdinR4UvnlhgAR4~YyrY4tAnroOFkE_IsHsOg9~RFPc7nEoj6YdiDgqZImAmb_xw9ZuFLvF91P2HzP5tlu1WX&url=http://142.250.4.103%2f
Connecting to 142.250.4.103:6080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3248 (3.2K) [text/html]
Saving to: ‘index.html’

index.html                                           100%[===================================================================================================================>]   3.17K  --.-KB/s    in 0s

2022-06-25 10:12:39 (332 MB/s) - ‘index.html’ saved [3248/3248]

Next, let’s try to ban the traffic from this IP address.

Drop packets source from one specific IP address

As we mentioned above, multiple callback functions are allowed to be registered on the same Netfilter hook. So we will define the second hook function with a different priority. The logic of this hook function goes like this: we can get the source IP address from the IP headers and make the drop or accept decision according to it. The code goes as follows

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// In mini-firewall.c 
#define IPADDRESS(addr) \
((unsigned char *)&addr)[3], \
((unsigned char *)&addr)[2], \
((unsigned char *)&addr)[1], \
((unsigned char *)&addr)[0]

static char *ip_addr_rule = "142.250.4.103";

static unsigned int nf_blockipaddr_handler(void *priv, struct sk_buff *skb, const struct nf_hook_state *state)
{
if (!skb) {
return NF_ACCEPT;
} else {
char *str = (char *)kmalloc(16, GFP_KERNEL);
u32 sip;
struct sk_buff *sb = NULL;
struct iphdr *iph;

sb = skb;
iph = ip_hdr(sb);
sip = ntohl(iph->saddr); // get source ip address; 

sprintf(str, "%u.%u.%u.%u", IPADDRESS(sip)); // convert to standard IP address format
if(!strcmp(str, ip_addr_rule)) {
return NF_DROP;
} else {
return NF_ACCEPT;
}
}
}

This hook function uses two interesting techniques:

• ntohl: is a kernel function, which is used to convert the value from network byte order to host byte order. Byte order is related to the computer science concept of Endianness. Endianness defines the order or sequence of bytes of a word of digital data in computer memory. A big-endian system stores the most significant byte of a word at the smallest memory address. A little-endian system, in contrast, stores the least-significant byte at the smallest address. Network protocol uses the big-endian system. But different OS and platforms run various Endianness system. So it may need such conversion based on the host machine.

• IPADDRESS: is a macro, which generates the standard IP address format(four 8-bit fields separated by periods) from a 32-bit integer. It uses the technique of the equivalence of arrays and pointers in C. I will write another article to examine what it is and how it works. Please keep watching my updates!

Next, we can register this hook function in the same way discussed above. The only remarkable point is this callback function should have a different priority as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13

static int __init nf_minifirewall_init(void) {
<-omit code->
nf_blockipaddr_ops = (struct nf_hook_ops*)kcalloc(1, sizeof(struct nf_hook_ops), GFP_KERNEL);
if (nf_blockipaddr_ops != NULL) {
nf_blockipaddr_ops->hook = (nf_hookfn*)nf_blockipaddr_handler;
nf_blockipaddr_ops->hooknum = NF_INET_PRE_ROUTING;  // register to the same hook
nf_blockipaddr_ops->pf = NFPROTO_IPV4;
nf_blockipaddr_ops->priority = NF_IP_PRI_FIRST + 1; // set a higher priority

nf_register_net_hook(&init_net, nf_blockipaddr_ops);
}
<-omit code->
}

Let’s see how it works with a demo.

Demo time

After re-build and re-load the module, we can get:

1
2
3
4

chrisbao@CN0005DOU18129:~$ wget 142.250.4.103
--2022-06-25 10:20:07--  http://142.250.4.103/
Connecting to 142.250.4.103:80... failed: Connection timed out.
Retrying.

The wget 142.250.4.103 can’t return response. Because it is dropped by our mini-firewall. Great!

1
2
3
4
5
6

chrisbao@CN0005DOU18129:~$ dmesg | tail -n 5
[ 3162.064284] Drop packet from 142.250.4.103
[ 3166.089466] Drop packet from 142.250.4.103
[ 3166.288603] Drop packet from 142.250.4.103
[ 3174.345463] Drop packet from 142.250.4.103
[ 3174.480123] Drop packet from 142.250.4.103

More space to expand

You can find the full code implementation here. But I have to say, our mini-firewall only touches the surface of what Netfilter can provide. You can keep expanding the functionalities. For example, currently, the rules are hardcoded, why not make it possible to config the rules dynamically. There are many cool ideas worth trying. I leave it for the readers.

Summary

In this article, we implement the mini-firewall step by step and examined many detailed techniques. Not only code; but we also verify the behavior of the mini-firewall by running real demos.

In the last article, we examined the basics of Netfilter and Linux kernel modules in theory. Starting from this article, we will make our hands dirty and start implementing our mini-firewall. We will walk through the whole process step by step. In this article, let’s write our first Linux kernel module using a simple hello world demo. Then let’s learn how to build the module(which is very different from compiling an application in the user space) and how to load it in the kernel. After understanding how to write a module, in the next article, let’s write the initial version of our mini-firewall module using Netfilter’s hook architecture. All right. Let’s start the journey.

Make the first Kernel module

First, I have to admit that Linux Kernel module development is a kind of large and complex technology topic. And there are many great online resources about it. This series of articles is focusing on developing the mini-firewall based on Netfilter, so we can’t cover all the aspects of the Kernel module itself. In future articles, I’ll examine more in-depth knowledge of kernel modules.

Write the module

You can write the hello world Kernel module with a single C source code file hello.c as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#include <linux/init.h> /* Needed for the macros */
#include <linux/kernel.h> 
#include <linux/module.h> /* Needed by all modules */

static int __init hello_init(void)
{
    printk(KERN_INFO "Hello, world\n");
    return 0;
}

static void __exit hello_exit(void)
{
    printk(KERN_INFO "Goodbye, world\n");
}

module_init(hello_init);
module_exit(hello_exit);

MODULE_LICENSE("GPL");

We can write a Kernel module in such an easy and simple way because the Linux Kernel does the magic for you. Remember the design philosophy of Linux(Unix): Design for simplicity; add complexity only where you must.

Let’s examine several technical points worth to remark as follows:

First, Kernel modules must have at least two functions: a “start” function which is called when the module is loaded into the kernel, and an “end” function which is called just before it is removed from the kernel. Before kernel 2.3.13, the names of these two functions are hardcoded as init_module() and cleanup_module(). But in the new versions, you can use whatever name you like for the start and end functions of a module by using the module_init and module_exit macros. The macros are defined in include/linux/module.h and include/linux/init.h. You can refer there for detailed information.

Typically, module_init either registers a handler for something with the kernel (for example, the mini-firewall developed in this article), or it replaces one of the kernel functions with its own code (usually code to do something and then call the original function). The module_exit function is supposed to undo whatever module_init did, so the module can be unloaded safely.

Second, printk function provides similar behaviors to printf, which accepts the format string as the first argument. The printk function prototype goes as follows:

1

int printk(const char *fmt, ...);

printk function allows a caller to specify log level to indicate the type and importance of the message being sent to the kernel message log. For example, in the above code, the log level KERN_INFO is specified by prepending to the format string. In C programming, this syntax is called string literal concatenation. (In other high-level programming languages, string concatenation is generally done with + operator). For the function printk and log level, you can find more information in include/linux/kern_levels.h and include/linux/printk.h.

Note: The path to header files for Linux kernel module development is different from the one you often used for the application development. Don’t try to find the header file inside /usr/include/linux, instead please use the following path /lib/modules/`uname -r`/build/include/linux (uname -r command returns your kernel version).

Next, let’s build this hello-world kernel module.

Build the module

The way to build a kernel module is a little different from how to build a user-space application. The efficient solution to build kernel image and its modules is Kernel Build System(Kbuild).

Kbuild is a complex topic and I won’t explain it in too much detail here. Simply speaking, Kbuild allows you to create highly customized kernel binary images and modules. Technically, each subdirectory contains a Makefile compiling only the source code files in its directory. And a top-level Makefile recursively executes each subdirectory’s Makefile to generate the binary objects. And you can control which subdirectories are included by defining config files. In detail, you can refer to other documents.

The following is the Makefile for the hello world module:

1
2
3
4
5
6

obj-m += hello.o
PWD := $(CURDIR)
all:
        make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
clean:
        make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

The make -C dir command changes to directory dir before reading the makefiles or doing anything else. The top-level Makefile in /lib/modules/$(shell uname -r)/build will be used. You can find that command make M=dir modules is used to make all modules in specified dir.

And in the module-level Makefile, the obj-m syntax tells kbuild system to build module_name.o from module_name.c, and after linking, will result in the kernel module module_name.ko. In our case, the module name is hello.

The build process goes as follows:

1
2
3
4
5
6
7
8
9

chrisbao:~/develop/kernel/hello-1$ sudo make
make -C /lib/modules/4.15.0-176-generic/build M=/home/DIR/jbao6/develop/kernel/hello-1  modules
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-176-generic'
  CC [M]  /home/DIR/jbao6/develop/kernel/hello-1/hello.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/DIR/jbao6/develop/kernel/hello-1/hello.mod.o
  LD [M]  /home/DIR/jbao6/develop/kernel/hello-1/hello.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-176-generic'

After the build, you can get several new files in the same directory:

1
2

chrisbao:~/develop/kernel/hello-1$ ls
hello.c  hello.ko  hello.mod.c  hello.mod.o  hello.o  Makefile  modules.order  Module.symvers

The file ends with .ko is the kernel module. You can ignore other files now, I will write another article later to have a deep discussion about the kernel module system.

Load the module

With the file command, you can note that the kernel module is an ELF(Executable and Linkable Format) format file. ELF files are typically the output of a compiler or linker and are a binary format.

1
2

chrisba:~/develop/kernel/hello-1$ file hello.ko
hello.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=f0da99c757751e7e9f9c4e55f527fb034a0a4253, not stripped

Next step, let’s try to install and remove the module dynamically. You need to know the following three commands:

• lsmod: shows the list of kernel modules currently loaded.
• insmod: inserts a module into the Linux Kernel by running sudo insmod module_name.ko
• rmmod: removes a module from the Linux Kernel by running sudo rmmod module_name

Since the hello world module is quite simple, you can easily install and remove the module as you wish. I will not show the detailed commands here and leave it to the readers.

Note: It doesn’t mean that you can easily install and remove any kernel module without any issues. If the module you are loading has bugs, the entire system can crash.

Debug the module

Next step, let’s prove that the hello world module is installed and removed as expected. We will use dmesg command. dmesg (diagnostic messages) can print the messages in the kernel ring buffer.

First, a ring buffer is a data structure that uses a single, fixed-size buffer as if it were connected end-to-end. The kernel ring buffer is a ring buffer that records messages related to the operation of the kernel. As we mentioned above, the kernel logs printed by the printk function will be sent to the kernel ring buffer.

We can find the messages produced by our module with command dmesg | grep world as follows:

1
2
3
4
5
6

chrisbao:~$ dmesg | grep world

[2147137.177254] Hello, world
[3281962.445169] Goodbye, world
[3282008.037591] Hello, world
[3282054.921824] Goodbye, world

Now you can see that the hello world is loaded into the kernel correctly. And it can be removed dynamically as well. Great.

Summary

In this article, we examine how to write a kernel module, how to build it and how to install it into the kernel dynamically. Next article we can work on the mini-firewall as a Netfilter module.

Firewalls are an important tool that can be configured to protect your servers and infrastructure. Firewalls’ main functionalities are filtering data, redirecting traffic, and protecting against network attacks. There are both hardware-based firewalls and software-based firewalls. I will not discuss too much about the background here, since you can find many online documents about it.

Have you ever thought of implementing a simple firewall from scratch? Sounds crazy? But with the power of Linux, you can do that. After you read this series of articles, you will find that actually, it is quite simple.

You may once use various firewalls on Linux such as iptables, nftables, UFW, etc. All of these firewall tools are user-space utility programs, and they are all relying on Netfilter. Netfilter is the Linux kernel subsystem that allows various networking-related operations to be implemented. Netfilter allows you to develop your firewall using the Linux Kernel Module. If you don’t know the techniques such as the Linux Kernel module and Netfilter, don’t worry. In this article, let’s write a Linux firewall from scratch based on Netfilter. You can learn the following interesting points:

• Linux kernel module development.
• Linux kernel network programming.
• Netfilter module development.

Netfilter and Kernel modules

Basics of Netfilter

Netfilter can be considered to be the third generation of firewall on Linux. Before Netfilterwas introduced in Linux Kernel 2.4, there are two older generations of firewalls on Linux as follows:

• The first generation was a port of an early version of BSD UNIX’s ipfw to Linux 1.1.
• The second generation was ipchains developed in the 2.2 series of Linux Kernel.

As we mentioned above, Netfilter was designed to provide the infrastructure inside the Linux kernel for various networking operations. So firewall is just one of the multiple functionalities provided by Netfilter as follows:

• Packet filtering: is in charge of filtering the packets based on the rules. It is also the topic of this article.
• NAT (Network address translation): is in charge of translating the IP address of network packets. NAT is an important protocol, which has become a popular and essential tool in conserving global address space in the face of IPv4 address exhaustion. If you don’t know NAT protocol, you can refer to other documents. I will examine it in other future articles.
• Packet mangling: is in charge of modifying the packet content(In fact, NAT is one kind of packet mangling, which modifies the source or destination IP address). For example, MSS (Maximum Segment Size) value of TCP SYN packets can be altered to allow large-size packets transported over the network.

Note: this article will focus on building a simple firewall to filter packets based on Netfilter. So the NAT and Packet Mangling parts are not in the scope of this article.

Packet filtering can only be done inside the Linux kernel (Netfilter’s code is in the kernel as well), if we want to write a mini firewall, it has to run in the kernel space. Right? Does it mean we need to add our code into the kernel and recompile the kernel? Imagine you have to recompile the kernel each time you want to add a new packet filtering rule. That’s a bad idea. The good news is that Netfilter allows you to add extensions using the Linux kernel modules.

Basics of Linux Kernel modules

Although Linux is a monolithic kernel, it can be extended using kernel modules. Modules can be inserted into the kernel and removed on demand. Linux isolates the kernel but allows you to add specific functionality on the fly through modules. In this way, Linux keeps a balance between stability and usability.

I want to examine one confusing point about the kernel module here: what is the difference between driver and module:

• A driver is a bit of code that runs in the kernel to talk to some hardware device. It drives the hardware. Standard practice is to build drivers as kernel modules where possible, rather than link them statically to the kernel since that gives more flexibility.
• A kernel module may not be a device driver at all.

Summary

In the first post of this series, we examine the basics of Netfilter and Linux kernel modules. In the next post, let’s start implementing the mini firewall.

In the previous article, we examined how to develop a network sniffer with PF_SOCKET socket in Linux platform. The sniffer developed in the last article captures all the network packets. But a powerful network sniffer like tcpdump should provide the packet filtering functionality. For instance, the sniffer can only capture the TCP segment(and skip the UPD), or it can only capture the packets from a specific source IP address. In this article, let’s continue to explore how to do that.

Background of BPF

Berkeley Packet Filter(BPF) is the essential underlying technology for packet capture in Unix-like operating systems.
Search BPF as the keyword online, and the result is very confusing. It turns out that BPF keeps evolving, and there are several associated concepts such as BPF cBPF eBPF and LSF. So let us examine those concepts along the timeline:

• In 1992, BPF was first introduced to the BSD Unix system for filtering unwanted network packets. The proposal of BPF was from researchers in Lawrence Berkeley Laboratory, who also developed the libpcap and tcpdump.

• In 1997, Linux Socket Filter(LSF) was developed based on BPF and introduced in Linux kernel version 2.1.75. Note that LSF and BPF have some distinct differences, but in the Linux context, when we speak of BPF or LSF, we mean the same packet filtering mechanism in the Linux kernel. We’ll examine the detailed theory and design of BPF in the following sections.

• Originally, BPF was designed as a network packet filter. But in 2013, BPF was widely extended, and it can be used for non-networking purposes such as performance analysis and troubleshooting. Nowadays, the extended BPF is called eBPF, and the original and obsolete version is renamed to classic BPF (cBPF). Note that what we examine in this article is cBPF, and eBPF is not inside the scope of this article. eBPF is the hottest technology in today’s software world, and I’ll talk about it in the future.

Where to place BPF

The first question to answer is where should we place the filter. The last article examines the path of a received packet as follows:

The best solution to this question is to put the filter as early as possible in the path. Since copying a large amount of data from kernel space to the user space produces a huge overhead, which can influence the system performance a lot. So BPF is a kernel feature. The filter should be triggered immediately when a packet is received at the network interface.As the original BPF paper said To minimize memory traffic, the major bottleneck in most modern system, the packet should be filtered ‘in place’ (e.g., where the network interface DMA engine put it) rather than copied to some other kernel buffer before filtering.
Let’s verify this behavior by examining the kernel source code as follows (Note the kernel code shown in this article is based on version 2.6, which contains the cBPF implementation.):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

/* source code file of net/packet/af_packet.c */
/* packet_create: create socket */
static int packet_create(struct net *net, struct socket *sock, int protocol)
{
    /* some code omitted ... */
po = pkt_sk(sk);
sk->sk_family = PF_PACKET;
po->num = proto;

spin_lock_init(&po->bind_lock);
po->prot_hook.func = packet_rcv; // attach hook function to socket

if (sock->type == SOCK_PACKET)
po->prot_hook.func = packet_rcv_spkt; // attach hook function to socket

if (proto) {
po->prot_hook.type = proto;
dev_add_pack(&po->prot_hook);
sock_hold(sk);
po->running = 1;
}
}

packet_create function handles the socket creation when the application calls the socket system call. In lines 11 and 14, it attaches the hook function to the socket. The hook function executes when the packet is received.

The following code block shows the hook function packet_rcv:

1
2
3
4
5
6
7
8
9
10
11
12

/* hook function packet_rcv is triggered, when the packet is received */
static int packet_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *orig_dev)
{
    /* some code omitted ... */
    sk = pt->af_packet_priv;
    snaplen = skb->len;
    res = run_filter(skb, sk, snaplen); // filter logic
    if (!res)
    goto drop_n_restore; // drop the packet

    __skb_queue_tail(&sk->sk_receive_queue, skb); // put the packet into the queue
}

packet_rcv function calls run_filter, which is just the BPF logic part(Currently, you can regard it as a black box. In the next section, we’ll examine the details). Based on the return value of run_filter the packet can be filtered out or put into the queue.

So far, you can understand BPF(or the packet filtering) is working inside kernel space. But the packet sniffer is a user-space application. The next question is how to link the filtering rules in user space to the filtering handler in kernel space.

To answer this question, we have to understand BPF itself. It’s right time to understand this great piece of work.

BPF machine

As I mentioned above, BPF was introduced in this original paper written by researchers from Berkeley. I strongly recommend you read this great paper based on my own experience. In the beginning, I felt crazy to read it, so I read other related documents and tried to understand BPF. But most documents only cover one portion of the entire system, so it is difficult to piece all the information together. Finally, I read the original paper and connected all parts together. As the saying goes, sometimes taking time is actually a shortcut.

Virtual CPU

A packet filter is simply a boolean-valued function on a packet. If the value of the function is true the kernel copies the packet for the application; if it is false the packet is ignored.

In order to be as flexible as possible and not to limit the application to a set of predefined conditions, the BPF is actually implemented as a register-based virtual machine (for the difference between stack-based and register-based virtual machine, you can refer to this article) running a user-defined program.

You can regard the BPF as a virtual CPU. And it consists of an accumulator, an index register(x), a scratch memory store, and an implicit program counter. If you’re not familiar with these concepts, I add some simple illustrations as follows:

• An accumulator is a type of register included in a CPU. It acts as a temporary storage location holding an intermediate value in mathematical and logical calculations. For example, in the operation of “1+2+3”, the accumulator would hold the value 1, then the value 3, then the value 6. The benefit of an accumulator is that it does not need to be explicitly referenced.
• An index register in a computer’s CPU is a processor register or assigned memory location used for modifying operand addresses during the run of a program.
• A program counter is a CPU register in the computer processor which has the address of the next instruction to be executed from memory.

In the BPF machine, the accumulator is used for arithmetic operations, while the index register provides offsets into the packet or the scratch memory areas.

Instructions set and addressing mode

Same as the physical CPU, the BPF provides a small set of arithmetic, logical and jump instructions as follows, these instructions run on the BPF virtual machine(or CPU):

The first column opcodes lists the BPF instructions written in an assembly language style. For example, ld, ldh and ldb means to copy the indicated value into the accumulator. ldx means to copy the indicated value into the index register. jeq means jump to the target instruction if the accumulator equals the indicated value. ret means return the indicated value. You can check the functionality of the instructions set in detail in the paper.

This kind of assembly-like style is more readable to humans. But when we develop an application (like the sniffer written in this article), we use binary code directly as the BPF instruction. This kind of binary format is called BPF Bytecode. I’ll examine the way to convert this assembly language to bytecode later.

The second column addr modes lists the addressing modes allowed for each instruction. The semantics of the addressing modes are listed in the following table:

For instance, [k] means the data at byte offset k in the packet. #k means the literal value stored in k. You can read the paper in detail to check the meaning of other address modes.

Example BPF program

Now let’s try to understand the following small BPF program based on the knowledge above:

1
2
3
4

(000) ldh      [12]
(001) jeq      #0x800           jt 2    jf 3
(002) ret      #262144
(003) ret      #0

The BPF program consists of an array of BPF instructions. For example, the above BPF program contains four instructions.

The first instruction ldh loads a half-word(16-bit) value into the accumulator from offset 12 in the Ethernet packet. According to the Ethernet frame format shown below, the value is just the Ethernet type field. The Ethernet type is used to indicate which protocol is encapsulated in the frame’s payload (for example, 0x0806 for ARP, 0x0800 for IPv4, and 0x86DD for IPv6).

The second instruction jeq compares the accumulator (currently stores Ethernet type field) to 0x800(stands for IPv4). If the comparison fails, zero is returned, and the packet is rejected. If it is successful, a non-zero value is returned, and the packet is accepted. So the small BPF program filters and accepts all IP packets. You can find other BPF programs in the original paper. Go to read it, and you can feel the flexibility of BPF as well as the beauty of the design.

Kernel implementation of BPF

Next, let’s examine how kernel implements BPF. As mentioned above, the hook function packet_rcv calls run_filter to handle the filtering logic. run_filter is defined as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

/* Copied from net/packet/af_packet.c */
/* function run_filter is called in packet_rcv*/
static inline unsigned int run_filter(struct sk_buff *skb, struct sock *sk,
      unsigned int res)
{
struct sk_filter *filter;

rcu_read_lock_bh();
filter = rcu_dereference(sk->sk_filter); // get the filter bound to the socket
if (filter != NULL)
res = sk_run_filter(skb, filter->insns, filter->len); // the filtering is inside sk_run_filter function
rcu_read_unlock_bh();

return res;
}

You can find that the real filtering logic is inside sk_run_filter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

unsigned int sk_run_filter(struct sk_buff *skb, struct sock_filter *filter, int flen)
{
struct sock_filter *fentry;/* We walk down these */
void *ptr;
u32 A = 0;/* Accumulator */
u32 X = 0;/* Index Register */
u32 mem[BPF_MEMWORDS];/* Scratch Memory Store */
u32 tmp;
int k;
int pc;

/*
 * Process array of filter instructions.
 */
for (pc = 0; pc < flen; pc++) {
fentry = &filter[pc];

switch (fentry->code) {
case BPF_ALU|BPF_ADD|BPF_X:
A += X;
continue;
case BPF_ALU|BPF_ADD|BPF_K:
A += fentry->k;
continue;
case BPF_ALU|BPF_SUB|BPF_X:
A -= X;
continue;
case BPF_ALU|BPF_SUB|BPF_K:
A -= fentry->k;
continue;
case BPF_ALU|BPF_MUL|BPF_X:
A *= X;
continue;
/* some code omitted ... */
case BPF_RET|BPF_K:
return fentry->k;
case BPF_RET|BPF_A:
return A;
case BPF_ST:
mem[fentry->k] = A;
continue;
case BPF_STX:
mem[fentry->k] = X;
continue;
default:
WARN_ON(1);
return 0;
}
}

return 0;
}

Same as we mentioned, sk_run_filter is simply a boolean-valued function on a packet. It maintains the accumulator, the index register, etc. as local variables. And process the array of BPF filter instructions in a for loop. Each instruction will update the value of local variables. In this way, it simulates a virtual CPU. Interesting, right?

BPF JIT

Since each network packet must go through the filtering function, it becomes the performance bottleneck of the entire system.

A just-in-time (JIT) compiler was introduced into the kernel in 2011 to speed up BPF bytecode execution.

• What is a JIT compiler? A JIT compiler runs after the program has started and compiles the code(usually bytecode or some type of VM instructions) on the fly(or just in time) into a form that’s usually faster, typically the host CPU’s native instruction set. This is in contrast to a traditional compiler that compiles all the code to machine language before the program is first run.

In the BPF case, the JIT compiler translates BPF bytecode into a host system’s assembly code directly, which can optimize the performance a lot. I’ll not show details about JIT in this article. You can refer to the kernel code.

Set BPF in sniffer

Next, let’s add BPF into our packet sniffer. As we mentioned above in the application level, the BPF instructions should use bytecode format with the following data structure:

1
2
3
4
5
6

struct sock_filter {    /* Filter block */
        __u16   code;   /* Actual filter code */
        __u8    jt;     /* Jump true */
        __u8    jf;     /* Jump false */
        __u32   k;      /* Generic multiuse field */
};

How can we convert the BPF assembly language into bytecode? There are two solutions. First, there is a small helper tool called bpf_asm(which is provided along with the Linux kernel), and you can regard it as the BPF assembly language interpreter. But it is not recommended to application developers.

Second, we can use tcpdump, which provides the converting functionality. You can find the following information from the tcpdump man page:

• -d: Dump the compiled packet-matching code in a human-readable form to standard output and stop.

• -dd: Dump packet-matching code as a C program fragment.

• -ddd: Dump packet-matching code as decimal numbers (preceded with a count).

tcpdump ip means we want to capture all the IP packets. With options -d, -dd and -ddd, the output goes as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

baoqger@ubuntu:~$ sudo tcpdump -d ip
[sudo] password for baoqger:
(000) ldh      [12]
(001) jeq      #0x800           jt 2    jf 3
(002) ret      #262144
(003) ret      #0

baoqger@SLB-C8JWZH3:~$ sudo tcpdump -dd ip
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 1, 0x00000800 },
{ 0x6, 0, 0, 0x00040000 },
{ 0x6, 0, 0, 0x00000000 },

baoqger@SLB-C8JWZH3:~$ sudo tcpdump -ddd ip
4
40 0 0 12
21 0 1 2048
6 0 0 262144
6 0 0 0

Option -d prints the BPF instructions in assembly language (same as the example BPF program shown above). Options -dd prints the bytecode as a C program fragment. So tcpdump is the most convenient tool when you want to get the BPF bytecode.

The BPF filter bytecode (wrapped in the structure sock_fprog) can be passed to the kernel through setsockopt system call as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

// attach the filter to the socket
// the filter code is generated by running: tcpdump tcp
struct sock_filter BPF_code[] = {
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 1, 0x00000800 },
{ 0x6, 0, 0, 0x00040000 },
{ 0x6, 0, 0, 0x00000000 }
};    
struct sock_fprog Filter;
// error prone code, .len field should be consistent with the real length of the filter code array
Filter.len = sizeof(BPF_code)/sizeof(BPF_code[0]); 
Filter.filter = BPF_code;


if (setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, &Filter, sizeof(Filter)) < 0) {
perror("setsockopt attach filter");
close(sock);
exit(1);
} 

setsockopt system call triggers two kernel functions: sock_setsockopt and sk_attach_filter (I’ll not show the details for these two functions), which binds the filters to the socket. And in run_filter kernel function (mentioned above), it can get the filters from the socket and execute the filters on the packet.

So far, every piece is connected. The puzzle of BPF is solved. The BPF machine allows the user-space applications to inject customized BPF programs straight into a kernel. Once loaded and verified, BPF programs execute in kernel context. These BPF programs operate inside kernel memory space with access to all the internal kernel states available to it. For example, the cBPF machine which uses the network packet data. But this power can be extended as eBPF, which can be used in many other varied applications. As someone said In some way, eBPF does to the kernel what Javascript does to the websites: it allows all sorts of new application to be created. In the future, I plan to examine eBPF in depth.

Process the packet

We examined the BPF filtering theory on the kernel level a lot in the above section. But for our tiny sniffer, the last step we need to do is process the network packet.

• First, the recvfrom system call reads the packet from the socket. And we put the system call in a while loop to keep reading the incoming packets.

• Then, we print the source and destination MAC address in the packet(the packet we got is a raw Ethernet frame in Layer 2, right?). And if what this Ethernet frame contains is an IP4 packet, then we print out the source and destination IP address. To understand more about it, you can study the header format of various network protocols. I will not cover in details here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

while(1) {
printf("-----------\n");
n = recvfrom(sock, buffer, 2048, 0, NULL, NULL);
printf("%d bytes read\n", n);

/* Check to see if the packet contains at least
* complete Ethernet (14), IP (20) and TCP/UDP
* (8) headers.
*/
if (n < 42) {
perror("recvfrom():");
printf("Incomplete packet (errno is %d)\n", errno);
close(sock);
exit(0);
}

ethhead = buffer;
printf("Source MAC address: %.2x:%.2x:%.2x:%.2x:%.2x:%.2x\n",
ethhead[0], ethhead[1], ethhead[2], ethhead[3], ethhead[4], ethhead[5]
);
printf("Destination MAC address: %.2x:%.2x:%.2x:%.2x:%.2x:%.2x\n",
ethhead[6], ethhead[7], ethhead[8], ethhead[9], ethhead[10], ethhead[11]
);

iphead = buffer + 14; 

if (*iphead==0x45) { /* Double check for IPv4
* and no options present */
printf("Source host %d.%d.%d.%d\n",
iphead[12],iphead[13],
iphead[14],iphead[15]);
printf("Dest host %d.%d.%d.%d\n",
iphead[16],iphead[17],
iphead[18],iphead[19]);
printf("Source,Dest ports %d,%d\n",
(iphead[20]<<8)+iphead[21],
(iphead[22]<<8)+iphead[23]);
printf("Layer-4 protocol %s\n", transport_protocol(iphead[9]));
}
}

You can find the complete source code of the sniffer in this Github repo.

Summary

In this article, we examine how to add filters to our sniffer. First, we analyze why the filter should be running inside kernel space instead of the application space. Then, this article examines the BPF machine design and implementation in detail based on the paper. We reviewed the kernel source code to understand how to implement the BPF virtual machine. As I mentioned above, the original BPF(cBPF) was extended to eBPF now. But the understanding of the BPF virtual machine is very helpful to eBPF as well.

When we refer to network packet sniffer, some famous and popular tools come to your mind, like tcpdump. I have shown you how to capture network packets with such tools in my previous articles. But have you ever thought about writing a packet sniffer from scratch without dependencies on any third-party libraries? We need to dig deep into the operating system and find the weapons needed to build this tool. Sounds complex, right? In this article, let us do it. After reading this article, you can find that it is not as difficult as you think.

Note that different operating system kernels have different internal network implementations. This article will focus on the Linux platform.

Introduction

Firstly, we need to review how tcpdump is implemented. According to the official document, tcpdump is built on the library libpcap, which is developed based on the remarkable research result from Berkeley, in details you can refer to this paper.

As you know, different operating systems have different internal implementations of network stacks. libpcap covers all of these differences and provides the system-independent interface for user-level packet capture. I want to focus on the Linux platform, so how does libpcap work on the Linux system? According to some documents, it turns out that libpcap uses the PF_PACKET socket to capture packets on a network interface.

So the next question is: what the PF_PACKET socket is?

PF_PACKET socket

In my previous article, we mentioned that the socket interface is TCP/IP’s window on the world. In most modern systems incorporating TCP/IP, the socket interface is the only way applications can use the TCP/IP suite of protocols.

It is correct. This time, let’s dig deeper about socket by examining the system call executed when we create a new socket:

1

int socket(int domain, int type, int protocol);

When you want to create a socket with the above system call, you have to specify which domain (or protocol family) you want to use with that socket as the first argument. The most commonly used family is PF_INET, which is for communications based on IPv4 protocols (when you create a TCP server, you use this family). Moreover, you have to specify a type for your socket as the second argument. And the possible values depend on the family you specified. For example, when dealing with the PF_INET family, the values for type include SOCK_STREAM(for TCP) and SOCK_DGRAM(for UDP). For other detailed information about the socket system call, you can refer to the socket(3) man page.

You can find one potential value for the domain argument as follows:

1

AF_PACKET    Low-level packet interface

Note: AF_PACKET and PF_PACKET are same. It is called PF_PACKET in history and then renamed AF_PACKET later. PF means protocol families, and AF means address families. In this article, I use PF_PACKET.

Different from PF_INET socket, which can give you TCP segment. By PF_PACKET socket, you can get the raw Ethernet frame which bypasses the usual upper layer handling of TCP/IP stack. It might sound a little bit crazy. But, that is, any packet received will be directly passed to the application.

For a better understanding of PF_PACKET socket, let us go deeper and roughly examine the path of a received packet from the network interface to the application level.

(As shown in the image above) When the network interface card(NIC) receives a packet, it is handled by the driver. The driver maintains a structure called ring buffer internally. And write the packet to kernel memory (the memory is pre-allocated with ring buffer) with direct memory access(DMA). The packet is placed inside a structure called sk_buff(one of the most important structures related to kernel network subsystem).

After entering the kernel space, the packet goes through protocol stack handling layer by layer, such as IP processing and TCP/UDP processing. And the packet goes into applications via the socket interface. You already understand this familiar path very well.

But for the PF_PACKET socket, the packet in sk_buff is cloned, then it skips the protocol stacks and directly goes to the application. The kernel needs the clone operation, because one copy is consumed by the PF_PACKET socket, and the other one goes through the usual protocol stacks.

In future articles, I’ll demonstrate more about Linux kernel network internals.

Next step, let us see how to create a PF_PACKET socket at the code level. For brevity, I omit some code and only show the essential part. You can refer to this Github repo in detail.

1
2
3
4

if ((sock = socket(PF_PACKET, SOCK_RAW, htons(ETH_P_IP))) < 0) {
    perror("socket");
    exit(1);
}

Please ensure to include the system header files: <sys/socket.h> <sys/types.h>.

Bind to one network interface

Without the additional settings, the sniffer captures all the packets received on all the network devices. Next step, let us try to bind the sniffer to a specific network device.

Firstly, you can use ifconfig command to list all the available network interfaces on your machines. The network interface is a software interface to the networking hardware.

For example, the following image shows information of network interface eth0:

1
2
3
4
5
6
7
8

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.230.49  netmask 255.255.240.0  broadcast 192.168.239.255
        inet6 fe80::215:5dff:fefb:e31f  prefixlen 64  scopeid 0x20<link>
        ether 00:15:5d:fb:e3:1f  txqueuelen 1000  (Ethernet)
        RX packets 260  bytes 87732 (87.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 178  bytes 29393 (29.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0