Friday, April 22, 2016

Security Considerations: Enabling/Disabling Ping /Traceroute for Your Network

There are two Linux tools (i.e., Ping and Traceroute) common for monitoring network connections. However, because of cycle attacks, most systems have been hardened to disable them in hope of staving off attacks and the surveillance of network mapping tools.

In this article, we will demonstrate how to secure your network by using two approaches:
  • Uses IP Filters
    • Use filters to prevent attacks from crossing the firewall
  • Use NAT (Network Address Translation)
    • Define the name/address mappings for trusted nodes in your secure network

Ping and Traceroute

The main difference between Ping and Traceroute commands is that
  • Ping
    • Is a quick and easy way to tell you if the destination server is online and estimates how long it takes to send and receive data to the destination
  • Traceroute
    • Tells you the exact route you take to reach the server from your computer and how long each hop takes


Ping generates an ICMP echo request message and expects to receive an echo reply message in response. Echo request is a relatively safe message, but any of the ICMP messages can be used by an outsider in order to gain some knowledge of your network or to directly attack your system. Also, like every protocol that you allow, ICMP messages can be used to overwhelm your systems in a denial of service attack ( i.e., ICMP ping responses are used as a covert-channel). Unlike the higher layer protocols, such as TCP or UDP, there is not a source port nor a destination port, just the message type and code.[7]

Traceroute (or tracert on Windows)

Traceroute is useful in allowing network administrators to track the path that an IP packet is following in order to reach its final destination. It works by sending UDP packets from one high port (port number > 1023) to another high port. It selects a free UDP Port and starts to send packets to different high ports. If you see a series of UDP packets within this port range (i.e., 33434 - 33600), then it is probably indicative of traceroute.

In order to discover the path, it plays some tricks with the TTL value of the packet (this field must be decremented by routers everytime they forward the packet). First it sends a UDP packet with TTL=1, so the first router gets the packet, decrements the TTL field, and then discards the packet because the TTL reached 0. After discarding the packet, the router sends an ICMP TTL exceeded message to the sender, so the sender learns the address of the first hop. Then it uses a TTL value of two, and it gets the second router address. It keeps getting router addresses with TTL exceeded messages until the packet reaches the destination host.

IP Filters

Firewall implementation normally use IP filters to control which packets are passed and which are blocked on each side.[1] The information it uses to decide whether to block or pass a packet is largely contained in the packet headers.

Some of the filtering criteria are:
  • The source and destination IP address
  • The direction of flow
  • The IP protocol (ICMP, TCP, UDP or other protocols)
  • The interface where the packet is detected (secure or nonsecure)

Considerations of ICMP Filtering

ICMP is a protocol designed to communicate errors and information between hosts that are processing IP datagrams. In other words, ICMP messages are the "control messages" for TCP/IP. There are many different types of ICMP messages. For example, type 8 ICM messages are echo requests and type 0 ICM messages are echo replies.

The echo (a,k.a. echo request) message is used to check if a host is up or down. When a host receives the request, it sends back an echo reply message. These messages are usually generated by a ping command, but may also be generated by a network management station that is polling the nodes of a network.

The simplest approach to secure ICMP protocol is to block all ICMP messages from crossing the firewall. However, the problem is that ICMP are the "control messages" for TCP/IP. If you block all incoming ICMP, then you may break some essential networking. So, you want to be more selective about which ICMP messages you want to allow.

The absolute minimum ICMP traffic to allow is the packets dealing with TCP path MTU discovery. Fragmenting a stream is more efficient at the TCP layer rather than the IP layer, so the TCP layer will try to discover when IP packets are being inadvertently fragmented. They do this by setting the "DF" (Don't Fragment) on all outgoing packets. When a router cannot forward the packet because it is too big, rather than fragmenting it, it sends back a "fragmentation needed" ICMP packet (type=3/code=4). The TCP stack then starts sending smaller IP packets, segmenting the data at the TCP layer rather than allow routers to fragment at the IP layer. Therefore, firewalls must be configured to allow incoming ICMP type=3, code=4 packets.

How to Protect against Ping?

To counter the surveillance from ping commands, you may consider to:[1]
  • Permit the outgoing echo request and incoming echo reply
  • Deny the incoming echo request and outgoing echo reply.
You could consider enabling this settings to some key hosts, such as the router of your network provider.

Another consideration is Host unreachable and Destination Unreachable packets (type=3/code=1). Allowing these to come in through your firewall will allow connections to timeout faster, but they can also be used as a denial of service attack (by disconnecting clients from servers).

How to Protect against Traceroute?

Using traceroute, we know it involves several UDP packets flowing from the sender to the destination, The danger of exposing your traceroute service to nonsecure network is that an attacker can use it to find out which hosts are the routers in your network. This tool manipulates the TTL option of a UDP packet, in order to receive an ICMP TTL exceeded message in response . Blocking the outgoing TTL messages (type=11/code=0) will help you hide your network structure.

To counter any attacks from traceroute command, the simplest approach is blocking TTL exceeded messages (type=11/code=0) from going from the secure network to the nonsecure network. In summary, here are what you want to do:[1]

Traceroute from the Firewall:
  • This configuration can be safely permitted. In order to do this, you must send high UDP packets and accept ICMP TTL (type=11/code=0) and port unreachable messages (type=3/code=3).

Traceroute from Internet to the Firewall:
  • Block the outgoing TTL messages (type=11/code=0)
  • In addition, you also want to block the outgoing ICMP port unreachable messages (type=3/code=3) because it would be useful to an attacker as a fast way to discover which services you are providing.

To find out which high port is used by your traceroute service, do:

#cat /etc/services
traceroute 33434/tcp
traceroute 33434/udp

Securing Network by Using NAT

Network address translation (NAT) can be utilized to secure your network by
  1. Providing access to nonsecure network name/address mappings for users in the secure network
  2. Hiding the secure network names and addresses from users outside the secure network
  3. Providing name/address mapping for resources that you want to reveal (usually servers and gateways
Here is how the address translation is done, For each,
  • Outgoing IP packet
    • The source address is checked by the NAT configuration rules.
      • If a rule matches the source address, the address is translated to an official address from the predefined address pool.
  • Incoming packet
    • The destination address is checked if it is used by NAT.
      • When this is true the address is translated to the original unofficial address.

Note that only TCP and UDP packets are translated by NAT. The ICMP protocol is not supported by NAT. For example, pinging to the NAT addresses does not work, because ping uses the ICMP protocol.

Allowing Ping/Traceroute from Firewall to Internet

Users will constantly ask for the ability to ping and traceroute machines on the Internet. Most firewall adminsitrators will eventually give in to these demands. Nobody really needs to ping/traceroute, but they really want to.

To protect your network, basically you will create filter rules that will route packets from a secure network to the Internet and back. NAT will take care of the address translation of the secure addresses. Normally, NAT translation will occur for the outgoing packet after the packet has gone through both packet filters (secure and non-secure). This means that you should never mention NAT addresses in the filter rules.


  1. Protect and Survive Using IBM Firewall 3.1 for AIX (pdf)
  2. differences between ping and traceroute
  3. Default TCP Port
  4. Ping Blocking: How to do and how to break?
  5. tracert (Windows; traceroute in Linux)
  6. Firewall Forensics (What am I seeing?)
  7. ICMP Types and Codes
    • There are many different types of ICMP messages.
  8. ICMP (RFC 792)
  9. IP Fragmentation: How to Avoid It?
  10. Docker Container Networks: All Things Considered (Xml and More)

Tuesday, April 19, 2016

Docker: Btrfs Storage in Practice

One of the ways Docker makes containerization so easy is by managing an overlay-style filesystem, allowing containers and images to incrementally change the filesystem layout of the image without requiring large copies of multiple images kicking around. This is a copy-on-write approach: parent layers are held read-only, and changes are reflected in the working layer.

The Docker has support for different image/container layer storage drivers:[7]
  • aufs
  • btrfs
  • devicemapper
  • overlay
  • vfs
  • zfs 
Your choice of storage driver can affect the performance of your containerized applications. So it’s important to understand the different storage driver options available and select the right one for your application.

In this article, we will focus only on btrfs (B-tree file system) storage.

Storage Driver and Backing Filesystem

To begin with, let's study different storage-related entities that form a container:
  • Images
    • Is a tagged hierarchy of read-only layers plus some metadata
    • docker images command can be used to list all images and report their virtual sizes
  • Image layers
    • Each successive layer (with a UUID tag) builds on top of the layer below it
    • Reuse layers can improve image build time[8]
      • Each Dockerfile instruction generates a new layer
      • You should put instructions least likely to change at the top of your Dockerfile to reuse layers as much as possible and try to make changes only at the bottom of your Dockerfile.
    • Image layers can be shared among images
    • Docker limits the number of layers to 127
      • Layers don’t come for free, depending on storage driver used there are some penalties to pay
        • For example, in AUFS, each layer can introduce latency to container write performance on the first write to each file existing in the image layers stack, especially if the file is big and exists below many image layers.
    • docker history command can be used to list all layers of an image
  • Storage drivers
    • Docker has a pluggable storage driver architecture. 
      • This gives you the flexibility to “plug in” the storage driver that is best for your environment and use-case.
    • Each Docker storage driver is based on a Linux filesystem or volume manager
    • The Docker daemon can only run one storage driver, and all containers created by that daemon instance use the same storage driver.
    • Each storage driver is free to implement the management of image layers and the container layer in its own unique way. 
      • This means some storage drivers perform better than others in different circumstances.
      • See [7] to learn more on which storage driver you should choose

Storage Driver and Backing Filesystem

Which storage driver you use, in part, depends on the backing filesystem you plan to use for your Docker host’s local storage area. Some storage drivers can operate on top of different backing filesystems. However, other storage drivers require the backing filesystem to be the same as the storage driver. For example, the btrfs storage driver requires a btrfs backing filesystem. 

The following table lists each storage driver and whether it must match the host’s backing file system or not:

|Storage driver |Must match backing filesystem |
|overlay        |No                            |
|aufs           |No                            |
|btrfs          |Yes                           |
|devicemapper   |No                            |
|vfs*           |No                            |
|zfs            |Yes                           |

The btrfs Backend

The backing filesystem refers to the filesystem that was used to create the Docker host’s local storage area under /var/lib/docker.  The brtfs (B-tree file system) backend requires /var/lib/docker to be on a btrfs filesystem and uses the filesystem level snapshotting to implement layers.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdb4       11G  7.1G  2.5G  75% /var/lib/docker

# mount -l
/dev/xvdb4 on /var/lib/docker type btrfs (rw)

You can find the layers of the images in the folder /var/lib/docker/btrfs/subvolumes.  Each layer is stored as a btrfs subvolume inside the folder  and start out as a snapshot of the parent subvolume (if any).

The btrfs driver is very fast for docker build - but like devicemapper does not share executable memory between devices. Mounting /var/lib/docker on a different filesystem than the rest of your system is recommended in order to limit the impact of filesystem corruption.

You can set the storage driver by passing the --storage-driver= option to the docker command line, or by setting the option on the DOCKER_OPTS line in the /etc/default/docker file.  For example, to set the btrfs storage driver, do:
# docker -d -s btrfs -g /mnt/btrfs_partition ...

  -s, --storage-driver=""  Storage driver to use
  -g, --graph=""           Path to use as the root of the Docker runtime.  
                             Default is /var/lib/docker.

To verify if btrfs storage driver is used in your docker container, do:
# docker info
Containers: 1
Images: 19
Storage Driver: btrfs


  1. Btrfs
  2. LVM dangers and caveats
  3. 20 Linux Server Hardening Security Tips
  4. ZFS Vs. BTRFS
  5. How to Use Different Docker Filesystem Backends
  6. Daemon storage-driver option
  7. Select a storage driver
  8. Optimizing Docker images for image size and build time
  9. Docker Filesystems: Understanding the btrfs Backend (Xml and More)