Deep dive into WCCP load balancing

Quick Overview

WCCP (Web Cache Communication Protocol) is a content routing protocol developed by Cisco that allows you to redirect traffic in real time.  A typical use case for WCCP would be if you have a proxy or load balancer that you want to redirect traffic to, all transparent to the end user(no configuration needed on browser).  Each WCCP setup has at least one WCCP client and one WCCP server where the proxy would be the client, and the Cisco switch/router would be the server. An access list on the switch/router defines which traffic should be redirected via WCCP, and which traffic should flow through as normal.  WCCP allows for easy scaling, fault tolerance, and load balancing.  The load balancing piece of WCCP gets a little involved so let’s take a look at how that works.

Masks and Buckets

In the case when you have more than one WCCP client, maybe you have two web proxies, WCCP provides built-in load balancing.  The way that WCCP determines which traffic is sent to each proxy is through the use of a Mask value that it applies to the IP addresses as they pass through the redirect on the switch or router.  Whether the mask gets applied to the source or destination IP is controlled by a setting on the WCCP client.  Where does the mask get set? It’s set on the WCCP client, for this example we’ll use a Websense proxy, which sets the default value to the hex value 0x1741.  The logical product of the mask and IP address, produces a value which will be called the bucket.  The buckets then get evenly distributed between WCCP clients, and your traffic is distributed accordingly.  Confused yet? Let’s break it down piece by piece.


First let’s convert everything into binary. For this example, let’s use the source IP and the default Websense mask of 0x1741.

Converting the IP to binary:      11000000 10101000 01100100 00000101

Converting the mask to binary: 00000000 00000000 00010111 01000001

Now let’s see how many possible buckets we can have with this mask. This is controlled purely by the number of ‘1’s in the mask.  If you take 2^number of 1 bits in mask, you will get the number of buckets available, in this case the mask has 6 bits set, so 2^6 = 64 buckets.  There are 64 possible combinations you could come up with when you logically AND any IP address with this specific mask

Let’s perform a sample logical AND.


Logical AND means that any any place there is a ‘1’ in both columns of the source IP and mask, it will generate a ‘1’ in the result.  Any other combination(0 and 1, 0 and 0, 1 and 0, all equal 0).


So the final result(Bucket) is 00000000 00000000 00000100 00000001, or 0x401 in hex. If you took different source IP addresses and went through the math to logically AND them together you would end up with different resulting buckets, but only 64 buckets total(2^6).  Here is the output from a Cisco switch that was connected via WCCP to two proxies( and using the default mask 0x1741. You can see that it split up the 64 buckets into two groups (buckets 0 – 31 assigned to WCCP client ID and (buckets 32 – 64 assigned to WCCP client ID I added a couple comments in bold and highlighted the row where the resulting value was 0x401, from our example.

switch#show ip wccp 90 detail
WCCP Client information:
WCCP Client ID:
Protocol Version: 2.0
State: Usable
Redirection: L2
Packet Return: L2
Packets Redirected: 99
Connect Time: 1d19h
Assignment: MASK

Mask SrcAddr DstAddr SrcPort DstPort
—- ——- ——- ——- ——-
0000: 0x00001741 0x00000000 0x0000 0x0000 <——— This is our mask 0x1741, under the ‘SrcAddr’ column

Value SrcAddr DstAddr SrcPort DstPort CE-IP
—– ——- ——- ——- ——- —–
0000: 0x00000000 0x00000000 0x0000 0x0000 0x0A141E32 (
0001: 0x00000001 0x00000000 0x0000 0x0000 0x0A141E32 (
0002: 0x00000040 0x00000000 0x0000 0x0000 0x0A141E32 (
0003: 0x00000041 0x00000000 0x0000 0x0000 0x0A141E32 (
0004: 0x00000100 0x00000000 0x0000 0x0000 0x0A141E32 (
0005: 0x00000101 0x00000000 0x0000 0x0000 0x0A141E32 (
0006: 0x00000140 0x00000000 0x0000 0x0000 0x0A141E32 (
0007: 0x00000141 0x00000000 0x0000 0x0000 0x0A141E32 (
0008: 0x00000200 0x00000000 0x0000 0x0000 0x0A141E32 (
0009: 0x00000201 0x00000000 0x0000 0x0000 0x0A141E32 (
0010: 0x00000240 0x00000000 0x0000 0x0000 0x0A141E32 (
0011: 0x00000241 0x00000000 0x0000 0x0000 0x0A141E32 (
0012: 0x00000300 0x00000000 0x0000 0x0000 0x0A141E32 (
0013: 0x00000301 0x00000000 0x0000 0x0000 0x0A141E32 (
0014: 0x00000340 0x00000000 0x0000 0x0000 0x0A141E32 (
0015: 0x00000341 0x00000000 0x0000 0x0000 0x0A141E32 (
0016: 0x00000400 0x00000000 0x0000 0x0000 0x0A141E32 (
0017: 0x00000401 0x00000000 0x0000 0x0000 0x0A141E32 (
0018: 0x00000440 0x00000000 0x0000 0x0000 0x0A141E32 (
0019: 0x00000441 0x00000000 0x0000 0x0000 0x0A141E32 (
0020: 0x00000500 0x00000000 0x0000 0x0000 0x0A141E32 (
0021: 0x00000501 0x00000000 0x0000 0x0000 0x0A141E32 (
0022: 0x00000540 0x00000000 0x0000 0x0000 0x0A141E32 (
0023: 0x00000541 0x00000000 0x0000 0x0000 0x0A141E32 (
0024: 0x00000600 0x00000000 0x0000 0x0000 0x0A141E32 (
0025: 0x00000601 0x00000000 0x0000 0x0000 0x0A141E32 (
0026: 0x00000640 0x00000000 0x0000 0x0000 0x0A141E32 (
0027: 0x00000641 0x00000000 0x0000 0x0000 0x0A141E32 (
0028: 0x00000700 0x00000000 0x0000 0x0000 0x0A141E32 (
0029: 0x00000701 0x00000000 0x0000 0x0000 0x0A141E32 (
0030: 0x00000740 0x00000000 0x0000 0x0000 0x0A141E32 (
0031: 0x00000741 0x00000000 0x0000 0x0000 0x0A141E32 (

WCCP Client ID:
Protocol Version: 2.0
State: Usable
Redirection: L2
Packet Return: L2
Packets Redirected: 8
Connect Time: 1d19h
Assignment: MASK

Mask SrcAddr DstAddr SrcPort DstPort
—- ——- ——- ——- ——-
0000: 0x00001741 0x00000000 0x0000 0x0000

Value SrcAddr DstAddr SrcPort DstPort CE-IP
—– ——- ——- ——- ——- —–
0032: 0x00001000 0x00000000 0x0000 0x0000 0x0A141E28 (
0033: 0x00001001 0x00000000 0x0000 0x0000 0x0A141E28 (
0034: 0x00001040 0x00000000 0x0000 0x0000 0x0A141E28 (
0035: 0x00001041 0x00000000 0x0000 0x0000 0x0A141E28 (
0036: 0x00001100 0x00000000 0x0000 0x0000 0x0A141E28 (
0037: 0x00001101 0x00000000 0x0000 0x0000 0x0A141E28 (
0038: 0x00001140 0x00000000 0x0000 0x0000 0x0A141E28 (
0039: 0x00001141 0x00000000 0x0000 0x0000 0x0A141E28 (
0040: 0x00001200 0x00000000 0x0000 0x0000 0x0A141E28 (
0041: 0x00001201 0x00000000 0x0000 0x0000 0x0A141E28 (
0042: 0x00001240 0x00000000 0x0000 0x0000 0x0A141E28 (
0043: 0x00001241 0x00000000 0x0000 0x0000 0x0A141E28 (
0044: 0x00001300 0x00000000 0x0000 0x0000 0x0A141E28 (
0045: 0x00001301 0x00000000 0x0000 0x0000 0x0A141E28 (
0046: 0x00001340 0x00000000 0x0000 0x0000 0x0A141E28 (
0047: 0x00001341 0x00000000 0x0000 0x0000 0x0A141E28 (
0048: 0x00001400 0x00000000 0x0000 0x0000 0x0A141E28 (
0049: 0x00001401 0x00000000 0x0000 0x0000 0x0A141E28 (
0050: 0x00001440 0x00000000 0x0000 0x0000 0x0A141E28 (
0051: 0x00001441 0x00000000 0x0000 0x0000 0x0A141E28 (
0052: 0x00001500 0x00000000 0x0000 0x0000 0x0A141E28 (
0053: 0x00001501 0x00000000 0x0000 0x0000 0x0A141E28 (
0054: 0x00001540 0x00000000 0x0000 0x0000 0x0A141E28 (
0055: 0x00001541 0x00000000 0x0000 0x0000 0x0A141E28 (
0056: 0x00001600 0x00000000 0x0000 0x0000 0x0A141E28 (
0057: 0x00001601 0x00000000 0x0000 0x0000 0x0A141E28 (
0058: 0x00001640 0x00000000 0x0000 0x0000 0x0A141E28 (
0059: 0x00001641 0x00000000 0x0000 0x0000 0x0A141E28 (
0060: 0x00001700 0x00000000 0x0000 0x0000 0x0A141E28 (
0061: 0x00001701 0x00000000 0x0000 0x0000 0x0A141E28 (
0062: 0x00001740 0x00000000 0x0000 0x0000 0x0A141E28 (
0063: 0x00001741 0x00000000 0x0000 0x0000 0x0A141E28 (

Choosing the best mask

So we go through all the math, see the number of buckets, how traffic would be distributed evenly but how can we use the mask value to our advantage when deploying WCCP?  First, with the default mask it allows for 64 buckets to be distributed between only two proxies. We don’t really need all of those different buckets if we only have two WCCP clients(proxies).  If we remember from above that the number of buckets is equal to 2^number_of_bits_in_mask, then at a minimum we need only one ‘1’ bit somewhere in the mask to generate two buckets, one bucket going to proxy A and one bucket going to proxy B.  This has an added benefit on the switch by using up less of the TCAM resources.  See this link, table 3 for more info.  How you choose the best mask really depends on the type of traffic in your environment, how many proxies/WCCP clients you have, and how you want to load balance it.  Cisco recommends not using the default of 0x1741. If you have multiple sites, each one having a /16 address space, you might want to create a mask that results in each /16 getting balanced through a different proxy. If you have a single site with a number of /24 subnets you probably want to look at the third or fourth octet of the IP address so the hash is more effective(since the first two octets will always be the same a hash taking effect on those octets will be less effective at balancing traffic).  Here are a couple of examples:

  • A mask of 0x0, we end up with one bucket(2^0=1), which means there could only be one proxy, and no load balancing would take place.
  • A mask of 0x1 (00000000 00000000 00000000 00000001), we end up with two buckets (2^1=2), with even numbered last octet IP addresses going to one proxy and odd numbered last octet IP addresses going through the other proxy.
  • A mask of 0x100 (00000000 0000000 00000001 00000000), we end up with two buckets again, with even third octets going to one proxy and odd numbered third octets going to a different proxy

Cisco has a good writeup on their recommendations on the WCCP mask values for different environments available on this page. Here is an excerpt from Cisco:

  • We do not recommend using the WAAS default mask (0x1741). For data center deployments, the goal is to load balance the branch sites into the data center rather than clients or hosts. The right mask minimizes data center WAE peering and hence scales storage. For example, use 0x100 to 0x7F00 for retail data centers that have /24 branch networks. For large enterprises with a /16 per business, use 0x10000 to 0x7F0000 to load balance the businesses into the enterprise data center. In the branch office, the goal is to balance the clients that obtain their IP addresses via DHCP. DHCP generally issues client IP addresses incrementing from the lowest IP address in the subnet. To best balance DHCP assigned IP addresses with mask, use 0x1 to 0x7F to only consider the lowest order bits of the client IP address to achieve the best distribution.

Choosing a mask that works best with your environment allows you to have better control of how traffic will be distributed between proxies and makes it much more deterministic so if for example you choose 0x1 as your mask you know that any clients with even last octets are going through one proxy and all the odd last octets are going through another proxy.  During troubleshooting if you get reports that users are having issues possibly related to the proxy, by knowing what their IP ends in you can quickly correlate if all the odd numbered IPs are having an issue but even numbered IPs aren’t that Proxy A may need to be looked further.


X-Forwarded-For, proxies, and IPS

When deploying an IPS appliance I saw a challenge that might come up if you are installing the IPS appliance in addition to a web proxy. One of the by-products of using the default settings of the proxy is that all user traffic going through the proxy ends up being NATted to the IP address of the proxy prior to going to the firewall.  Normally this wouldn’t cause a problem but when you want to setup the IPS appliance to look at all traffic between the inside and firewall it presents an issue.  We lose visibility into what the original client IP address is, all traffic appears as it is coming from one single IP address of the web proxy making IPS logs less useful. In an ideal situation you would be able to place the IPS in a position where it would examine the actual source IP address but not all networks may be able to accommodate this.  One workaround is to utilize the x-forwarded-for header option on your proxy.

X-Forwarded-For Header

There is an industry standard(but not RFC) header available for HTTP called x-forwarded-for, that identifies the originating IP address of an HTTP request, regardless of if it goes through a proxy or load balancer. This header would typically be added by the proxy or load-balancer, but it’s worth noting that there are plugins out there that let a web browser insert this field(whether it is real or spoofed).

Current State

Our current state and traffic flow looks something like this:


The IP starts as the original ‘real’ client IP, and as it goes through the proxy(websense in this case) it gets changed to the IP of websense.  As it goes through the firewall it then gets changed to the IP of the firewall prior to hitting the Internet. Here’s a screenshot of a HTTP GET in wireshark, without any header:

no xforward

Adding in the header

To add the header in Websense you can find the option here in the content gateway GUI:


X-Forwarded State

After enabling the addition of x-forwarded-for headers in Websense this is what our traffic looks like:


Here’s a screenshot of an HTTP GET in Wireshark that includes the header, spoofed to



Once this header is added it allows some IPS appliances/software to inspect the x-forwarded-for header and report on the actual client IP address.  Snort currently supports this and there is more detail here. I believe that other IPS appliances such as Cisco’s Sourcefire also supports this option through enabling the HTTP inspect preprocessor and checking ‘Extract Original IP address’ option.  Will work on confirming this and updating the post sometime soon. If you want to look at this traffic in wireshark there is a display filter ‘http.x_forwarded_for’ that will let you filter on x-forwarded-for.


I’d like to point out that the x-forwarded-for header gets carried in the packet out into the Internet which may or may not concern some people as it releases more information about your internal IP addresses structure than you might have wanted.  I tried to see if there was an ASA feature to strip this header out but couldn’t find anything that looked like it fit besides this Cisco bug report/request for the feature. Also, as mentioned above you can spoof this header pretty easily, it is not authenticated or signed, and is presented in plain text.  Each deployment will be unique and you’ll have to weigh out the risks and whether this is a feature that is worth implementing for your specific environment.