Deep dive into WCCP load balancing

Quick Overview

WCCP (Web Cache Communication Protocol) is a content routing protocol developed by Cisco that allows you to redirect traffic in real time.  A typical use case for WCCP would be if you have a proxy or load balancer that you want to redirect traffic to, all transparent to the end user(no configuration needed on browser).  Each WCCP setup has at least one WCCP client and one WCCP server where the proxy would be the client, and the Cisco switch/router would be the server. An access list on the switch/router defines which traffic should be redirected via WCCP, and which traffic should flow through as normal.  WCCP allows for easy scaling, fault tolerance, and load balancing.  The load balancing piece of WCCP gets a little involved so let’s take a look at how that works.

Masks and Buckets

In the case when you have more than one WCCP client, maybe you have two web proxies, WCCP provides built-in load balancing.  The way that WCCP determines which traffic is sent to each proxy is through the use of a Mask value that it applies to the IP addresses as they pass through the redirect on the switch or router.  Whether the mask gets applied to the source or destination IP is controlled by a setting on the WCCP client.  Where does the mask get set? It’s set on the WCCP client, for this example we’ll use a Websense proxy, which sets the default value to the hex value 0x1741.  The logical product of the mask and IP address, produces a value which will be called the bucket.  The buckets then get evenly distributed between WCCP clients, and your traffic is distributed accordingly.  Confused yet? Let’s break it down piece by piece.

Math

First let’s convert everything into binary. For this example, let’s use the source IP 192.168.100.5 and the default Websense mask of 0x1741.

Converting the IP to binary:      11000000 10101000 01100100 00000101

Converting the mask to binary: 00000000 00000000 00010111 01000001

Now let’s see how many possible buckets we can have with this mask. This is controlled purely by the number of ‘1’s in the mask.  If you take 2^number of 1 bits in mask, you will get the number of buckets available, in this case the mask has 6 bits set, so 2^6 = 64 buckets.  There are 64 possible combinations you could come up with when you logically AND any IP address with this specific mask

Let’s perform a sample logical AND.

logicaland

Logical AND means that any any place there is a ‘1’ in both columns of the source IP and mask, it will generate a ‘1’ in the result.  Any other combination(0 and 1, 0 and 0, 1 and 0, all equal 0).

logicalandtable

So the final result(Bucket) is 00000000 00000000 00000100 00000001, or 0x401 in hex. If you took different source IP addresses and went through the math to logically AND them together you would end up with different resulting buckets, but only 64 buckets total(2^6).  Here is the output from a Cisco switch that was connected via WCCP to two proxies(10.20.30.40 and 10.20.30.50) using the default mask 0x1741. You can see that it split up the 64 buckets into two groups (buckets 0 – 31 assigned to WCCP client ID 10.20.30.50) and (buckets 32 – 64 assigned to WCCP client ID 10.20.30.40). I added a couple comments in bold and highlighted the row where the resulting value was 0x401, from our example.

switch#show ip wccp 90 detail
WCCP Client information:
WCCP Client ID: 10.20.30.50
Protocol Version: 2.0
State: Usable
Redirection: L2
Packet Return: L2
Packets Redirected: 99
Connect Time: 1d19h
Assignment: MASK

Mask SrcAddr DstAddr SrcPort DstPort
—- ——- ——- ——- ——-
0000: 0x00001741 0x00000000 0x0000 0x0000 <——— This is our mask 0x1741, under the ‘SrcAddr’ column

Value SrcAddr DstAddr SrcPort DstPort CE-IP
—– ——- ——- ——- ——- —–
0000: 0x00000000 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0001: 0x00000001 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0002: 0x00000040 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0003: 0x00000041 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0004: 0x00000100 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0005: 0x00000101 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0006: 0x00000140 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0007: 0x00000141 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0008: 0x00000200 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0009: 0x00000201 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0010: 0x00000240 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0011: 0x00000241 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0012: 0x00000300 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0013: 0x00000301 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0014: 0x00000340 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0015: 0x00000341 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0016: 0x00000400 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0017: 0x00000401 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0018: 0x00000440 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0019: 0x00000441 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0020: 0x00000500 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0021: 0x00000501 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0022: 0x00000540 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0023: 0x00000541 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0024: 0x00000600 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0025: 0x00000601 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0026: 0x00000640 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0027: 0x00000641 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0028: 0x00000700 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0029: 0x00000701 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0030: 0x00000740 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)
0031: 0x00000741 0x00000000 0x0000 0x0000 0x0A141E32 (10.20.30.50)

WCCP Client ID: 10.20.30.40
Protocol Version: 2.0
State: Usable
Redirection: L2
Packet Return: L2
Packets Redirected: 8
Connect Time: 1d19h
Assignment: MASK

Mask SrcAddr DstAddr SrcPort DstPort
—- ——- ——- ——- ——-
0000: 0x00001741 0x00000000 0x0000 0x0000

Value SrcAddr DstAddr SrcPort DstPort CE-IP
—– ——- ——- ——- ——- —–
0032: 0x00001000 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0033: 0x00001001 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0034: 0x00001040 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0035: 0x00001041 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0036: 0x00001100 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0037: 0x00001101 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0038: 0x00001140 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0039: 0x00001141 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0040: 0x00001200 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0041: 0x00001201 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0042: 0x00001240 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0043: 0x00001241 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0044: 0x00001300 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0045: 0x00001301 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0046: 0x00001340 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0047: 0x00001341 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0048: 0x00001400 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0049: 0x00001401 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0050: 0x00001440 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0051: 0x00001441 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0052: 0x00001500 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0053: 0x00001501 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0054: 0x00001540 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0055: 0x00001541 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0056: 0x00001600 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0057: 0x00001601 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0058: 0x00001640 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0059: 0x00001641 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0060: 0x00001700 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0061: 0x00001701 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0062: 0x00001740 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)
0063: 0x00001741 0x00000000 0x0000 0x0000 0x0A141E28 (10.20.30.40)

Choosing the best mask

So we go through all the math, see the number of buckets, how traffic would be distributed evenly but how can we use the mask value to our advantage when deploying WCCP?  First, with the default mask it allows for 64 buckets to be distributed between only two proxies. We don’t really need all of those different buckets if we only have two WCCP clients(proxies).  If we remember from above that the number of buckets is equal to 2^number_of_bits_in_mask, then at a minimum we need only one ‘1’ bit somewhere in the mask to generate two buckets, one bucket going to proxy A and one bucket going to proxy B.  This has an added benefit on the switch by using up less of the TCAM resources.  See this link, table 3 for more info.  How you choose the best mask really depends on the type of traffic in your environment, how many proxies/WCCP clients you have, and how you want to load balance it.  Cisco recommends not using the default of 0x1741. If you have multiple sites, each one having a /16 address space, you might want to create a mask that results in each /16 getting balanced through a different proxy. If you have a single site with a number of /24 subnets you probably want to look at the third or fourth octet of the IP address so the hash is more effective(since the first two octets will always be the same a hash taking effect on those octets will be less effective at balancing traffic).  Here are a couple of examples:

  • A mask of 0x0, we end up with one bucket(2^0=1), which means there could only be one proxy, and no load balancing would take place.
  • A mask of 0x1 (00000000 00000000 00000000 00000001), we end up with two buckets (2^1=2), with even numbered last octet IP addresses going to one proxy and odd numbered last octet IP addresses going through the other proxy.
  • A mask of 0x100 (00000000 0000000 00000001 00000000), we end up with two buckets again, with even third octets going to one proxy and odd numbered third octets going to a different proxy

Cisco has a good writeup on their recommendations on the WCCP mask values for different environments available on this page. Here is an excerpt from Cisco:

  • We do not recommend using the WAAS default mask (0x1741). For data center deployments, the goal is to load balance the branch sites into the data center rather than clients or hosts. The right mask minimizes data center WAE peering and hence scales storage. For example, use 0x100 to 0x7F00 for retail data centers that have /24 branch networks. For large enterprises with a /16 per business, use 0x10000 to 0x7F0000 to load balance the businesses into the enterprise data center. In the branch office, the goal is to balance the clients that obtain their IP addresses via DHCP. DHCP generally issues client IP addresses incrementing from the lowest IP address in the subnet. To best balance DHCP assigned IP addresses with mask, use 0x1 to 0x7F to only consider the lowest order bits of the client IP address to achieve the best distribution.

Choosing a mask that works best with your environment allows you to have better control of how traffic will be distributed between proxies and makes it much more deterministic so if for example you choose 0x1 as your mask you know that any clients with even last octets are going through one proxy and all the odd last octets are going through another proxy.  During troubleshooting if you get reports that users are having issues possibly related to the proxy, by knowing what their IP ends in you can quickly correlate if all the odd numbered IPs are having an issue but even numbered IPs aren’t that Proxy A may need to be looked further.

Advertisements

One thought on “Deep dive into WCCP load balancing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s