Steve Goodman's Exchange Blog

Exchange 2010 HAProxy Virtual Load Balancer

An extra added cost to Exchange 2010 deployments is often a hardware load balancer, or even virtual load balancer appliances. These start at over £1000 for some of the cheaper ones and can cost tens of thousands, however there’s open source software out there that can do the same thing, just as well.

HAProxy is a widely used, reliable and stable Load Balancer for Linux and a few weeks ago I began looking at it as an alternative for Exchange 2010 load balancing and whilst having a look to see if anyone had done it before, I found a good article here. The one problem for many Exchange administrators is that they don’t have time to learn about Linux just to try it out, so I began to think that it would be great if someone made a virtual appliance (just like many of the HLB vendors sell) with an easy to use management interface aimed squarely at Exchange 2010 environments.

The Exchange 2010 HAProxy VLB Appliance is a free Layer 4 based virtual load balancer that runs in VMware or Hyper-V environments. It doesn’t require Linux knowledge to get up and running and is managed using a simple, easy to use, web-based management interface (screenshots below). For the initial release it’s not aimed at your production environment yet but as more people test it out and help refine it, future versions will be (and will be free, naturally).

Prerequisites

Before you start, you need to have an understanding of how the load balancer fits in your environment. Typically clients will connect to it for web services, like OWA, and also through Outlook via MAPI, using a Client Access Array as illustrated by this simplified diagram:

image

In addition to planning your environment, you’ll also need some other information for the load balancer setup:

  • IP address for management
  • IP address for the virtual load balancer interface, in the same subnet
  • DNS and NTP server addresses
  • Client Access Server IP addresses
  • Network Access from the virtual load balancer interface to the following TCP ports on the Client Access Servers:
    • 80 and 443 for HTTP/HTTPS
    • 135,139,6001-6004,60000 and 60001 for RPC Client Access

Note that if you’re testing this in a perimeter network, you only need ports 80 and 443 open for external access to Exchange servers.

Installation

The process for installation is fairly straightforward:

image

  • Download and import the appliance
  • Boot it up and set the management IP address via the console
  • Visit the web-based management interface and set a password, load balanced virtual IP address, set a few details like time zone and DNS server and finally add your first client access server.
  • Log in and add your other client access servers and follow instructions within the management interface as to how to set up static RPC TCP/IP ports on your client access servers.

The following videos show the installation and initial setup procedure both for VMware and Hyper-V environments:

VMware vSphere 

Hyper-V 2008 R2

Management

The management interface is intended to be fairly simple. After initial setup and login, you should (after the settings have taken effect) basic statistics for the underlying HAProxy load balancer, showing the number of sessions and state of the Client Access Servers:

image

The initial version uses Layer 4 load balancing, and uses the client source IP address for client affinity and doesn’t have intelligent application-level monitoring and SSL offload (yet). Therefore you just need to correctly configure static RPC ports (RPC Client Access, port 60000 and Address Book Service, port 60001) on the client access servers and add the IP addresses of each client access server to load balance:

image

On the remaining tabs, you have access to change the management and load balancer IP addressing, set the time zone and NTP servers, update the management interface password and check the logs for the management interface and underlying software that propagates changes made through the UI:

imageimage

imageimage

Should you wish, you can also log in at the console using the root password credentials you’ll set on initial startup. From here, it’s a standard minimal Ubuntu installation though any changes to settings that are usually managed through the UI will be overwritten the next time they are changed through the UI:

image

To actually send traffic to the load balancer, you need to consider the configuration of your environment and have setup your Client Access Array then the associated DNS names for web and RPC Client Access.

Download

Version 0.1 (initial release) is available for download here:

VMware vSphere Compatible OVF file, zipped (md5sum: b60388c5aa1012abe71f5864e79a6828)

Hyper-V compatible VHD, zipped (md5sum a9ae7f9b498f96a4d6d1bb58c4c542ee)

To check md5sum values, use Microsoft File Checksum Integrity Verifier

Notes

This is the first version, so just to repeat it’s only aimed at use in your lab environment.

It’s intended that with subsequent versions it will be production ready, as this is totally aimed at being an easy to use free alternative to paid-for hardware and virtual load balancers for Exchange 2010. It needs a few extra features but most importantly it needs your feedback and testing in the wild to ensure the management interface is good enough, and to get some ideas back from the field on what sort of load it can handle in it’s current form.

One thing I can’t guarantee is support for this – comments and reports of bugs are always appreciated, but the downside of free is it doesn’t come with a support contract. However the intention is to keep developing this and add other features to the so it can compete with the expensive equivalents.

Issues

Currently, the downloadable registry file for setup of static RPC ports does not work as expected. I would suggest following the link to the technet wiki on the setup pages for instructions on manually making the changes on each Client Access Server.

Comments (68) Trackbacks (1)
  1. Hi Stevie,

    great Job, but I miss the settings for stickyness, which are so important when load balancing Exchange “Services”.
    Bernd

  2. Hiya,

    It does have sticky persistence, based on source-IP, which can’t be changed in this version. There will be options to configure this and switch to other methods eg cookie persistence in subsequent versions.

    Steve

    • Ok. Normally the LBs with their IP address have to be the gateway on the Exchange Servers. Is that not the case with your implementation?

      As far as I understand the LB is pre configured for ports 80,443, 135 (MAPI CAS), 6000, 6001 an the non SSL POP3 and IMAP ports. How about port 25 to load balance HT Servers for incoming mail?

      Bernd

      • Hi Bernd,

        At the moment, this isn’t how it’s setup. It was originally my intention but I wanted to start simple first with a “dev” version to garner feedback and add features incrementally.

        I’ve not added LB for inbound SMTP as the current version will give the source IP to the Hub Transport as the LB which could have unintended consequences, like RBLs not working. I was instead intending to ship the appliance with an MTA built in that when switched on functions like a mini Edge Transport server.

        Steve

        • Regarding to LB for SMTP, what do you exactly mean with “source Ip”? Normally customers have mail hygiene appliances in their DMZ and the SMTP traffic goes from this appliances to the Exchange 2010 SMTP Receive Connectors. So, if the IP address of this appliances will get through the LB to the Receive Connectors everything would be fine ….
          Bernd

  3. For small and midsize companies I normally recommend Kemp Technologies HLBs and all installations I did so far had a virtual service for incoming SMTP and my customers love it. So I think this is an issue.
    Bernd

  4. Hi Steve,

    Great work, do let me know if you need help with this project. I thank you for pointing a link to my blog :) appreciate it…

    This is a very good functionality. I have written a PHP script for the management of the HA proxy for Exchange but sadly it isn’t made in a VM format yet … good thinking and great job.

    – Alok

    • Hi Alok (or is it Thunder Emperor ;-) )

      Thanks for the offer and thanks for the great article, After reading it, it was is the reason I wrote this management interface and put together the VM – to bring HAproxy to people who aren’t confident with Linux. Depending on how popular this is, I may well appreciate some assistance and as the codebase grows I’ll end up putting the PHP bits into Sourceforge or similar.

      I’m still learning about HAProxy, but next I want to look at the SSL offload/cookie based affinity and more intelligent service availability, along with passing through the source IP (using the LB as a gateway, like some other HLBs). I’m also thinking of using Heartbeat (little rusty with it, been 3 or 4 years since I last used it in anger) to implement the HA ability.

      Steve

  5. Oh Cool .. I have written another article with Cookie Based persistence for https and http (for OWA) and SSL offload using Pound. (Since HAProxy itself doesn’t offload the SSL) . I have written an small blog (http://3-4-5-6.blogspot.com/2011/08/haproxy-load-balancer-with-ssl.html )

    My HAProxy implementation is in HA Pair (using linux-ha) and it works great, we may incorp that into the design for a production level system.

    BTW, the deployment I did was a production system for about 2K users :)

    - Alok

    P.S_ Thunder emperor is just my screen name :) Stole it off Get Backers :)

  6. Hello Stevie

    This is well done job :)

    Can i ask very stupid question?
    Why cant my Outlook 2007 connect to the CAS server via Load Balancer? Am I missing somethin?

    Thank you

  7. Hi Stevie, do i something wrong? when i start the vhd in my Hyper-V Server, i can not login to the console. i tried it with root or setup and without pass, but unable to login. any idea?

  8. Thanks
    awesome
    for first version it looks amazing

  9. have a look at using tproxy with haproxy, you can then get around the issue of the connection appearing to be from the haproxy instead of the actual source.

    • Arne,

      If you do that, then the HA Proxy must be inline with the devices to function. If you want to keep the HA Proxy in a “One Arm” mode, then tproxy will not work :) , but yes … in inline deployments (which I dont suggest all that much) it is possible.

      For the restriction of SMTP for the relays and such you can apply the ACL on the HAproxy itself …

      • Thanks both – that is a good suggestion, I had been wondering what would be the best way to give an option for a inline deployment (the way Kemp, for example, recommend setup).

        Does sound like I need to provide the option to do both, though.

        Steve

        • I would say so… Most people wanting to put in a LB would prefer a one arm mode (A lot of people don’t want the LB to be the default gateway of the CAS servers) … Unless you have 2 different NIC’s, even so only one default gateway is permitted.

          In any event, lets work for the worst case if both the Load balancers crash, we should still be able to quickly change the DNS and repoint to one of the CAS servers, if we need this, we will need (One -Arm Mode)

          If we want to keep inline, then we will need the tproxy for the IP address to be transparent

  10. This looks pretty awesome! You mention that you don’t recommend using this in a production environment. Is it to buggy, unreliable, missing features or just that it’s not done yet? Are you still developing this project?

    • Hi Patric,

      It’s an initial reslease, so it’s best to say it’s not done yet. As it’s not “supported” in the traditional sense and makes use of other open source products it needs a community of testing before it can be gauged as reliable enough to host production workloads. The downside with something free like this is if it goes wrong there’s no-one to call on, so to mitigate that my intention is to make it as simple as possible (it breaks, throw a new one in) and have a concensus from people who are using it in test/dev that it’s reliable.

      Steve

  11. Do you have the app available without the OS, as we could not seem to import this into XenServer 5.6. It keeps dropping out to BusyBox due to no driver support for Xen. I have both Debian Squeeze, and Ubuntu Natty virtuals running that I could add this to.

  12. Nice work so far!

    I didn’t see an option to drain connections to a host for maintenance, etc reasons. I take it if as an alternative we delete a CAS from the server pool that it drops all active sessions immediately?

  13. Whats the root password for the appliance? Iv just installed it but no dice at the console?

  14. It’s ok figured it out…..

  15. I’m very impressed with your appliance. We are looking to get some exchange load balancing appliances and I came across this project. The one thing we require over what this provides is the ability to cluster two of them together so you don’t have a single point of failure. Is this something that may be added in the future?

    • Hiya

      At the moment it’s only aimed at lab/small environments, but yes it may be added in the future. As it’s only worked on in my (very limited!) spare time, I can’t guarantee anything though.

      Steve

  16. Steve noticed one very small bug. MAPI clients are prompted to authenticate during auto discover at the logon process. When clients go directly to the CAS this additional prompt does not occur. Entering the credentials does complete the autodiscover process but the appliance doesn’t seem to parse the credentials as normal during MAPI profile creation.
    If you need any help with testing more than happy to help. Spent many years working with Ubuntu and am very familiar with the distribution.

    • Shawn,

      I am assuming, you have set static ports, but may be only for the MAPI connection

      The only time this will happen is if you have not set the RPC ports statically for the Address book. Again, depending on the Version, there is a Reg Key that we need to change (or modify a text file). If that is done, you will not have the password prompt for the Auto discover as well.

      HAProxy does very well with selected number of ports, that’s why we will have to restrict the RPC ports (in this case 60K and 60K1)

      Please read

      http://social.technet.microsoft.com/wiki/contents/articles/configuring-static-rpc-ports-on-an-exchange-2010-client-access-server.aspx

      After you have made the reg file/text file based on the above link (and rebooting the CAS / HT)

      from your exchange CAS environment go to command prompt and type

      netstat -an | find “LISTENING”

      and ensure the 2 ports that you opened are shown as listening . After this the LB should work flawlessly

      If you still need assistance, just shout at http://3-4-5-6.blogspot.com/2011/03/ha-proxy-for-exchange-2010-deployment.html and I will try and assist ….

    • Hi Shawn,

      Ubuntu is just the linux distribution.. It could be any, and it was solely chosen because it had Hyper-V kernel modules built in. I’ve not seen this issue in my testing but I would follow Alok’s advice on checking everything is setup correctly on the CAS servers. The downloadable registry key (and I will update the page to state this) doesn’t work correctly so I would follow the link to the Technet wiki to apply the correct setting.

      Steve

  17. Hi Steve,
    Wanted to try this out in lab, but don’t have vsphere but the good old fashioned vmware server, would vmware server not work? I noticed I can’t use the file or convert using the converter, do I have to actually have to take a physical machine and install esxi to make this work?

    • Alok’s tip will work, but you should also be able to install the free VMware Convertor to import the OVF and then import it to a file share as a VMDK.

      If you have probs let me know and I will convert it to an earlier version suitable for use with VMware Server 2 and upload as a ZIP’ed VMDK.

      Steve

  18. Damian,

    Just to answer your question, you can install “ESXi” on to a VMWare Workstation, that should help :) Once you have booted, you can convert the format, or export that in another format

    -Alok

  19. Great job!

    Do you plan to launch a version compatible with IPV6?

  20. Hi,

    can you add the option to have more than one HAProxy (Load Balancer) IP Configuration?

    For example:
    server A, server B, server C, server D.

    LB-1 –> server A, server B

    LB-2 –> server C, server D.

    Duckie

  21. Duckie, why would you want that … In order to do what you are asking, just deploy two of the devices. If you re looking for redundancy then thats not the way to go …

  22. Duckie, I have written a blog post http://3-4-5-6.blogspot.com/2011/11/haproxy-for-exchange-2010part-2.html .. detailing how to make a redundant solution

  23. hi,
    redundant is great.

    what i’m try to do is, when you have two or more CAS servers & two or more HUB servers.
    so one HAProxy VIF or CAS and and second VIF for HUB.

    it is also good if you have Sharepoint servers and etc….

    Duckie

    • If you’re doing seperate services then a single HAproxy (or load balanced one as per Alok’s article) can still do this. It’s just this pre-packaged LB is not aimed at more advanced scenarios, it’s hoped for those you will explore more about what HAproxy has to offer. Basically whatever you can do with a Kemp or similar can be done with HAproxy, it just needs to be done via config files.

      Steve

  24. Thank Steve.

    I’ll go to production with HAproxy in the next few week, and let you know the results.

    Eran

    • Duckie … You can create more than one virtual server on HA Proxy… You should be able to add another sub-interface

      Interface settings

      auto eth0:1
      iface eth0:1 inet static
                 address 10.10.10.11
                 netmask 255.255.255.0
                 network 10.10.10.0
                 broadcast 10.10.10.255
                 gateway 10.10.10.1
      auto eth0:2
      iface eth0:2 inet static
                 address 10.10.10.15
                 netmask 255.255.255.0
                 network 10.10.10.0
                 broadcast 10.10.10.255
                 gateway 10.10.10.1

      In the haproxy.cfg (which should be at /etc/haproxy/haproxy,cfg)

      You can add it like this

      listen  Exchange2010-CAS 10.10.10.11:80
              bind 10.10.10.11:110,10.10.10.11:135
              bind 10.10.10.11:139,10.10.10.11:443
              bind 10.10.10.11:60000,10.10.10.11:60001
              bind 10.10.10.11:6001-6004
              bind 10.10.10.11:993-995
              mode    tcp
              option  persist
              balance roundrobin
              stick-table type ip size 10240k expire 30m
              stick on src
              server HC-CAS1 10.10.10.20 weight 1 check port 80 inter 5000 rise 2 fall 3
              server HC-CAS2 10.10.10.21 weight 1 check port 80 inter 5000 rise 2 fall 3
              option redispatch
              option abortonclose
              maxconn 40000
      listen  Exchange2010-HubT 10.10.10.15:25
              mode    tcp
              option  persist
              balance roundrobin
              stick-table type ip size 10240k expire 30m
              stick on src
              server HC-CAS1 10.10.10.23 weight 1 check port 80 inter 5000 rise 2 fall 3
              server HC-CAS2 10.10.10.24 weight 1 check port 80 inter 5000 rise 2 fall 3
              option redispatch
              option abortonclose
              maxconn 40000

      You can see that there are 2 Virtual servers load balancing 4 different servers

      VIP : 10.10.10.11 is load balancing 10.10.10.20 and 10.10.10.21
      VIP : 10.10.10.15 is load balancing 10.10.10.23 and 24

      This way you can put the instances.

      Let me know if you have questions

  25. Ina situation where i have 2 hub/cas role installed servers can i seperate the CAS and smtp traffic through dns or how can i use the haproxy ?

  26. What is default management port? I zipped through the setup process.. I know the ip using ifconfig but how do I find the listing port.. I tried the netstat command to see a “listening” port. I will try going into the config text file now.. Please comment or update article for management port.

    • Hi Tong,

      Off the top of my head, I think it’s port 8080

      Steve

      • Thx Steve,
        It was 8080. I assume once I set the listening ip, I just have to set my exchange up with the new cas array command and set correct ports? We appreciate your work here. I was about to try to setup ubuntu and haproxy without any prior linux experience. @_@ You saved us hours.

  27. Download Links are dead? Anyone have a mirror

  28. Hi Steve,
    thank you for sharing.

    is this software need to work in network card promisc mode?

  29. Hi Steve, great product, works wonderfully, thanks
    is there any way i can change the ports to support OCS
    I am presuming this article i found is using your product?
    http://blog.loadbalancer.org/load-balancing-microsoft-office-communications-server-ocs-with-haproxy/
    Thanks

    • Hi Martin,

      I’ve got your email about this – Hopefully in the next release I can make it a bit more customizable for these purposes.

      Steve

      • Got A simple Question, why use v load balancing vs WNLB?

        • Simple answer David,

          Windows NLB creates havoc in the network, due to the association of multicast MAC for unicast IP, and the network devices don’t like it at all, so even MS suggest that we use a 3rd Party LB instead of the native NLB.

          Since 3rd party loadbalancers are expensive, we created a free solution :)

          Hope this answers it!!!

          • Was about to answer with much of the same, but that sums it up pretty neatly!

            Only thing to add – here’s it from the horses mouse (MS) saying why they prefer LB for Exchange over NLB:

            http://www.stevieg.org/2010/11/exchange-team-no-longer-recommend-windows-nlb-for-client-access-server-load-balancing/

            Steve

          • Hmmm WNLB has been a cost effective & viable solution with reasonable limitations for my deployments over the last, While id doesnt provide Application aware Load Balancing, it has proven Effective in 3 Of my Environments… Process has kept WLNB around for sometime, the truth of the matter is; drain stopping a host after a few calls to our service desk normally results into an isolated issue with a particular Host. We may take a hit on the service desk calls, however we tend to identify problems immediately. I have yet to have the need to expand to a 8 node configuration..

            However after reading this post it sparked an interest..
            Thanks Alok & Steve….
            Still looking for some serious incentive to buy some new toys

  30. Steve two questions: 1) how well do you believe this appliance scales to? In other words how many concurrent users do you think it can have before it falls over.
    2) How long does it take to fail over MAPI clients? As when i take a server offline it doesn’t fail them over at all?


Leave a comment

(required)