Tuesday 22 March 2016

FreeRADIUS installation and configuration

I eventually abandoned shelved getting a working PacketFence installation (the learning curve and my time availability were not friends); I'll probably go back to setting that up (in the very least so there is a working config example), but I needed a production ready system, fast.

So, now that I've "simplified" to a working "just RADIUS" environment, I should be able to "complicate" it with PacketFence later on (and probably will do - electronic device registration deeply appeals to me on a "proper process" level, and helps with nonsense like RICA, although properly configured RADIUS logging might just obviate that need).

The stuff I learned pounding my head against PacketFence helped, but it wasn't the whole story...


One of the "joys" of managed enterprise wireless systems is they're (sometimes eye-wateringly) expensive. Whilst I'm wary of "bad mouthing" products in print, my experiences to date suggest if there is one enterprise wireless product rage you probably don't want to consider, it's HP's MSM range. I visited another school a week or so ago to evaluate a school MIS they had deployed, and they had HP access points littered about the place which they were replacing with Samsung units. They were not complementary about them either! HP's lifetime warranty replacement service is about the only good feature that isn't widely available in others - often for considerably less money.

Another way of solving the problem (too much demand, not enough resources)

Ubiquiti make a range of access points with what I like to think of as an "enterprise-lite" feature set - but they're getting very close to full enterprise level software features. Their pricing is really rather competitive. Once they get VLAN assignment by RADIUS response right (currently in beta), they'll be adequate for most sites; currently you can do something like that with up to 4 VLANS by simply associating 4 VLANs with 4 SSIDs. A little more Radio Resource Management sprinkled in would really make it rather good, and some features that have to be adjusted per AP could do with global controls. Another "in beta" feature is assigning VLANs to the wired ethernet ports on the In-wall access points, which is very useful. But it will get there, and probably in the very near future.

As a boarding school, you can imagine the demand for wireless here is practically insatiable. The existing HP MSM solution was costing us around 3-4 times as much as broadly equivalent Ubiquiti units per AP (and that's comparing 802.11n [MSM 430/460] with 802.11ac [UAP-AC-PRO]; the 802.11ac HP units are really expensive; we only have one on campus [MSM 560]), meaning our ability to expand the network footprint was hamstrung by finances. (Not to mention the recurring annual maintenance costs). We therefore took the plunge and bought one-for-one replacements of the existing MSM HP 430/460 (specifically UniFi AP AC Pro) units in boarding (the HP units will in turn be "upcycled" to properly cover academic areas previously ignored [against my better judgement] due to the loud wailing of boarding) - and a certain about of "magical thinking" by the school about just how far wireless (in a device dense, production BYOD environment) would actually spread. (Protip: The fastest way to thumb-suck how many access points you need is one per room if room occupancy is less than 30; if it's higher, add another AP until you have a maximum of 30 users per AP per room. Yeah, that's a lot of APs, but you need to talk to management about that and work with them to find a solution that fulfills the mandate of the wireless roll-out and the budget; you may have to phase it in over several years - make sure "stakeholder engagement" to manage expectations is adequately built into the plan). If you can, bring your users to wifi, rather than bringing wifi to your users (particularly in relatively low density spaces - and especially in boarding - make sure "common rooms" are one of the first areas you cover, or people will litter the hallways and staircases where wireless is strong!). Initial results from my pilot sites suggests the UAP-AC-PROs perform very well in the "wild".

I still think a full enterprise wireless feature-set is worth the money (at least it is at the moment) - particularly in academic spaces, but if "money's too tight to mention", give Ubiquiti's Unifi a serious look. I'm reasonably convinced that their system will be good enough within 6 months, and excellent within 2 years. (Their Edgeswitches are also offer a compelling feature/price trade-off). The in-wall access points are very attractive for spaces like staff housing and remote boarding rooms with low occupancy rates (like "student assistants" - generally University students who help out with keeping order in the house and sports coaching and the like in return for free board and lodging and a small stipend; they often think internet access is an inalienable part of the deal!) at around 1/3rd of the cost of a UAP-AC-PRO. If you were to push me on the product I'd most be interested in evaluating as a replacement for the HP, I'd probably say "Ruckus"; I would like to so so within the next 2-3 years, because 802.11n just isn't going to work in the long haul (incidentally, they have a webinar specifically for K-12 this week [8:00 PDT (17:00 SAST) 24 March 2016]).

Within the next fortnight, we'll have over 20 of these things in production. The first five UAP-AC-PROs and one in wall in the pilot deployment are doing sterling work.

I strongly suspect companies like Ubiquiti and Mikrotik are going to "disrupt" the marketplace quite effectively (to our benefit). Hopefully, this will spur traditional vendors of enterprise gear to work towards their competitive advantage being something other than proprietary protocols and FUD. I suspect it will trend towards "support", which for the sysadmin has varying levels of value (mainly depending on their own time resources). Certainly, HP's use of EMEA support technicians that speak only vaguely comprehensible English doesn't really make me feel the "support" is all that good - although they do *eventually* help you through most problems - I find email is far preferable to telephonic support due to accent mis-match. The user support community on platforms like UBNT and Mikrotik is far better that what HP manages (perhaps in part because they are more affordable, less skilled users make up a larger proportion of the user base, hence there is more discussion and documentation). Perhaps, shock horror, it may even lead to lower prices. Other areas for traditional enterprise vendors to aim are "better" antennas and noise immunity; a wider range of AP models (for different circumstances); rock solid fail-over if controllers are used. I'm not a real fan of  wirelessly "meshing" APs, but that's something that might be useful on occasion (Ubiquitis can do it).

Of course, deploying this system then meant we needed to have a RADIUS server to authenticate our clients, as the HP MSM controller previously did this for the "old" access points (including handling all the AD binding). Microsoft's NPS is horrible, so let's go FreeRADIUS.

So, down to business. 

Due to recent vulnerabilities in OpenSSL (like heartbleed) FreeRADIUS refuses to start without either forcing it to start with an "insecure" version (by editing the config file) or by upgrading OpenSSL to a non-vulnerable version. Many Linux distributions (Ubuntu among them) have "patched" the OpenSSL version they "ship", but kept the version number the same; this is what FreeRADIUS checks on startup, and then promptly goes COMPUTER SAYS NO and exits (unless you edit the config file to allow it). As the pre-packaged (apt-get) version of FreeRADIUS is a little old, I decided to start with a "clean slate", so I downloaded the compile-able source files for both FreeRADIUS and OpenSSL and compiled them both. These processes I'll leave as a useful learning exercise (hint: read the output of the configuration scripts to ensure you have all the requisite dependencies - installing the "<package name>-dev" version of whatever package is being complained about goes a long way to resolving most). You will likely have to move some files around, or create some symbolic links to get things into the right place for your system. In part because I charged ahead without documenting this process, I'm not going to detail it for you, but it will be well within the (possibly Google-enhanced) skill set of the intended audience for these posts! Note that doing this means you have to now manually maintain those packages and can't rely on apt-get upgrade to help you out. 

FreeRADIUS is a glorious and frustrating puzzle. For such a commonly used piece of software, the how-to guides have "and then a miracle occurs" steps left, right and centre. There don't seem to be any canonical solutions to basic problems that don't seem leave you hanging somewhere in the process. Hopefully this will save people days and hours of googling and trying things. Much of this information was pieced together by reading mailing list postings (Alan de Kok, a major contributor to FreeRADIUS and its support, would make BOFH look like a very friendly, helpful individual), and smatterings of various postings around the 'net. I like to charitably think of his approach as "go and Google the hell out of this topic so you learn something useful, instead of having the solution spoon-fed to you. Also, if you don't understand it, it's not going to be secure". This approach is solid, but it can be vexing if you want a solution RealSoonNow, or are confused about what the hell is going on. We've gotten quite used to open source software that "just works" or that has great documentation and support communities; it's quite an experience to come up against one that isn't quite like that.

You're going to get used to two things - running FreeRADIUS in debug mode, and wondering where the hell your time went. Protip: Shells are cheap and "free". Open lots of them, and watch what happens at all stages of the process using tail -f and/or debug modes for anything relevant.

Config "Walk-through" 

I'm going to assume you've already successfully compiled and installed both OpenSSL and FreeRADIUS with the latest versions (in this example, mine are 1.0.2g and 3.0.11 respectively). As usual, I'm assuming a server install of Ubuntu. I'm also going to assume you want to authenticate and authorise your users against Active Directory. I'm further going to assume you want to get PEAP/MSCHAPv2 and TTLS/MSCHAPv2 working; anything beyond that is an exercise for the reader; these provide reasonably secure enterprise-grade connections that work reasonably well in a heterogeneous BYOD environment without mucking about with things like client-side certificates. Finally, I'm going to assume that you want to get users assigned to specific SSIDs based on membership of a particular group (and ensure they only connect to the right SSID - i.e. students to your student SSID and staff to your staff SSID), and that your AD server(s) is(are) Windows Server 2012r2 with a GUI.

Please follow Alan de Kok's guides to getting the basic authentication against your domain working: First, get PAP working http://deployingradius.com/documents/configuration/pap.html; then add the extra AD bits: http://deployingradius.com/documents/configuration/active_directory.html. You will also need to learn about EAP http://deployingradius.com/documents/configuration/eap.html.

You will likely have to symlink things from mod-available to mods-enabled and sites-available to sites-enabled if you've compiled from sources; starting radiusd in debug mode (radiusd -X) will generally tell you if you've forgotten anything important.

Follow every.single.step. in sequence. Skip nothing. And make sure it works as expected. If you don't get this right, you will be down the rabbit hole before you know it, wondering if you'll ever get this working.

Once you have those steps right, set up your wireless system to point its RADIUS requests at your FreeRADIUS server; make sure you know what ports it is running on and what its IP address is - and that the host based firewall (and any firewall appliance between it and your server) allow that traffic.

Now let's get messy and play with the RADIUS configurations to get it working as we desire.

Backups/comments are your friends

First, backup your "working" configs before every single change - and certainly before a major change. In particular, you should make a copy of the "fresh" ones that come with the installation before you change anything.

I usually track my changes by adding new lines and commenting out the old ones, with comments on those lines about why I made the changes and the date/time I made them - it's also important to also note what the old ones did. In common with many unix-y config files, # marks a comment, and you can tack them onto the end of lines if you want to (I find this hurts legibility, so I usually put them above the relevant line [following whatever the convention in that file is - occasionally, that might be below the line, but most projects seem to go for above the line]).
You can also use a file "versioning" system if you want to get fancy.
Simply copying working config files to another location/filename also works quite well, but you can rapidly end up with rather large numbers of .old or .backup files/folders...!

It is amazing how quickly you will forget why you made a change somewhere and what it's supposed to do (particularly if your days are anything like mine with the odd snatched sysadmin moment between helpdesk tasks...) - at least, if you don't document it somewhere - and it might as well be electronic and in the file that affects the process.

As FreeRADIUS does most of its documentation within the configuration files anyway, you might as well follow their lead!

File locations

If your install is the same as mine (if you compiled from source on Ubuntu, it ought to be):
  • FreeRADIUS files are in /usr/local/etc/raddb and its subdirectories (which is [by design] owned by root) - sudo -i to escalate to a root user so you can easily cd into that directory and edit things; <insert disclaimer about being root>. Those files contain priviledged information (not least of which may be useful AD credentials; passwords to PKI root certificates; RADIUS secrets). 
  • FreeRADIUS logs to /usr/local/var/log/radius/ (if yours doesn't, check radiusd.conf for where it is going). Remember, it doesn't log in debug mode (it spews things [often including passwords, so watch out] onto your screen instead)!
  • OpenSSL is in /usr/local/ssl/ You may want to symlink to it from other locations like /usr/bin/openssl. Find all the places you need to do this until your shell always returns the latest version of openssl when you issue openssl version. You may need to restart software packages that leverage openssl (I don't know what runs on your server, so here's some "homework" for you). 
If you can't find something, the locate command is quite useful (make sure to run updatedb before you search after major changes like software installs).

NB the sections below will not generally allow iterative testing - you'll have to make all or most of the changes before it will work again. USE COMMENTS/BACKUPS! 


/usr/local/etc/raddb/radiusd.conf

This is the "central" freeRADIUS configuration file, which forms the scaffolding around which everything else works. 

unless you want to customise logging, then about the only thing you need to do is add ldap under the instantiate { section:
which causes the LDAP module to load - which is rather useful for your user group lookups. 

Speaking of logging (log { ), my edits to that are currently:
  • stripped_names = yes
  • auth = yes
  • auth_badpass = no # because you don't want to log passwords in plaintext
  • auth_goodpass = no # because you don't want to log passwords in plaintext
Be aware you can often see passwords in debug output, so make sure you don't run that in production.

/usr/local/etc/raddb/clients.conf

This file defines the "clients" you expect to connect to the RADIUS server - "clients" are access points and/or wireless controllers (not end users!). You will need to either set up one client definition per access point IP address (painful, unless you are a scripting wizard or have very few APs) or you will need to use the rather convenient facility to designate subnets in CIDR notation. If your APs are in their own management subnet (and they ought to be) simply designate that. You ought to change the RADIUS secret from the default testing123 (but remember when you do this, and make sure your management controller and APs have matching secrets...!).  You can leave the localhost definition as the default secret (testing123) until you are ready to put this into production, at which point you probably ought to choose a different secret.

Client definitions are fairly straightforward:
client <convenient name - will show up in some logs> {
ipaddr = <IP Address>/<CIDR subnet - leave blank for /32>
proto = *
secret = <secret>
nas_type = other
}
For example, assuming you want to call your client APs myschoolwifi and they are in 10.10.10.0/24 and you want a RADIUS secret of testing123:
client myschoolwifi {
ipaddr = 10.10.10.0/24
proto = *
secret = testing123
nas_type = other
}

/usr/local/etc/raddb/proxy.conf

You probably won't have to edit anything in this file, unless you want to get fancy with realms, or if/when change the localhost RADIUS secret.

/usr/local/etc/raddb/mods-enabled/

You're going to want to symlink a lot of modules from mods-available. The ones I have are: 
always -> ../mods-available/always
attr_filter -> ../mods-available/attr_filter
chap -> ../mods-available/chap
detail -> ../mods-available/detail
digest -> ../mods-available/digest
eap -> ../mods-available/eap
exec -> ../mods-available/exec
expiration -> ../mods-available/expiration
expr -> ../mods-available/expr
files -> ../mods-available/files
krb5 -> ../mods-available/krb5
ldap -> ../mods-available/ldap
logintime -> ../mods-available/logintime
mschap -> ../mods-available/mschap
ntlm_auth -> ../mods-available/ntlm_auth
pap -> ../mods-available/pap
preprocess -> ../mods-available/preprocess
realm -> ../mods-available/realm
unix -> ../mods-available/unix
You don't necessarily need all of these, but you'll get it running without errors and with less editing of config files if you do all of them. The four in bold need to be edited to work properly - see the following sections to see the edits required.

You're going to need to edit a number of them. Because of the symlink, you can edit the mods-enabled version, and the "original" in mods-available will be edited - so don't assume the mods-available is a good "backup". You could, of course, copy the files from one to the other for a backup to remain in mods-available, but I prefer the symlink method.

mschap 

Edit the ntlm_auth line to:
  ntlm_auth = "/usr/bin/ntlm_auth --request-nt-key --username=%{mschap:User-Name:-None} --domain=%{%{mschap:NT-Domain}:-<YOURDOMAIN>} --challenge=%{mschap:Challenge:-00} --nt-response=%{mschap:NT-Response:-00}"
replacing <YOURDOMAIN> with your Windows domain - the old NT style domain name, not the newer @ style one. We're doing this because we're not going to expect users to know they need to fill in <DOMAIN>\<username> when they connect to wireless. This passes it for them.

NB there are no line breaks, that is a single line in the config file.

This depends heavily on SAMBA, so make sure that is resilient. In our /etc/samba/smb.conf file, we once pointed to the primary domain controller as the  password server.

password server = <primary domain controller DNS NAME>

As with LDAP below, I strongly suggest you make a number of DNS entries that can "round robin" for all your domain controllers with the same name (i.e. make an identical A record for each one). I used kerberos.<mydomain> for all three, and changed the password server to match - this makes your RADIUS more resilient as it removes a SPOF.  SAMBA, and RADIUS, can now do MSCHAP auth against all three of our DCs. You may also be able to figure out a way to use the kerberos SRV records in _tcp.<yourdomain>, which will scale easily and automagically keep track of changes - but that's an exercise for the reader :)
My relevant smb.conf line therefore now reads:

password server = KERBEROS.<MYDOMAIN.TLD>

eap

 Change:

  • default_eap_type = ttls
  • private_key_password = to whatever the private key password is (if you change from the default, and you ought to!). 
  • disable_tls_2 = yes (you should try with no but test all your clients across all major OS variants; in theory, the latest OpenSSL version should address the problems this introduced particularly with Windows clients)
  • Under ttls
    • default_eap_type = mschapv2
    • copy_request_to_tunnel = yes
    • use_tunneled_reply = yes
    • virtual_server = inner_tunnel
  • Under peap
    • default_eap_type = mschapv2
    • copy_request_to_tunnel = yes
    • use_tunneled_reply = yes
    • virtual_server = yes
Other than that, the eap configuration should be pretty much OK.

ldap

You're going to be using LDAP to check user groups, so you'll need to configure this.

Ideally, create a user with the correct LDAP lookup permissions rather than recycling any other credentials. (If you followed the previous packetfence examples, that user should work well). 

You will need to edit:
  • server = '<your domain controller>' #ideally, a round robin DNS entry like ldap.yourdomain across your DCs, but an IP address will work
  • port = 3286
  • identity = '<username in LDAP form>' # i.e. 'cn=radiususer,cn=softwareusers,dc=mydomain,dc=mytld'
  • password = password for that user
  • base_dn = '<ldap basedn>' # e.g. 'dc=mydomain,dc=mytld"  
  • under user
    • filter = "(SAMAccountName=%{%{Stripped-User-Name}:-%{User-Name}})" # this took me AGES to get right and is really the key to the whole auth by group thing. Very poorly documented. 
  • under group
    • base_dn = <your ldap basedn> # leave as the default "${..base_dn}" if you wish
    • scope = sub
    • name_attribute = cn
    • membership_filter = "(&(objectClass=group)(member=%{Ldap-UserDn}))"
    •  membership_attribute = 'memberOf'

ntlm_auth

edit the program line to: 
  • program = "/usr/bin/ntlm_auth --request-nt-key --domain=<YOURDOMAIN> --username=%{mschap:User-Name} --password=%{User-Password}"
replacing <YOURDOMAIN> with your old NT style domain (without the angle brackets).

/usr/local/etc/raddb/sites-enabled/

You should symlink default and inner-tunnel from sites-available. 

default

Under authorize {
make sure that

  • filter_username
  • preprocess
  • mschap
  • suffix
  • eap
  • files
  • ldap #optional
  • expiration
  • logintime
  • pap
are all uncommented; once you have a working RADIUS configuration, you can probably comment the ones you're not using out (notably ldap, files, pap), but when you're testing the RADIUS server, several of them will be needed.

Under authenticate {
make sure that

  • pap
  • cgao
  • mschap
  • digest 
  • eap
are uncommented; add ntlm_auth above chap:

In the post-auth { section, you'll want to add the if/elsif/else statements that control your authorisation of users to specific SSIDs. Here is my setup: 

Called-Station-ID has a regex in it that searches for the relevant wifi name; watch out, it will match the first available full match (so if you have a wifi name KC_Staff and another KC_Staff-boarding it will match both of those) - be as specific as you need to. The bit between the two /<this bit>/ is what it will match the LDAP-Group is the (case sensitive) group name you want to check - so if your staff users are in the Staff_WiFi group in AD, then that's what you put in there. As previously, I strongly recommend adding one or more specific WiFi group(s) so you can revoke WiFi access if needed to without revoking other priviledges (i.e. keying off "Staff" or "Pupil" membership groups is not ideal). Work through all the SSID and user groups you need to with elsifs; at the end, make sure you reject anything that doesn't match! 

Here's a code snippet you can recycle: 


post-auth {
        if (Called-Station-Id =~ /<YOURSSID>/ && LDAP-Group == "<YOURGROUP>" ) {
                noop
        }
        elsif (Called-Station-Id =~ /<YOUROTHERSSID>/ && LDAP-Group == "<YOUROTHERGROUP>" ) {
                noop
        }
        else {
                update reply {
                Reply-Message := "Login failed. Your login credentials are not correct for the network you attempted to connect to. (Check the SSID you're using and your username and password)."
                }
                reject
        }

Simply replicate the elsif line as many times as you need to - above the final else statement. If multiple groups can auth against a particular SSID, obviously repeat the line with the SSID regex remaining the same with a different LDAP group name each time. Make sure you don't inadvertently escalate privileges by allowing people to access the wrong SSID.  

inner-tunnel

This file is very similar to the default one, above. Edit all the sections mentioned above to match.

Virtual servers like this give you the power to route different types of users in different directions; we're just using it to handle the tunneled connections the same way as un-tunneled connections would be handled.

Testing... 

And now, start radiusd -X and see what happens if you try to authenticate a client. I'll assume you've pointed your access point/controller at your new RADIUS server! It ought to work if your environment matches mine and you've carefully completed all the steps. If not, carefully read through the output (yes, ALL of it, line-by-line) to work out where it breaks. Step through all the expected SSID/client group configurations - test with both "should work" and "should not work" users and SSID combinations to make sure nobody slips through any cracks!

The icing on the cake

Reboot your server. Make sure it still works. If it doesn't, work out what doesn't automatically restart on reboot (hints: check all the AD binds and Samba services work automatically; check radiusd starts automatically; check your firewall rules are correct and come up automatically) and do what you need to do to make sure it does. You do not want to be caught out down the line with a server that doesn't do what it needs to do months from now when a random powercut or kernel update forces a reboot and you wonder why RADIUS magically fails for no apparent reason. You do not want a production machine whose purpose is to do RADIUS not to do that pretty much as soon as you turn it on without needing human intervention.

If you work with virtual machines, clone this working machine and edit it (particularly hostname and IP address(es)) to make a second RADIUS server. Update your wireless controller software to make use of it. Make sure it lives on different hardware to your primary RADIUS server. If not, pencil in creating a second RADIUS server some time in the (very) near future. You do have two domain controllers, don't you....?

Still to do

Pending production support of VLAN assignation by RADIUS response in Ubiquiti, I've not yet integrated VLAN mapping. This will likely use the LDAP group to return specific VLANs. Once this is available in software, I'll test this out, get it working and report back!

Better logging of requests is also needed.

Conclusion

Hopefully, I've saved you days of trouble-shooting and config crafting, and this should get you up and running in a hour or two (at most) - rather than several weeks!

Obviously, you'll not have benefited from building it from the ground up and will be missing some of the "intuitive" feel of how the different components bolt together, but sometimes, what you want is a thing that works NOW!

Now I'm going to see if another member of my team can follow these instructions and build a working server...!

2 comments:

  1. Hi, Can you please suggest me why I am getting this error while restarting radius server by #service radiusd restart command.

    err: [root@freeradius ~]# service radiusd restart
    Stopping radiusd: [FAILED]
    Starting radiusd: /bin/bash: line 1: 3799 Segmentation fault (core dumped) /usr/sbin/radiusd -d /etc/raddb
    [FAILED]

    ReplyDelete
  2. Hi Ady, I'm afraid not - perhaps try one of the FreeRadius mailing lists to see if anyone else has had this problem before. SegFaults usually indicate extreme unhappiness however, so something may be very wrong with your installation - perhaps try recompiling or re-downloading a binary?

    ReplyDelete