Tuesday, 20 February 2018

MPLS causes some weird effects - aka Why is traceroute so much slower than ping for some hops?

Recently, my attention was drawn to something Odd about our traceroutes - namely, that traceroute and ping to an intermediate host on a route could have wildly different values.

This really bothered me, once I was forced to think about it.

I had previously assumed (wrongly) that the unexpectedly high second hop RTTs  (and similar subsequent) values across our service provider were due to low priority in processing ICMP/tracereoute packets (many routers treat these things as low priority, for various good reasons).
That was a good enough "explanation" that I'd not really thought beyond that (or, it hadn't bothered me enough to get properly intrigued).
And I hadn't done pings to those intermediate hosts, and compared them side-by-side.
Shame on me.

And sure, ping and traceroute by default use different protocols (until you do traceroute -I).
But that's not it either.

Maybe traceroute sends so many more packets at a time than a ping that you hit a rate limit (1 per 500ms is a rate limit on some routers)?  ping is ~1 per second; traceroute fires out loads in groups of 3 spaced per hop (well, TTL increment) quite closely together.
That's not it either.

Maybe a firewall was breaking things?
But no, that makes no sense; both in this case are ICMP Echo, and it's unlikely they're going to treat ICMP Echo to destination A differently to Destination B on the Internet.

I'm familiar with a bunch of other common pitfalls with interpreting traceroutes, but this wasn't one of those.

As someone who really likes networking, this should have prompted investigation long ago, but it's not bothered me enough to go work it out (aka "I had more pressing concerns").

Until someone said "Explain this" and presented a side-by-side ping and traceroute with Odd Results...

Then, of course, you start THINKING about the problem, and, if you're not familiar with the underlying configuration and particularly some potential configurations of service provider networks outside your own control will probably cause you to pull your hair out.

So why...?

Friday, 16 February 2018

Chromebooks? Yes Please.

We've started seeing more and more Chromebooks.

To those in education overseas, they're not exactly news, but they have recently become (slightly) less unusual in South Africa, and are (intermittently) available from local suppliers. With the advent of Android-compatible models, we can now use them across all of our "core" software.

So far, I've been very pleased with them from a sysadmin point of view.
Acer R11 C738T

Read on for more experiences....

Tuesday, 6 February 2018

The joys of bash - some light scripting for n00bs

There's a lot to be said for scripting in a sysadmin's life - indeed, if you don't do any scripting, are you even a sysadmin...?

I've slowly been learning bits of bash and various related Unix utilities that are useful for processing text files - like the copious log files FreeRADIUS spits out with all sorts of useful information. I like "just in time" learning - it's often the only learning I have time for...
A screenshot of a bash shell script.
In particular, I wanted to know whether certain users are abusing the system, by connecting far too many devices, and to have a count of unique devices running on our LAN for a measure of the popularity of our wireless system (and perhaps for some capacity planning).  We limit device numbers for various reasons, and devices must be registered (on paper) prior to connection - there are only n spaces on the form, so n+1 or more == breaking the rules. Sure, electronic NAC is the obvious next step, and re-investigating Packetfence remains on my list of things to do...

Looking through currently connected clients where more than n clients are simultaneously connected is a fool's errand, even if your wireless system has such a view (UniFi has one) - and it won't catch people who break the rules over the course of a day, or when you're not actively looking, or across two different manufacturers of Access Points. A 24 hour long RADIUS logfile record on the other hand... Well, that has a lot of potential.

Of course, processing of this kind is quite easily accomplished in a simple shell script...

Thursday, 1 February 2018

Poorly Documented Feature: Canned Responses in Delegated Accounts (Gmail)

It's no secret. We love Gmail. However, sometimes, there are features that don't get used much that are absolute "killer" features - but it turns out they're not always well documented.

A common scenario is creating generic "role based" accounts that receive large volumes of mail, and to then delegate access to this mailbox to several individuals to deal with the responses.

Of course, a lot of incoming emails means a lot of outgoing responses (often the exact same thing hundreds of times), and there's a really handy feature, Canned Responses, in Labs that makes this a pleasure.
Labs Canned Responses.
Thank you Googler Chad P,
bulk email responders LOVE you.
However, if you switch across to a delegated account, shock, horror!
No Labs!
Does that mean no canned responses in delegated accounts? No it does not (phew)...

Tuesday, 21 November 2017

Weak defaults in IIS 8 cryptography (TLS/HTTPS/SSL)

So, this isn't exactly a breaking news headline. IIS isn't the most secure web server in the universe. However, it came as a shock to get a "C" grade on testing our school's management information system portal on Qualys' rather awesome SSL Labs tester.

Even if you're not governed by things like GDPR or POPI, it should be a point of ethical professional practice to ensure there isn't a hole large enough to drive a bus through in your security infrastructure.

Fortunately, there's a really easy way of fixing this.

Monday, 23 October 2017

RTFM: Speeding up your (Fortigate) firewall performance

I was witness to the installation of the (yes, single) Fortigate 300C firewall the school uses, however, it was not my own configuration/installation/design, although I've maintained it for several years now.

We've been having intermittent issues with close to 100% CPU usage and a sort of live lock up where the Fortigate responds, but packets do not flow (some of the scanning engines (ipsengine, or ipsmonitor) monopolise CPU time and need to be restarted several times [three, that's the magic number - diagnose test application ipsmonitor 99 - and wait several minutes between attempts] - or the unit rebooted, with resulting network chaos). And packets, like the spice, must flow.


The "death knell" for the 300C was a member of senior management unilaterally decreeing (without asking IT) that a whole year of pupils could have twice as many devices - a year ahead of schedule, and before the planned replacement to deal with the load... So I've been looking for ways to eke out a little more performance until we can afford/acquire a replacement for it.

It turns out that one of the design decisions that was made was not ideal - it completely disables the use of the onboard dedicated traffic ASICs...

Unfortunately, schools need quite paranoid and intense filters, and this comes at a cost (in terms of power, and price!).

Friday, 15 September 2017

Linux Game Server Manager

One of the teachers at school has declared that "eSports" will now be a Thing at school; his dream is that we thrash our arch-nemesis (more resourced  [top ten most expensive schools in the country] and bigger [more pupils]) school across the valley during the annual inter-school sports day. Indeed, he even talked to the Headmaster about it, who was initially confused, but eventually gave it his (tentative!) blessing.

This then means we need to facilitate such things as "gaming"...