WSANDERS.NET

Onward and Upward

The Juniper project is back on the rails. The DHCP problem seems to be under control, through a combination of reducing DHCP lease times (to hours instead of days), and disabling ICMP blocking in Windows Firewall. It wasn't my decision to block ICMP. Sometimes you can be too paranoid for your own good. For example, ICMP is used to negotiate MTU sizes between disparate networks. I can tell you a story about a major website that blocked all ICMP and wasn't able to communicate with anyone running smaller-than-normal MTUs. Which is a lot of people.

Happy New Year

Not much news from the field. We've stopped rolling out Junipers for a while because of massive FAIL in the JunOS DHCP server. Actually, it serves us right, trying to use the switches as DHCP servers. Serves up double-right, for this now-seems-silly idea of assigning one routable subnet to each switch port, a-la service provider. Our end users do have a propensity to hang strings of cheap-ass STP-incapable wall-wart-powered hubs off their drops and then "store" patch cables by plugging both ends into one of the hubs, but modern switches have broadcast controls that will effectively allow only the deserving to have their service hosed in this manner. (When I started working here, it was different. Campus-wide outages from looped ports occurred nearly every other day. But my predecessors had disabled spanning tree everywhere and never enabled broadcast controls for some reason I can't fathom.)

Anyway, back to DHCP. JunOS just could not handle it. It turned out to be a mix of our fault and theirs. First, in some buildings but not all, the PCs have Windows Firewall blocking ICMP. This always encourages DHCP fail since hosts (clients and server) can't ping each other to see if an address is claimed. Second, JunOS was making a horrible mess of the leases database. Third, we made it worse by specifying week-long lease times. Fourth, the JunOS dhcpd would just dump core form time to time.

Well, after setting lease times short, disabling Windows Firewall, and upgrading to the latest JunOS, we about ready to start more rollouts. Cross our fingers.

UPDATE: Sun X4540: Best. Box. Ever? Maybe.

The X4540 was brought to a standstill a few weeks ago by one dead SATA disk. The box didn't hang, but any ZFS IO did. Didn't lose any data, and it might be buggy hardware and drivers, but still, Sun support had no explanation. That should not happen.

Eventually, we're going to give Symantec Netbackup the finger and move to Amanda, which will enable us to upgrade to OpenSolaris. I posted on Slashdot about this and got a reply from "greg1104":

"People need to understand that SATA disks and chipsets are fundamentally weak at error reporting and recovery. There's only so much you can do about that at the driver or OS level if a problem drives the chipset crazy. You really need hardware optimized for that purpose, like a mature and battle-tested RAID controller."

I agree 100%. For now, ZFS is worth the risk. The box is a virtual tape library, so 100% uptime is not a requirement. I'm not going to start shorting the stock of midrange storage companies just yet.

OpenVPN Rules - Finally!

The WSANDERS ORGANIZATION has been struggling for years to find a quick and dirty VPN solution that was out-of-box enterprise ready, reliable, and didn't require the secret incantations of Security High Priests to get working. We've tried:

- Microsoft Windows Remote Access: Not bad. Microsoft fixed the cryptological problems with PPTP long ago,and every Windows PC came with a VPN client until recently (oops, M$.) There was a decent Linux client, but no easy to install Mac client. You could piggyback the server off any random Windows box behind your VPN, and use Active Directory or any LDAP for authentication. Rating: 3 out of 5.

- El Cheapo EBay Anything-but-Cisco Special: You could buy an old Juniper or something firewall and use a generic client. Usually you could get it to authenticate against Radius (but not usually LDAP or AD.) If your magic box did PPTP, cool, but usually you had to fiddle with handing out generic, fiddly IPSec clients to your users. Rating: 2 out of 5 stars.

- "F*** it, just open up the firewall": Run Remote Desktop Services on VNC on the desktops. But once one power user gets a firewall hole opened, everybody wants in. Do you really want to open your entire LAN to VNC? Rating: 2 out of 5 stars.

- Magic Boxes: Well, you can just grit your teeth and pay thousands for a magic box. Thy usually work, except when the vendor decides to break a protocol and force you to use their client, which may or may not install easily or even work, or, worse, force you to pay even more per-seat for licensing. Oh - you wanted encryption with your VPN - just write us another check, please! Rating: Varies widely with size and, mostly, ease of client installation.

- Poptop, OpenSWAN, SSH tunnels: Promising, but we could never get Poptop or OpenSWAN to work. SSH tunneling is OK, but requires expert knowledge and only forwards one or two protocols at the same time.

So finally we had a chance to give OpenVPN a try. What a surprise. Better yet, there is a commercial enterprise OpenVPN Technologies that offers a added-value product for $5 per seat that makes OpenVPN fiddle-free. Rating: 5 out of 5 stars for smallish installs.

This approach solves several big problems we've had with VPN deployments: Licensing and fiddly hard to install clients.

Licensing is straightforward: $5 per seat, period. For huge installs, it might be cheaper to buy a Magic Box. But for smaller deployments, for $5 you get: Super-easy installation on most Linux platforms, a web GUI, added value support for the parts that are different from "free" OpenVPN, defaults that make it work right out of the box (with LDAP, too), a Windows client that works, and as part of the GUI, a place where Windows clients can log in and download it, and non Windows users can download a config that simply plugs in to OpenVPN for use as a client.

No more client fiddling: OpenVPN Technologies supplies the Windows client, Linux users use the OpenVPN that comes with their distro, and Mac users can either use OpenVPN or Tunnelblick. All this guarantees that the client will be compatibke with the OpenVPN Technologies server, which is OpenVPN itself, with the value-added parts wrapped around the server.

Well, enough fan mail for OpenVPN. Time to get back to work, doing real work from home instead of fiddling with a VPN.

[UPDATE: RedHat / Centos seems to have dropped OpenVPN from their repositories. You may have to build OpenVPN from source. Not too hard, but no longer fiddle-free.]

Cool Color Ilusion

Clicking on each image will open a full-size popup window.

Open each black and while image and put it in the center of your screen.

Open the corresponding color image, align the popup window directly over the black and white image, and pre-position your mouse cursor on the "X" or "-" control that will make the color image window go away.

After you have stared at the dot in the middle of the color image, click to make the window go away and see what the black and white image underneath looks like...

Not sure why, but the color bars don't work as well as the landscape. Either the illusion works better with "earth tones" or our brain is hardwired to "imagine" landscape colors?

If you're a Javascript maven, send me some code so I can learn how to make Javascript mouseovers work. The WSANDERS ORGANIZATION are not Javascipt experts - yet.

Thanks to johnsadowski.com by way of Simon Singh

It's Always Something -- Weird

I discovered a very mission-critical server (the one that handles mail for 5000 people) had only one of its two redundant power supplied plugged in. When I plugged in the second power supply, the chassis stayed up but one of the cheap-ass Escalade RAID controllers began dropping disks off line and the system crashed. When I unplugged the power supply, the disk errors stopped. Luckily, the filesystems fsck-ed clean.

That's about the weirdest hardware thing I've seen in a while.

Progress ...

The Juniper routers are configuring themselves and throwing themselves into the racks. This is easy!

We got balled up in a licensing dispute for our Cisco ASA firewall. Before I felt comfortable putting 3000 people behind a single box, I figured we ought to get a second unit and failover working. But someone was sold mismatched licenses, and we had to throw out a $5000 upgrade license to get the units to work together. Every other brand promises "No Surprises" but Cisco seems to not mind packing a show-stopper in every device. Their tech isn't bad, it's that the products are so complicated that their sales channels cannot understand them and sell you the right stuff the first time. I don't like surprises.

Why We Just Bought 100 Juniper Boxes Instead of 100 Cisco Boxes

In spite of the sales reps outdoing themselves as to who could offer the deepest discounts (like hotel rooms in Vegas, only a fool pays list price) the Juniper EX series of switches offered a clear advantage over the Cisco Catalyst series. In the end both vendors offered us a big pile of little boxes for about the same pile of cash. About the same number of ports, about the same number of boxes.

To get the prices down in the same ballpark as Juniper, Cisco had to specify 12 different Catalyst models, all the way from lowly 3560's, which I consider old-timey, to 3750E's, with various port configurations and features. Juniper: Only five different models: The 24 port all-SFP EX4200, 24 and 48 Cu port EX4200s, and 24 and 48 Cu port EX3200s. (If we'd had a little more cash I would have like to have bought all 48-Cu-port models.)

All Juniper ports are gigabit. Most Cisco ports in our spec had to be 100 megabit to match Juniper's pricing.

There is a rumor floating around that some Cisco devices are coded to reject non-Cisco (i.e. non-ridiculously-overpriced) SFP modules. If it's true, that's just evil.

Some Juniper switches were spec'ed without any fiber ports, which made them cheaper. Adding four SFP fiber ports is a $500 slot option, and if you don't need it you don't have to buy it. You can swap the $500 slot for a $1500 slow with two 10-gig ports when it's time to upgrade.

All Juniper devices run the same OS. No fussing about which version of IOS to get, and especially whether the features you need are in the apparently random selection of features is in the IOS you get.

The Juniper EX series has field-swappable fans and power supplies. Lose a PS or fan in a Cisco 3600/3700 series, except for a few high-end models, you have a dead box.

The Junipers all have POE on the first 8 ports. The built-in JunOS web interface is generally better than the built-in IOS web interface and is good enough for many setups.

OTOH: The Junipers are loud. Do not expect to install the EX series under someone's desk or anywhere else out in the open.

OTOH: It's not terrible, but nothing compares to the vast collection of generally well-written documents on Cisco's web site, and their active user community.

OTOH: You have to learn JunOS. Not hard, it has it's pluses and minuses, and if you know the fundamentals of the parameters you are trying to set, it's not hard to learn.

Sun X4540: Best. Box. Ever?

Here is what I like to see: A Sun X4540 "Thumper" with 45 of its 47 disk drive (one removed for testing) lights blinking furiously at an estimated peak IO bandwith of 400 MBytes/sec read / 300 MBytes/sec write. Configured as a 46-physical-disk "raidz" (essentially RAID5) array, this system does everything a Netapp or similar "magic box" would do except NDMP, at half the price, and with full Solaris OS functionality thrown in for extra. (Can you run BIND or Apache on your storage applicance?) Everything is hot swappable, about 19T as-configured, spread out across 6 SATA controllers, fits in 4U, field-upgradeable to double that, and we got the whole kit and caboodle for half list price under a Sun educational grant program. This could be the THE BEST BOX EVER.

"Switchport mode dynamic desirable" is Evil?

You can get lazy with Cisco switches after a while. Take the default switchport mode for Catalyst 3500-series devices, "switchport mode dynamic desirable". OK, I can configure some VLANs on some switches, connect them together, and they will form up trunks and usually do the right thing.

But what if there's a router or firewall you are going to hook up to this mess, and it doesn't speak DTP or ISL?

I have a subnet with one 3500 ("SW1") and a PIX ASA5500 firewall ("PIX"), and I wanted to hang another 3500 ("SW2") off it. Only three connections - what could possibly go wrong? The interface on SW1 is set up like this:

 interface GigabitEthernet0/2
 switchport mode dynamic desirable
interface Vlan1
 ip address 10.0.112.1 255.255.0.0

And the PIX interface:

interface GigabitEthernet1/2
 nameif 10net
 security-level 60
 ip address 10.1.0.2 255.255.255.0

I can ping each interface from the other. Next, I configure an interface on SW2 exactly the same as SW1 (obviously with a different IP address):

interface GigabitEthernet0/2
 switchport mode dynamic desirable
interface Vlan1
 ip address 10.0.0.2 255.255.0.0

When I connect SW2, I immediately lose connectivity between SW1 and the PIX. What just happened?

DTP tries to do the right thing, and sets up a trunk between SW1 and SW2. But the PIX doesn't speak ISL (and I don't think it does DTP by default but I'm not sure.) The PIX is left twisting in the wind.

Remember "switchport mode dynamic desirable" is the default. It may save you minutes of configuration time, but you may spend hours figuring out why it blew up.