ZFS endian neutrality

One of the draws of ZFS is that a disk can be created on one machine architecture (say Intel) and used on another like Sparc. One has to be careful for this to work however.

The disk must also have an endian neutral partitioning scheme like EFI. If you use another partitioning scheme like Solaris VTOC, ZFS will work just fine on a single architecture but fail when taken to a different endian machine.

On Solaris, the fix was to run format with the -e (expert) field. -e enables both using EFI formats as well as removeable devices like a USB flash drive. See the OpenSolaris Bible (Ch. 7 and 8) and the System Administration Guide: Devices and File Systems.

Another gotcha is that some machines can't boot from an EFI formatted drive so that can complicate moving root disks between architectures.

NFS write performance

In prior posts, I've complained about the poor NFS write performance I've seen. A bit of research later, I've discovered that it's because OpenSolaris is strict about syncing NFS writes to disk (ref: NFS and ZFS, a fine combination)

There are ways to improve the time (ref: Disabling the ZIL, Don't) but the prevailing view is to put the ZIL on a separate device from the main ZFS data. I decided to give it a try using a 16 GB Sony USB flash disk.

I couldn't get it to work because the flash drive kept getting ZFS errors (ref: Bug 14676 – ZFS pool fails on Sony USB Flash drive - device disappears during use). A different flash drive worked but was too slow to help as a separate log device. Maybe I'll have to break down and buy a SATA flash drive. In the mean time, SMB is working out.

OpenSolaris Realtek Network Interface Redux

In my last post, I tried using the "gani" driver for my Realtek RTL8111/8168B Gigabit network interface. In OpenSolaris bug 6892693, they patched the original driver so I decided to try the update.

I switched to the development repository but I couldn't find a way to get the package system to install the rge driver by itself. I set up another boot environment for the entire dev tree and updated to rev 131. I played around a bit with the dev boot environment but it was too buggy for me. I then copied /kernel/drv/rge and /kernel/drv/amd64/rge back to my stable boot environment and replaced the original rge drivers.

The new drivers worked reliably but didn't help the NFS transfer rate. I did find that scp (secure copy over ssh) clocked in much faster at 22 MB/sec. so I decided to set up SMB file shares.

If you haven't already bought it, the OpenSolaris Bible has great history and timesaving advise for setting up SMB and other parts of OpenSolaris. Using Sun's kernel implementation of SMB, I also got 24 MB second. I have to see if I can find tuning advise for NFS.

OpenSolaris Realtek Network Interface

The ASUS M4A785-M motherboard has a built in Realtek RTL8111/8168B Gigabit network interface. While it worked out of the box with OpenSolaris 2009.06 (svn-111b) it would drop the connection at random points never to return. The logs were silent about the cause.

Found lots of trouble reports on the net including Bug ID: 6807184 rge driver drops off network. One of the later replies to serialized.net » The littlest Thumper: OpenSolaris NAS on an MSI Wind PC summed it up well: "There’s no know work around for this with the native rge driver. However, there is a third party driver “gani” that seems to have fixed the issue for me: http://homepage2.nifty.com/mrym3/taiyodo/eng/"

One of the many great things about OpenSolaris is that you can set up multiple boot environments, that is snapshot your existing setup, with beadm. See Working With Multiple Boot Environments on the OpenSolaris OS for details.

I pulled down the 2.6.4 "gani" driver and installed the amd64/gcc version into a new boot environment. It solved the network dropping problem but it's painfully slow. Writes over NFS top out at 2.1 MB/sec on a gig link. The old rge driver didn't do any better. I may have to get another network card...

OpenSolaris host

I decided to build an OpenSolaris server but wanted to keep the cost down. The parts were:

Component	Cost
Case and PS	$74.99
Motherboard and CPU	$179.99
Memory	$101.98
Mirrored drives	$0.00
Keyboard	$17.99
Mouse	$0.00
DVD-R, CD-RW	$0.00
Display	$0.00
Graphics card	$0.00
Total	$374.95

The case and power supply were a Fry's special. Antec Two Hundred Mid Tower case and Antec Basiq 550W Plus PSU ($20 rebate shown in cost). The case has lots of room for expansion but isn't a tool-less design.

I got a motherboard and CPU combo that was shown in the OpenSolaris compatibility list verified as compatible. The ASUS M4A785-M AMD AM3 Motherboard has good expandability and on-board graphics. The AMD Athlon II X4 620 2.6 GHz AM3 CPU is an inexpensive quad code CPU.

I got 2 Kingston 2GB 240-Pin DDR2 SDRAM DDR2 800 (PC2 6400) KVR800D2N5/2G memory sticks and have two slots for future expansion.

I used two 1 GIg drives from a failed Sans Digital external mirror case. I'm using ZFS to mirror the drives in OpenSolaris.

Other than a SIIG Mini USB Keyboard, I had spares for the mouse, DVD drive and display. Except for the initial install, I'm planning to run the machine headless.

The OpenSolaris initial install went smoothly so now it's time to have fun with ZFS, xVM, etc.