Making a Champagne VTL on a Beer Budget
This post will be about how to make some really nice backup storage and some
random thoughts about different ways to make it all work, without having
to be rich as Croesus to do it. :-) At work
(SGI), I happened to have an old version of our
MAID product on hand.
And I wanted to use it for backups. There's other ways to build a backup
server and SGI has some other products that might be more suitable too (more
on that later). But this is the hardware I happened to have available.
More on the MAID product. Basically, this is storage that's very dense, and
meant to be used strictly for archival purposes. And to prevent over-heating
and to keep the electricity costs down, only some of the drives are powered
on at any given time. SGI now has storage offerings that are even denser and
that allow all the drives to be on all the time, but of course, that uses
more electricity and costs more to operate. :-) See the
NAS products for more
info on them.
So, the MAID storage is, at the time of this writing, able to contain almost
3 petabytes in a single rack. Each rack contains up to 8 shelves. Each
shelf contains 26 usable LUNs and one special LUN. The drives that make up
a LUN are powered on and off on demand. And only a configurable number of
LUNs are allowed to be powered on at one time - in my case, 7. The 27th LUN
in each shelf contains what's called an "always on region" or AOR. A portion
of each of the other 26 LUNs is cached here. For instance, the default AOR
region for linux systems contains a portion of the start and end of each LUN.
That way, when a linux system boots, and it sees 26*8 disk devices (26 LUNs
on 8 shelves) and it scans the partition table on each disk device, it can
read this from the AOR and not have to power on every LUN. When the OS
tries to mount the partition on the LUN, it will need to access other parts
of the LUN and there could be a 15-20 second delay while the drives are
powered on and spun up. As an aside, I made a custom AOR that makes it
possible to mount and unmount a single XFS filesystem on each LUN without
having to spin up the drives, but I chose to do something else.
I've found one really effective way to make efficient use of this sort of
storage is to format each LUN's single partition with a nice label like,
shelf-X-lunYY, and then have the automounter setup to mount these on
demand. For instance, I'd have this line in my auto.master file:
And /etc/auto.maid would have:
echo "-fstype=xfs :-L$key"
So then I could access the LUN on shelf 3, LUN 24 just by accessing
/maid/shelf3-lun24 (assuming I'd formatted it with that filesystem label).
And it also means that most of the time, most of the LUNs aren't mounted.
When I reboot, there's few, if any, MAID LUNs to unmount first, and none
required to mount when the OS restarts. Nice 'n clean.
So how could I make use of this in a backup program? Most backup software
allows us to write to filesystems these days. But the tricky bit with the
MAID is how can I guarantee that the backup software will never use more than
7 filesystems on a shelf at any given time? What I'd wind up having to do
is to make "pools" of storage that use no more than 7 LUNs on any given
shelf. Then I have to juggle my schedules and clients around to make sure
they use the right pools on the right days. It could be done, but it means
I have to configure clients, pools, and schedules carefully or I wind up
with some pools running out of space, some with little space used, etc. And
then when you toss in the ability for anyone to start a recovery at any
time, it just gets more complicated.
The obvious solution is to just turn each shelf into a VTL (virtual tape
library) with 7 tape drives and 26 slots. Each slot is a LUN. That'd still
mean making sure the backup software would spread the load out across 8
separate tape libraries, and it could be horrifically expensive to license
the backup software that way (if you're using commercial software). If
massive I/O throughput isn't an issue (and it wasn't for me), you can also
just set it up as one giant tape library with 208 slots and 7 tape drives.
That way the VTL abstraction (7 tape drives) prevents any backup software
from ever using too many LUNs at the same time, but you have tons of storage
and you can treat it all the same if you like. No juggling of pools, clients
and schedules. Use pools and schedules when they benefit you, not to
try to trick the backup software into behaving a certain way.
So, I compiled and installed some opensource VTL software called
MHVTL. This is pretty
cool stuff. Basically, I configured a new virtual tape library with 256
slots (just a nice, round number bigger than 26 * 8) and then told it I wanted
208 tapes with volume names like "S00021D7". The trailing D7 tells MHVTL
that this tape is a DLT7000. The first 00 is a shelf number. The 02 would
be a LUN on that shelf, and the trailing 1 would be a partition. I chose
this naming convention just in case I ever wanted to have each LUN
broken into multiple partitions. I'd have to make a custom AOR for this
to work efficiently (one that contained the partition table plus the first
part of every partition so it'd contain all the filesystem labels too).
Then I told MHVTL that this tape library had 7 tape drives that were each
DLT7000 devices. By default, it wants to find a directory with the same name
as every tape label in /opt/mhvtl/ (you can specify a different
base directory for each VTL you setup). Initially, I used the automounter
to handle this so I wouldn't need to have them all mounted all the time. I
made /opt/mhvtl/30/tape-label automount the LUN with the filesystem labeled
tape-label. And it worked ok until the backup software tried to label all the
tapes, accessing them one by one in rapid succession to write a header
to each tape. The automounter didn't deal well with that as it didn't
unmount the filesystem when the VTL unmounted a virtual tape from a virtual
drive - only when it was idle for longer than X minutes.
As a quick workaround, I tweaked bacula (the backup software I was testing
with) so that it'd execute a script wrapper around all jukebox operations.
So I made a wrapper script that would mount /opt/mhvtl/30/whatever before
it asked MHVTL to mount that virtual slot in that virtual drive, and to
unmount it (and flush the buffers) after telling the jukebox to unmount that
virtual tape and put it back in it's virtual slot. That worked like a
charm. Bacula happily mounted 'n mounted every virtual tape, labeling them
all for the "Default" pool and MHVTL did it's job making all these XFS
filesystems look like really big tapes.
At the moment, one limitation is that MHVTL assumes every virtual tape is the
same length. So if one shelf has 500 gb drives and one shelf has 750 gb
drives (yeah, this is old hardware, remember, so I don't have 4 TB drives -
grin), then the LUNs will be different sizes. One possibility would be to
make multiple partitions per LUN, so each LUN would be exactly the same
size, but shelves with bigger drives would have more partitions per LUN
than shelves with smaller drives. Of course, this would require a custom
AOR file. But with shelves containing the same types of drives, one could
make a VTL that had 208 virtual tapes, each of which was around 2.5 TB
of uncompressed storage. MHVTL can do decent compression too (zlib or lzo).
Now that's some decent backups. :-)
I also tinkered with making a custom AOR so that I could mount
and unmount every LUN without it requiring the LUN's drives to power on.
The AOR was big enough that I was able to do this, and it worked. But the
downside is that if I ever had an unclean shutdown all of these filesystems
would be marked as dirty and would have to be scanned on a reboot. So I'd
much rather only mount them as needed and unmount them as soon as I'm done.
It just makes more sense.
Not long after I finished my quick 'n dirty proof of concept, SGI rolled
out a new product that I think will be an even better solution called
Gateway. This, to me, looks very exciting because it makes the whole
VTL abstraction unnecessary. Basically, you have hot, fast storage in an
appliance that keeps all the metadata for a giant filesystem in the hot
storage, but can take blocks of data that haven't been accessed in a while
and move them off to other tiers of storage. These other tiers might be
MAID storage, or even physical tape libraries.
So with this, I could just tell the backup software "Here, use this truly
humongous filesystem as your storage", write to it, and have the
InfiniteStorage Gateway automatically take backups that were, say, a week
old and migrate them to archival MAID storage. The data is still there
and if/when I had to do a recovery and the backup software tried to access
one of these files whose blocks had been migrated to archival, the gateway
would automagically pull it back into the hot storage within (I think)
15-20 seconds. Very cool!
Basically, I could have my low-power archival storage, but use it the same
as any other big, heat-generating, power-sucking storage array. :-) And if
that weren't compelling enough, you can configure the Gateway so that
it requires the migrated data to have multiple copies for extra redundancy.
I'll have to check, but it might even be possible to have it migrated to
other geographies. In a perfect world, I'd want the Gateway to not only
have NFS/CIFS support but to also run a bacula storage daemon. But I
suspect that's possible.
Gonna hafta lay my hands on one and give it a try, just to see... (grin)