NetBSD and RAIDframe History

May 25, 1998 - Initial idea to do some sort of RAID 5 for NetBSD. Sounded like a cool, interesting, and useful project, and so I spent the next few days digging up papers and kicking around a number of ideas. Pored over the ccd.c looking at how expandable that might be.
May 31, 1998 - First kick at coding up some RAID 5 stuff in ccd.c. Mostly just "proof of concept" types of hacks. Still learning about device drivers. The initial "disks" used in this project were vnd devices, exported from my main box to a development box.
June 2, 1998 - Second kick at a ccd-based framework. Ended up getting a pseudo RAID 4 and RAID 5 implementation done ("pseudo" because it just spread the data bits around, but made no attempt to store or even compute parity).
June 9, 1998 - After expressing an interest in a RAID project to some of the main NetBSD dudes, it was suggested that I have a look at the RAIDframe work. Spent the next few days looking at the pros and cons of switching (pros: RAIDframe was MUCH further along; cons: my stuff I could partially understand). Didn't take me long to figure out that RAIDframe was the way to go...
June 15, 1998 - First kernel compile that actually has all the references defined. Not even close to functioning, as many of the offending functions were just gutted, but it was a start. Still using vnd devices for "disks".
June 24, 1998 - Looked like disklabels and newfs were working, but due to some really obscure little buglets, they really wern't... Still, I was pretty happy that I had most of the driver in place, and it was just a "small" matter of debugging it.
July 2, 1998 - Was able to copy data to/from the RAID device.
July 5, 1998 - Played with RAID 0, RAID 1, RAID 4, and RAID 5. Still all drives are vnd's being pulled in over NFS.
July 8, 1998 - The first "clean" compile (i.e. warnings are considered errors). It wouldn't stay that way for long... :-)
August 10, 1998 - Returned from a 3-week holiday. Started using real (physical) drives instead of virtual ones. Unfortunately all I can find around here for cheap are a few IDE drives.
August 23, 1998 - Toasted one of my IDE drives.
September 2, 1998 - Fixed a bug that allowed the pmax port to be able to use the RAIDframe driver. Also got parity re-writing working.
September 3, 1998 - Fixed a major bug in the reconstruction code (Yes, the bug is in RAIDframe 1.1 as distributed by CMU). Once this was fixed, then reconstruction (to a hot spare) and copyback (to a "replacement disk") worked like a charm. I was able to fail a disk, reconstruct to a hot spare, and copyback to the original disk, all while the associated filesystem was mounted.
September 11, 1998 - Upgraded development machine to NetBSD 1.3H, as of Sept. 5 sources. Basically in a stress-testing/bug-fixing phase now.
September 13, 1998 - Have re-built the world via sources living on a RAID 5 partition.
September 29, 1998 - Started more serious testing of the driver, as I've purchased a number of old SCSI drives (Thanks Russ :-) ). Pmax port has no problems with the driver, and is quite stable.
October 5, 1998 - Built and ran the RAIDframe kernel on both sparc and sun3 machines.
October 8, 1998 - Converted driver to use the new kernel memory pool allocation code.
October 17, 1998 - Nuked a lot of little bugs which were causing panics when incorrect configurations were given.
October 27, 1998 - In certain configurations the RAIDframe driver and the old CCD driver are 100% binary compatible. Yes, this means that I could mount up a striped filesystem under either the CCD device, or the RAIDframe device. Benchmarking indicates that the new RAIDframe driver is only slightly slower than the old CCD for these disks. These results are for simple striping (RAID 0), 3 different SCSI disks (aggregate total of 1.5GB for the RAID unit), on a P133 w/ 32MB of RAM. The Sectors Per Strip Unit is 32 in both cases. (In fact, I simply mounted the drives via either the CCD or the RAIDframe driver to do this comparison).
```
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
CCD       500  3165 98.4  4486 14.1  1498  8.4  2781 92.9  4090 12.2  22.4  1.9
RAIDframe 500  3033 97.9  4557 18.2  1472 10.7  2662 91.7  3859 14.4  22.2  3.0
```
November 7, 1998 - re-wrote the main part of the raidctl program. Added additional functionality (e.g. obtain the status of the driver) and a few other features (can now specify a device to fail via it's component device name, (e.g. /dev/sd1e) instead of just a component number).
November 8, 1998 - wrote the bulk of the raidctl.8 and raid.4 man-pages.
November 9, 1998 - fixed a serious bug which would have prevented hot-swappable drives from being useful. With this bug fixed, the driver should be able to deal with new drives that come on-line after the device has been configured.
November 12, 1998 - Integration into the -current NetBSD tree. Whee!!
November 14, 1998 - Purchased another five 1GB drives, and a big tower case.
November 26, 1998 - Picked up an Asus SC-875 UW SCSI controller. I now have 2 controllers and 7 1GB SCSI disks in my test box... Whee!!! After a bit more testing I might actually have my test box functioning properly again. (As of Nov. 26, it's only been really back on-line for the past 3 days, due to lack of time to get things configured.)
December 2, 1998 - Finally isolated the bug that was preventing physical disk failures from being detected and handled properly.
December 12, 1998 - Not much new these last few weeks. Mostly doing testing of physical disk failures.
January 4, 1999 - Made the January edition of Daemon News in BSD News. :-)
January 6, 1999 - With holidays over and life returning to normal, perhaps I can find time to start shrinking that TODO list...
January 10, 1999 - Ok, so this is both good news and bad news. The bad news is that I lost one of my drives on my test box (it makes funny 'clicking' noises when it tries to spin up)... The good thing is that the disk died while I was actually stress-testing the RAID set, and the machine didn't even skip a beat.
January 25, 1999 - Phase 1 of 'The Great RAIDframe Rototilling' is now done. I nuked the simulator, user-land driver, old comments, and a few other things from the kernel code. The rationale here is a) we don't want to support the simulator in kernel code and b) if we're going to support RAIDframe on production boxes, we might as well clean up the code so we can understand what is going on.
February 4, 1999 - Phase 2 of 'The Great RAIDframe Rototilling' was just committed. Basically fixed up the source to be closer to KNF. This makes incorporating any diffs from CMU difficult, but that event seems very unlikely anyway.
February 15, 1999 - First part of the component label stuff is done. Labels can be read/written, and the kernel will detect if components are not specified in the correct order, or if parity needs to be computed. Not in the main tree yet, as there is more work/testing that needs to be done. RSN, I hope...
February 23, 1999 - Got the first bunch of component label changes into the tree. It's not done yet, but it's getting there. Stuff for hot-spare adding is also starting to take shape. Again, the code is there, but it's not enabled. (Adding seems to work like a charm. Removal is turning out to be an entirely different ball of wax.)
February 28, 1999 - The remainder of the component label changes, hot-adding spares, rebuilding in-place, and a re-working of the raidctl options is now *very* close to being done, and will be in the tree in the next day or two. This work also includes clean-bits for each component, and the use of shutdownhooks to help maintain those clean bits.
March 1, 1999 - Component labels, hot-adding spares, rebuilding in place, et al. are now in the main tree. Man-pages have been updated to reflect the new state of things. Label detection/analysis tries to be somewhat intelligent in deciding whether or not it can just fail one component and continue, or whether it should just bail on the user. The code also makes use of shutdownhooks, in an attempt to keep the clean bits up-to-date in the event of a normal reboot where the RAID set is not unconfigured before rebooting. If you're running RAIDframe from before this date here is the upgrade procedure.
March 8, 1999 - Re-worked how clean-bits are maintained. In addition to being set/unset at configuration and unconfiguration time, the cleanbits now get set/unset as appropriate each time a partition is open/closed. This has rendered the shutdownhooks stuff unnecessary.
March 14, 1999 - Fix the "crash on interrupting parity re-write or copyback" problem. Disallow interrupting parity re-writes or copybacks by removing PCATCH from the tsleep()'s (they probably shouldn't have been there in the first place!). Making parity re-writing and copybacks interruptable will require re-designing how a few things are done (e.g. how memory is freed for structures shipped off to routines that run asynchronously relative to the calling routine).
April 7, 1999 - 'raidctl -i' on a RAID 0 set now sets the clean bit. There wasn't any way to do this (for RAID 0) until now!
May 25, 1999 - I've now been playing with RAID in NetBSD for a whole year! My how time flies when you're having fun.
June 3, 1999 - with help from Martin J. Laubach (Thanks Martin!), a couple of evil bugs related to adding hot spares were caught and squished. The first would cause a panic when configuring a hot spare, and the second would cause a panic when doing a reconstruct to a hot spare.
June 26, 1999 - have added a proper fix to replace the "use synchronous writes" fix (this was related to the "use all kernel memory then panic bug" which was discovered ages ago).
July 7, 1999 - after a bunch of testing the June 26 fix is now in the main tree. Write performance is greatly improved - I've noticed 2x performance in some cases!
July 12, 1999 - no more putzing around with used SCSI controllers. I just picked up another ASUS SC875 (NCR 53C875) Ultra-Wide controller. My main box now has two of these controllers, and each controller has two 2GB Seagate ST32155W Hawks (Fast-Wide). Early benchmark results are very promising. :)
August 12, 1999 - first kick at cleaning up the RF_ETIMER stuff. Peter Galbavy gets credit for the code changes on this one. Thanks Peter. :) Other assorted cleanups have also been done.
August 10, 1999 - raidctl can now check the status of the parity with a single flag, and, with another flag, will check and rebuild the parity if necessary.
August 13, 1999 - more cleanup. In particular, got rid of some ugly 'struct proc *'-passing stuff that's been bugging me for ages.
November 12, 1999 - RAIDframe has now been an official part of NetBSD for one year!
November 8, 1999 - fixed a little bug in the RAID 1 code, where a read from a mirror could return data different from that on the primary disk. Boy, it's been a long time since there's been anything to add here :-/
December 12, 1999 - Did some serious rototilling of the IO throttle code. This fixes some weird panics when doing very heavy IO on a FFS, and also enables RAIDframe to work quite nicely with the softdeps code. (System load average was over 100 during the tests.. the maximum 15-minute load average was about 95 or so. All those processes were beating on the RAID set.) From what I can tell, this makes NetBSD the first free OS to have softdeps + RAID working.
December 15, 1999 - Have totally re-vamped how the reconstruction, parity re-writing, and copyback IOCTLs work. These IOCTLs now spawn a new kernel thread to do the real work, and the IOCTL returns immediately to the user. (not committed to main tree yet.)
December 20, 1999 - raidctl almost completely supports progress meters for reconstruction, copyback, and parity re-writing. This has been working for almost a week now, but hadn't been quite integrated. The '-v' option allows you to do something like 'raidctl -v -i raid0' and watch the progress of the parity re-write.
January 4, 2000 - got changes from Dec. 16 and 20 into the NetBSD tree.
January 7, 2000 - more code cleanup from the last few days.
January 8, 2000 - more code cleanup. Nuked a bunch of un-used files. For default disklabels, the cylinder size is now the size of the (data part of a) stripe.
January 26, 2000 - Booting from a RAID set raises a whole bunch of interesting issues and problems. With the ability to delay thread startup, we're now somewhat closer to booting from RAID, but there is a lot of work to be done first.
February 7, 2000 - Lots of news to report:
1. Have done a serious amount of rototilling of the disk configuration code.
2. A new 'Auto Configure' mechanism is mostly in place. Right now it needs a lot more testing, but is capable of configuring RAID sets before the root partition is mounted.
3. Booted successfully with / on a RAID 1. The kernel still has to be loaded in some "normal" fashion, but this is a *major* step forward.
4. Booted successfully with / on a RAID 0 set.
5. Booted successfully with / on a RAID 5 set.
The code needs a lot of cleanup, and lots of testing, but things are falling into place quite nicely. With any luck, this new code will be in the tree by the end of the week.
February 9, 2000 - disklabel now supports a 'RAID' partition type. While RAIDframe currently doesn't care about the partition type, it will only autoconfigure partitions marked as being of type 'RAID'.
February 10, 2000 - with a kernel loaded from sd0a, my test box now mounts / automatically on a RAID 5 set.
February 12, 2000 - checked in the changes since Feb. 7, but with the autoconfig and 'root on RAID' disabled. It's quite functional under normal operation, but needs more testing related to component failures.
February 14, 2000 - thanks to Hubert Feyrer, the notice about these recent RAIDframe changes made it to Daily DaemonNews. You can find the posting here.
February 15, 2000 - Ok, so this actually happened on the 14th, but I didn't figure it out until the 15th: On the morning of the 14th my main box appeared to have suffered a 2-drive failure on a 5 drive RAID-5 set. "Must be the controller," I thought. Nope... 2 drives did go south. That's the bad news. The good (?) news is that the first drive actually died on Feb. 2, 2000, but I didn't even notice!! (The bad news is that RAIDframe needs to be much crabbier when a disk fails.) I wondered why the performance seemed to be lagging a bit... but the machine had been up for 120 days, and I was blaming it on all the stuff I was doing.
February 15, 2000 - BTW: If you ever run 'raidctl -v -i' (i.e. use the '-v' option) you'll never want to go back to a raidctl without the option :)
February 21, 2000 - Fixed a bad component label initialization bug that made it into the Feb. 15 changes.
February 22, 2000 - more component label initialization code fixes. More importantly, if a component fails, component labels on the remaining disks are updated almost immediately.
February 23, 2000 - Still more component label fixes, and other 'bookkeeping' tweaks. Can now boot with / on a degraded RAID 5 set, add a hot spare, reconstruct to the hot spare, reboot, and have the system come up as though nothing was the matter :)
Feburary 25, 2000 - Still more fixes. Mostly fixing obscure stuff, although there was a mod_counter problem that was only applicable to auto-configurable RAID sets. Changed the SCSI ID of a drive that is part of the RAID set that / lives on - the autoconfig stuff worked like a charm - didn't complain, and configured everything up the way it was suposed to be. Fixed up documentation to reflect the new autoconfiguration stuff.
February 26, 2000 - Added an more 'real' mechanism to turn on the component auto-detection and auto-configuration stuff. You can now specify 'options RAID_AUTOCONFIG' in your kernel config file to turn on auto-detection and auto-configuration.

In case you are wondering, "Yes, I kept detailed notes of the progress", and "No, I havn't posted them all here" :-)

Page last modified: October 29, 2003. Send comments to oster@cs.usask.ca.