So it seems that bringing the system down to single user mode and letting fsck do its stuff has solved the immediate problem of terrible performance.
But the longer term problems still exist. Both of our RAID arrays are running degraded. Another disk failure and we could lose the data.
Current plan is to ditch the hardware and move the entire system to a VM on a server that being generously donated. Work is in progress on bringing up this VM and migrating everything over.
Stay tuned. More to come.