Honestly? Using 512GB of ram is pretty hard. I learned during some of this that you can actually get tmpfs to be CPU limited if you try to write to it too quickly.
Working with IPMI can be tough. We nailed the issues eventually, but there were a few frustrating days of images not booting properly with no explanation. Until we found the right combination of settings, we had issues with flakey BMCs that would stop responding for no apparent reason.
