SegFault

The Supermicro X10DRi-T server mainboard is a dual-socket LGA 2011 board that supports Intel Xeon E5-2600 v3/v4. In 2016, when I built my home server, it supported all the bells and whistles I could need with a relatively affordable price tag. Therefore, it became my mainboard of choice and has since reliably powered my home network. As an enterprise-oriented product, it supports various features, like PCIe bifurcation, that only recently became standard on consumer products.

Despite getting regular BIOS upgrades until 2021, which included several bug fixes and new CPU microcodes, it never got support for booting NVMe disks or resizeable BAR. You can work around the first issue by providing a USB stick that chain loads to the NVMe drive, however, that's not a clean solution. The second issue can not be worked around but most devices don't require support for resizeable BAR.

This is where the modern UEFI standard comes into play, which in addition to specifying a new BIOS interface for the operating system, also defines APIs and an environment for modular BIOS environments. We can use this facility to manually add support for arbitrary new features to an existing firmware as long as there is space in the onboard flash chip and we have a UEFI module that is compatible with the hardware.

Luckily the X10DRi-T ships with a large 16Mbyte flash chip and the module for booting from NVMe is part of the edk2 driver sdk, so we can easily compile our own or pick it from a different firmware of a board that does support NVMe boot. I decided to go with the second option and extracted the module from the firmware of another Supermicro board X10SRi-F, which does support NVMe boot.

While I could not find a Supermicro server board that supports ReBAR, I did find a genius bios mod that patches a couple of functions with the same result.

Getting ready

I prepared a all-in-one package that includes all the required tools and files, as well as a ready to flash firmware, however all the tools are freely available.

To modify the BIOS I used the excellent UEFITool which is available for both Windows and Linux. Make sure you use the latest version that does not use the "new" engine because those do not support patching firmware yet and can only be used to extract things. I used version 0.28.0 which was the latest available at the time.

In addition to the tool we also need to download the latest firmware for the board, the firmware of the nvme "donor" board, as well as the latest build of ReBarUEFI.

This is also a good moment to consider getting your hand on a license key for the integrated BMC, which allows you to flash the bios from the WebUI, which should work even if the system fails to boot properly. It's not a requirement but a strong recommendation.

Modifying the BIOS

Launch UEFITool and open your donor bios. You will need to change the file extension filter because Supermicro uses a non-standard file extension. You have to select the file that's roughly 16Mbyte in size and has a three-digit number as an extension, which in the case of X10SRi-F is "X10SRI1.609".

Extend the "Intel image" and "BIOS region" trees, which should present you with a couple of Volume entries. One of them will contain a number of elements of subtype "DXE driver". Look for the entry with the name "Nvme" and extract it using "Right click => Extract as is.." or press Ctrl+E after highlighting the entry.

You can now open the BIOS image for the X10DRi-T board, which is called "X10DRI1.816" and again navigate to the volume containing all DXE drivers. At the end of the volume you will find an entry called "Volume free space". Right-click the entry right before it, select "Insert after" and choose the nvme.ffs you extracted earlier. Repeat the step by right-clicking on the new NVMe module to insert ReBarDxe.ffs.

Save the image using "File > Save image file" and make sure you use the same file extension as the original firmware.

Make sure you have screenshots/notes of all changes you made to bios settings because they will be lost. Shut down the server you want to patch and use the BMC WebUI to flash the new BIOS. If everything worked correctly the server should reboot as normal. Enter the BIOS, reapply all your custom settings and make sure you enable "Above 4G Decoding".

Finishing up

The final step is to install a UEFI-capable OS on your NVMe drive. Once you reboot the bios should correctly pick up the NVMe drive as a boot medium and allow you to boot from it.

The last step is to enable ReBarUEFI, which by default is inactive to avoid bricking on consumer boards that have no out-of-band bios flashing. To do so download and run the "ReBarState" executable from the repository for the operating system you use. If you use Windows they provide a prebuilt version, while Linux users need to manually compile it using cmake.

Once you have done so you can launch the tool on your server with administrator/root privileges. It will ask you to configure the maximum per device ReBar size the system should support. While the system booted fine with any value I tested, the Intel Arc A380 GPU failed to initialize with it set to unlimited, so I settled on the lowest value that's larger than the GPU VRAM (8Gbyte), which has been working flawlessly so far.