I am going to share a story I think most engineers who deal with network equipment can relate to.
A vendor releases a critical firmware update to mitigate a vulnerability and now it is your responsibility to get this update installed with the least impact to the business. To make sure you meet this objective, you go through a series of steps:
- You install the firmware update on your test lab environment (or your home equipment) to make sure you do not encounter any obvious issues during a basic sanity check.
- You follow your established change management procedure and document the install instructions and test plan.
- You get your favorite energy drink and some junk food because you know this change is going to take place in the wee hours of the morning when normal folks are sleeping.
- You take a backup of the existing configuration and firmware to have on hand in case you need to revert to the previous state.
In many cases, engineers plan for everything to go right but what happens when things go wrong? I have encountered engineers who think the only roll back plan after a unsuccessful firmware upgrade on the FortiGate is to load the previous firmware and restore a backup. I have even encountered some engineers who think that a TFTP firmware update is the only way to restore an unsuccessful firmware update. While both of these methods work, they are time consuming and with the added stress of services being down during the maintenance window, an engineer needs to have a faster way to get everything up and running.
Fortunately, with the FortiGate hardware platforms, there’s a recovery partition that you can restore the last working firmware and configuration prior to upgrading. Below is an explanation below on how to use this feature.
Showing the partitions
The FortiGate platform has two partitions. They can be listed by using the following command in the CLI:
diagnose sys flash list
To explain the screenshot above, Partition 1 (Primary) contains FortiOS 5.6.7 (b1653) while Partition 2 (Secondary) contains FortiOS 6.0.3 (b0200). Although partition 2 is labeled as “secondary”, it is the active partition and is currently what loads when the FortiGate is booted. You can see the total size of the partition as well as the amount of each partition in use.
While this listing only displays the current firmware associated with the partition, there is also an configuration file tied to the partition as well. The firmware size and configuration size together add to the total “used” space of that partition. In the next section, I will explain my interpretation of what happens during a firmware update.
How Firmware Upgrade affects Partitions
Before I move forward with upgrading, I am going to make a quick change to the configuration to demonstrate the point I made earlier about the configuration being tied to the partition. For now, I am just going to add an address object called “h-historiantech” to the firewall:
Now, we are going to run through the scenario of upgrading the firmware through the most common procedure (via the WebGUI). First, you can see that the current firmware that is running on my test FortiGate 60E is FortiOS 6.0.3:
Remember from our previous image above, the current partition where this firmware is running from is Secondary (2). Now let’s proceed with the upgrade:
When the FortiGate goes through its upgrade procedure via the WebGUI, it applies the firmware upgrade to the non-active partition and takes the existing configuration on the active partition, upgrades it and assigns it to the non-active partition firmware. It also marks the non-active partition as active.
After the reboot has completed, we can verify via the WebGUI that indeed the firmware has been updated to FortiOS 6.0.4.
Now, if we run the command to list the partitions, we see what changed:
diagnose sys flash list
We see the following:
- Partition 1 (Primary) Image now reflects 6.0.4 (b0231)
- Partition 1 (Primary) became the “active” partition
- Partition 1 (Primary) “Used (KB)” increased in size (most likely due to larger firmware image and larger configuration [6.0.4 adds more configuration parameters than 5.6.7]
Assuming that all the post-upgrade tests pass, we can usually stop right here, but what happens when we need to revert the upgrade?
Reverting to the Previous State (Firmware)
As I mentioned earlier, the “Active” column on the output of the command to list the partitions instruct the FortiGate which partition to load upon start up. To restore to a different partition, you simply have to change the “Active” partition and reboot the FortiGate. To do this, use the following command:
execute set-next-reboot <primary/secondary>
As I alluded to earlier, “Primary” corresponds to Partition 1 while “Secondary” corresponds to Partition 2. This can be confusing because it is easy to interpret “Secondary” as the non-active partition, but this is not the case.
In this example, since I want to restore my FortiGate back to FortiOS 6.0.3 (b0200), I need to execute the following command:
execute set-next-reboot secondary
Now, before I reboot the firmware, I am going to remove the address object “h-historiantech” I added prior to the firmware update.
Okay, now we reboot the FortiGate and observe the result:
After the FortiGate boots, we can check and confirm the current firmware shows 6.0.3 in the GUI:
And lastly, we confirm that the old “h-historiantech” that was previously deleted on FortiOS 6.0.4 is indeed showing under the restored partition:
This is exactly what we expect to see. I hope that in the future, when you are performing a firmware update on a FortiGate, you can rest a little easier having this trick up your sleeve in case you need to revert a firmware update.
Please Note: This is only applicable to FortiGate Hardware models. For FortiGate VM, please use the snapshot facility of the hypervisor or other best practice backup methods (such as taking a configuration before performing an update).