I only want to restore part of the changes from the remove silence edit.

Clicking [Restore sequence] will revert the entire edit. Partial restoration is not supported.

If you want to continue editing after silence removal, it’s recommended to use the [Auto rough cut] menu to remove silence with more flexibility.