Keyboard under System Settings should give you options for changing all the modifier keys. I've got Caps Lock modified to be Escape on mine. Though the jokes on me as I just hit ESC all the time and forget about my remapped Caps.

There's similar options on iOS if you use an external keyboard there.

karabiner allows you to make caps lock escape when pressed alone and ctrl when pressed with a letter key (caps lock is where ctrl was on the keyboards vi was designed for)

No because Backspace is not considered a modifier key. The available options for Caps Lock are Control, Option, Command, fn, Escape or just disabling the key entirely.

If you have a workflow that involves RDP to Windows, you also basically have to use Karabiner to swap command and control for RDP only in order to maintain sanity of shortcuts between macOS/Windows.

There is a builtin utility for this on Macs with a few ways to configure. This is the commandline arg to remap caps lock to backspace

hidutil property --set '{"UserKeyMapping":[{"HIDKeyboardModifierMappingSrc":0x700000039,"HIDKeyboardModifierMappingDst":0x70000002A}]}'

Out of curiosity, is this approach flawless? Does it work always, in every app, including login screen?

Yeah it does. You can create a config file that persists the mapping on reboots (described in depth here https://rakhesh.com/mac/using-hidutil-to-map-macos-keyboard-...)