I'll try it again but I found that oddly using a Baseus iPad Pro USB C hub also fixes the issue. Unfortunately the headphone jack on the hub doesn't have enough gain for me so I still have to use the dongle, but at least I don't have to unplug and replug for every call.