Worst that happens is they get silently offended, while being political to you at the same time and secretly undermine your performance reviews during calibration meetings without you even knowing or just not standing up for you that much during this time will be enough due to stack ranking etc.
This - I've had a manager who was pretty ineffective, and if he wasn't doing 1:1s he'd feel like he was failing at managing. Easier to just talk in circles with him.