So I'm a bit embarrassed. I was going off my profiling work done 3 months ago when I was still having issues. I've been working a lot on optimization and hadn't done any proper non-headless profiling with high ship numbers since then. After seeing this your comment, I spent last night running some performance trials and the GPU rendering is down to 3ms even at a high number of ships. Any further slowdown is still coming from the physics step.

I unfortunately can't edit my comment anymore to correct what I wrote.