It's an ergonomics thing, not a "can't" issue. There is a reason I called out Swift in particular—their "unsafe" APIs are so horrid to use that they make you regret doing unsafe things in the first place. Plus, throw in FFI and now you've got an even worse problem because not only are you forced to use the unsafe types, but often a lot of the critical APIs you need to interface with (in *OS exploitation, mach is the worst offender) have such funky types due to their generic nature that you have to go through half a dozen different conversions to get access to the underlying data.
I still don't understand why would you prefer raw memory manipulation to bytestring manipulation. If you want, just make a Swift library that will implement the memory like you want but without unsafe raw access (but just a few methods over a byte array). Back in the days when I did CTFs, I used Python for writing binary exploits, never C.
Swift has the advantage that it can directly “speak” C, without any indirection needed. (C can already do this of course.) In particular operations like scanning an address space for things, easily expressing the layout of something, and so on are much easier to do in these languages. In theory you could do the same in Python but it’s often not worth the effort.
Raw memory access is something you normally need to create vulnerabilities, not to exploit them :)