Wouldn't that work completely naturally with unified memory? You mmap your assets, refer to that from GPU code, the OS would logically load the mapped data to RAM then push a subset of it to the GPU.