Mmap() is not the only obvious solution for unused assets. There are other ways with more explicit operational specs. If you design the solution for this problem from first principles, you'll come up with something more appropriate than mmap()
Sure you can figure out if a texture is used in the current visible set (like in a certain part of a game level). But then it starts to get tricky!
Any concrete ideas how to determine which parts (= memory pages) of a texture are actually needed to draw the scene?
If you don't know, you're going to waste a lot of I/O and memory capacity for something you don't need in the first place.
Remember a texture... :
1) ... has multiple mipmap [1] levels. Say you have 1024x1024 texture. You'll also need 512x512, 256x256, 128x128... etc versions. Depending on the triangle orientation [2], GPU might need some spatial areas (not all!) from any of those levels. 1024x1024 for the corner that's near camera and 128x128 for the faraway portion.
2) ... has spatial data order. It's not in row-major order (think scanlines), but in some space filling curve order (like Z-order curve [3]). This helps caching AND paging schemes -- other accesses are spatial and very likely to be nearby memory addresses in the texture.
3) ... is rarely drawn alone. There are often multiple objects using same texture. Sometimes that's true even if actual visible portion is completely different, like texture atlases [4]. This makes it non-trivial to consider points 1 and 2.
4) ... is sometimes huge. Uncompressed 4096x4096 float32 RGBA texture takes 256 MB (4096 * 4096 * 4 * 4) memory just for the first mipmap level. All (traditional "pyramid" type) mipmap levels together would take about 341 MB.
So how are you going to determine which memory ranges of the texture really need to be in the memory?
It doesn't have to be this complicated. It only seems complicated because your solutions are stuck in the context of memory mapping.
A very simple solution is to add another logic layer to GPU processing, like a shader. It could be the "asset shader," it would run on each frame and tells the GPU what assets to start preloading, etc. That would give the programmer tight control over asset loading latencies without having to load everything at once.
> and tells the GPU what assets to start preloading, etc.
How does that shader gain information about which portions of the texture are needed at each mipmap level?
Or do you just load whole texture and consume memory you don't actually need to render the image. It'd perform badly due to unnecessary I/O causing a long loading time. You'd also waste large portions of GPU RAM.
Or does your shader try to guess? Do you attempt to reverse engineer exactly how GPU trilinear texture sampler operates, because otherwise you won't know which parts of asset data is needed -- guess wrong and you get weird artefacts, when GPU samples your texture at a memory location you didn't load. Oops. Rounding differences compared to hardware texture sampler would almost certainly get you. Not sure if it's still true, but at least in past different brand/model GPUs implemented texture sampling slightly differently [1], enough to force you to have a version for many GPU vendors and models.
Or do you disable trilinear sampling and use just one mipmap level you somehow pick. You'd get bad image quality, blur and/or aliasing (like moire).
Even after considering all that, how are you going to deal with texture atlases?
Your way sounds really complicated. Unless you're willing to do rather serious compromises. Robustness, quality or loading time performance.
The "shader" would take in the current application-specific description of the scene.
This description in theory would be much much much smaller than the corresponding assets required to render the scene. It would be trivial to eagerly bind to the shader.
The asset shader API could provide information on the current GPU's texture rendering parameters/quirks.
