For myself, drawing never required having it visualized in my mind. I knew what I wanted to draw and could put it on paper. However not without difficulty. Using a model, or image to reference certainly helped.

As a simple example. Do you picture the letters you are going to write before writing a note? I don't know the mechanic, but I can write notes without effort. In the same vein, simple objects like geometric shapes are very easy for me to draw, no visualization needed. I suspect it's the same for most.

However, I did give up on drawing because I recognized I couldn't visualize a scene and place it on paper easily. It felt like I was drawing with poor planning which resulted in lots of mistakes, corrections and ultimately frustration.

However, I'm not sure if I qualify as someone with true aphantasia or not. I can visualize things, but I can't hold them in my mind. I see it briefly, like a flash, but can't hold onto it. I've tried to practice holding an image in my mind without success.