As long as you've put in enough consideration to avoid painting yourself into a corner with your system architecture so that you know how you'd scale the system should the need arise, you don't need to invest the engineering time up-front to actually build it. However, if you don't understand how your application is going to scale then you might be totally screwed by bad architectural decisions.
For example, you might assume that the data for all concurrent users fits within RAM on a single server. It's one thing to plan for an 8GB server for 50K concurrent load and then have to scale the RAM to 16GB, 32GB, 64GB, 128GB, and so on. But if your RAM usage is O(n^2) for n concurrent users and both your concurrent user count and the average data stored per user increases such that you need hundreds of terabytes of RAM on a single server in order to continue to function, maybe putting in a modicum of up-front work would have paid off.
I'm not in any way saying that you shouldn't plan for the future. We certainly have designed everything wherever possible not to get in the way of future scaling. It's just the problems you can't predict are what really get you.
For example, in theory the first "designed in" limitation that our architecture has should kick in around 1.5 million concurrents. There is a service in the backend that, due to the way it is designed, could never possibly handle more concurrents than that without redesigning some pretty core internals.
In reality though, when we get to 2 orders of magnitude less concurrents then that the realm starts to develop issues that we are currently resolving.
So yes, don't design things that can't scale, but don't expect your designs to actually scale without serious engineering effort.
For example, you might assume that the data for all concurrent users fits within RAM on a single server. It's one thing to plan for an 8GB server for 50K concurrent load and then have to scale the RAM to 16GB, 32GB, 64GB, 128GB, and so on. But if your RAM usage is O(n^2) for n concurrent users and both your concurrent user count and the average data stored per user increases such that you need hundreds of terabytes of RAM on a single server in order to continue to function, maybe putting in a modicum of up-front work would have paid off.