(Disclaimer: I'm a designer and I'm probably very biased about the matter)
In my opinion it's very often a matter of expectations.
When a lot of design decisions are taken by a bunch of people, from management, engineering and marketing, including a designer would simply mean having one more person to compromise with. Companies working this way tend to have designers focus more on making the product look good, thus this dribbblisation of design. On the other side, engineering has historically been expected to deal with design when building products, and nowadays a lot of engineers have learnt how to design on the field. This reinforces the view that designers are only useful to make things look good. Lastly, the tools we use in design are really easy to use, anyone with a couple hours to spare should be able to create some stuff with most of the available tools. With that, it makes deciders think that the only difference will be a matter of taste.
A lot of designers can actually do more than that, the basis of their job is usually to question things with the perspective of the user: "Does it make sense to do this? Is it really useful to have that amount of customisation? How much information do we need to see?" Most of the time the solutions are stupidly obvious, or they're chosen based of how complicated it makes the system, or how costly it can be, and that's where experts add a lot of value.
All in all, if all you need is to make stuff look good, sure a dribbble designer is probably a good fit, just don't expect them to automagically make the product usable. If you want someone to work on your experience, you should probably have your designer work on the problem instead of having them "upgrade" your solution.
Just like how the engineering field is, the design field is really diverse with people doing different things, differently.
This. I'm not a designer, but my wife is (we met in a design studio I was prototyping for). We were just having this conversation yesterday about how design is "appreciated" in her company, basically going through every point you listed above. It takes a huge cultural change in a company to make something better than "design by engineer" desirable. Many companies want jack of all trade generalists who can code a bit and design a bit, meaning they can be reassigned easily enough to where the work happens to be at the moment. But at the end of the day, it's all mediocre output (though sadly, it is considered acceptable).
And then anyone who plays around with photoshop or illustrator for a few days seems to think they are ready for a visual design career (and interaction design is worse!). Who needs design school? So people encounter so many bad poorly trained and experienced designers that their expectations of the profession begin to be lowered. It is frustrating for those who actually went through hard work to become skilled and experienced UXDs.
I'm personally a PL designer and see the same issues in our field (well, it is very niche, and anyone who can write a compiler wants to design a language....).
Communicating information clearly was a big reason for Swiss style's ultra-clean aesthetic. It's come back in a way as flat design, but designers should heed the advice that looks pretty doesn't equal good design.
In my opinion it's very often a matter of expectations.
When a lot of design decisions are taken by a bunch of people, from management, engineering and marketing, including a designer would simply mean having one more person to compromise with. Companies working this way tend to have designers focus more on making the product look good, thus this dribbblisation of design. On the other side, engineering has historically been expected to deal with design when building products, and nowadays a lot of engineers have learnt how to design on the field. This reinforces the view that designers are only useful to make things look good. Lastly, the tools we use in design are really easy to use, anyone with a couple hours to spare should be able to create some stuff with most of the available tools. With that, it makes deciders think that the only difference will be a matter of taste.
A lot of designers can actually do more than that, the basis of their job is usually to question things with the perspective of the user: "Does it make sense to do this? Is it really useful to have that amount of customisation? How much information do we need to see?" Most of the time the solutions are stupidly obvious, or they're chosen based of how complicated it makes the system, or how costly it can be, and that's where experts add a lot of value.
All in all, if all you need is to make stuff look good, sure a dribbble designer is probably a good fit, just don't expect them to automagically make the product usable. If you want someone to work on your experience, you should probably have your designer work on the problem instead of having them "upgrade" your solution.
Just like how the engineering field is, the design field is really diverse with people doing different things, differently.
(Wow, that was a long rant, sorry about that.)