I think it's important to realize that the reduced isolation of containers can also have pretty significant upsides.
For example monitoring the host and all running containers and all future containers only means running one extra (privileged) container on each host. I don't need to modify the host itself, or any of the other containers, and no matter who builds the containers my monitoring will always work the same.
The same goes for logging. Mainly there is an agreed-upon standard that containers should just log to stdout/stderr, which makes it very flexible to process the logs however you want on the host. But also if your application uses a log file somewhere inside the container, I can start another container (often called "sidecar") with my tools that can have access to that file and pipe it into my logging infrastructure.
If I want multiple containers can share the same network namespace. So I listen on "localhost:8080" in one container, and connect to "localhost:8080" in another, and that just works without any overhead. I can share socket files just the same.
I can run one (privileged) container on each host that starts more containers and bootstraps f.e. a whole kubernetes cluster with many more components.
You can save yourself much "infrastructure" stuff with containers, because the host provides them or they are done conceptually different. For example ntp, ssh, cron, syslog, monitoring, configuration management, security updates, dhcp/dns, network access to internal or external services like package repositories.
My main point is that by embracing what containers are and using that to your advantage, you gain much more than by just viewing them as lightweight virtualisation with lower overhead and a nicer image distribution.
Edit: I want to add that not all of that is necessarily exclusive to containers or mandatory. For example throwing away the whole VM and booting a new one for rolling updates is done a lot, but with containers it became a very integral and universally accepted standard workflow and way of thinking, and you will get looked at funny if you DON'T do it that way.
For example monitoring the host and all running containers and all future containers only means running one extra (privileged) container on each host. I don't need to modify the host itself, or any of the other containers, and no matter who builds the containers my monitoring will always work the same.
The same goes for logging. Mainly there is an agreed-upon standard that containers should just log to stdout/stderr, which makes it very flexible to process the logs however you want on the host. But also if your application uses a log file somewhere inside the container, I can start another container (often called "sidecar") with my tools that can have access to that file and pipe it into my logging infrastructure.
If I want multiple containers can share the same network namespace. So I listen on "localhost:8080" in one container, and connect to "localhost:8080" in another, and that just works without any overhead. I can share socket files just the same.
I can run one (privileged) container on each host that starts more containers and bootstraps f.e. a whole kubernetes cluster with many more components.
You can save yourself much "infrastructure" stuff with containers, because the host provides them or they are done conceptually different. For example ntp, ssh, cron, syslog, monitoring, configuration management, security updates, dhcp/dns, network access to internal or external services like package repositories.
My main point is that by embracing what containers are and using that to your advantage, you gain much more than by just viewing them as lightweight virtualisation with lower overhead and a nicer image distribution.
Edit: I want to add that not all of that is necessarily exclusive to containers or mandatory. For example throwing away the whole VM and booting a new one for rolling updates is done a lot, but with containers it became a very integral and universally accepted standard workflow and way of thinking, and you will get looked at funny if you DON'T do it that way.