At my last job we used a micro-service architecture on AWS (EC2 and RDS). Using Ansible playbooks for various types of servers and roles for each service, we created a new server instance every deploy. All servers were running FreeBSD and using daemontools to control services. For testing, hotfixes, and manual checking of logs, it was easy to complement with manual ssh. Save old and new instance in case something goes wrong. Ansible is just a thin layer on top of shell scripts, and reasonably straightforward to understand and parameterize. Worked wonderful in most cases (possible exception of build server because of shared libraries and a complex workflow with git pull/trigger, but I don't think that was the fault of the overall architecture).
That said, I agree that sbt is an abomination and doesn't lend itself to a sane and secure workflow, unfortunately.
I also learned from a place with good practices, which used daemontools to run services and a custom deploy system in bash and python which actually did the right kinds of things. (and I was fully-manually admin-ing my own linux systems for years before that)
As an early employee at a new place, I'm now using ansible and docker because nowadays people want to use that stuff, and it is a lot faster to get started with than writing a new proper deploy system from scratch.
But I build all the docker images we use and version them with the date. I also don't use ansible roles from ansible-galaxy, and I don't even organize tasks into roles, just into task-include files. Our ansible tasks use bash helper scripts wherever necessary to do things the right way, because the built-in modules are often too granular / not connected enough to fully check state. I also replaced the docker plugin for ansible, to manage state a bit better. So overall it's not too bad.
I guess my point is that, having done it all from-scratch first, using some of the modern automation stuff isn't too bad. But you have to know what not to use. People new to "devops", using all the fancy stuff now available, who didn't have the introduction I did... it's not surprising they end up in a mess and don't even recognize it.
This is the key. The OP's major complaint is with prebuilt containers from potentially untrustworthy sources, but he passes this off as a fundamental problem with containers themselves.
The reality is that you can (and probably should) build your own container rather than using a public one from docker hub. You know exactly what is in it, and can trust it completely.
in reality a dev will pass a prebuilt and non updatable container to the sysadmin tho.
so the op is exactly right! it doesnt matter where its coming from if you cant verify,rebuild or update it.
That said, I agree that sbt is an abomination and doesn't lend itself to a sane and secure workflow, unfortunately.
http://martinfowler.com/bliki/PhoenixServer.html
http://www.ansible.com/