> They introduced X-Pack by bundling it with the default distribution as a time-limited trial without explicitly stating that it is just a demo.

I assume you're referring to the Elastic docker images, as I believe that's the only place where we've ever bundled X-Pack without any explicit opt-in (Our windows installer also includes X-Pack, but it's clearly marked as an optional & commercial component).

We totally underestimated the confusion and difficulties that would cause, and it was fixed with the 6.0 release.

The difficult was balancing the different needs of different users, with the constraints of how docker containers are typically managed. X-Pack requires a file-system level install, which is not something that docker users expect or want - an image should be built once and then be essentially immutable. No one likes to have to enter the container in order to install a new plugin.

Since it was possible to disable X-Pack functionality, or install a free license for a subset of the features, it seemed like shipping the container with X-Pack pre-installed and letting users dial it back as needed, was the better option compare with shipping without X-Pack and forcing customers to reconfigure their container so that they could get the features that they needed.

We didn't expect that there would be so many users for whom Docker was the primary/initial point of contact with our stack. We believed that we would be mostly working with users who already understood what X-Pack was and what they were getting. When it became clear that that wasn't true, we had to come up with a solution, which is what we shipped in the 6.0 release. We didn't want to change the behaviour in a minor release and cause more confusion for users who were relying on X-Pack being installed, so it wasn't simply a case of changing the way the images.

The X-Pack licensing code was built on the premise that X-Pack was a plugin the was explicitly installed so that those who installed it would know what was going on. And when it was written, that was true. One of the consequences of that assumption was that it would automatically generate a trial license on start-up, and then you could install your own purchased license after the fact. In order to offer Docker containers that worked the way users would expect, we had to make changes to that so that X-Pack could be installed, but default to only enabling the free features, so since 6.0 we are now able to provide 3 different docker "flavours" - pure open source, basic (free license), platinum (trial license for paid features).

We do want people to use X-Pack. We believe that the free (basic license) features offer something useful that users should know about, and the success of the company relies on users knowing that our commercial features exist, and being able to evaluate them and decide if that's something they want to purchase. But the docker situation was never intended to be a "bait-and-switch", it was just a problem that caught us by surprise and took some time to rectify.

For historical accuracy, the inclusion of X-Pack in our docker images was not the introduction of X-Pack, it had been available for quite a long time before that and the underlying commercial IP was several years old before we started publishing Docker images.

I personally was running an instance on FreeBSD and just went with the latest port after a quick glance at the docs. There was a new "lite" (or similar) port that did not include xpack, but after skimming the docs I saw no reason not to use these new features (even though in my setup they don't really provide any additional security). A few weeks later it all stopped working, some docs were broken (I checked against the source) and then I saved the rest of my weekend by simply rolling back to the previous setup.

To sum it up: My decision to move from Solr to ES due to much nicer configuration backfired and I effectively lost time compared to just using Solr (which I know but despise). But some early ES blog post (IIRC) indicated that the ridiculous setup of Solr was the initial motivation to write ES in first place, so it is a sad twist that ES messed up exactly that :(

There either has to be some fingerprinting and a phone home, or a licence you "manually" set to the env.

Since I am unaware of any sufficiently reliable host/cluster fingerprinting opportunity in say Kube, I assume it needs a licence to be configured in the pod yaml file.

What I don't understand is how this confusion can arise given the explicit licence installation.

Or where did I go wrong?

Elasticsearch is stateful, it's essentially useless if you don't have some form of persistent storage for it, because its purpose is to act as a datastore. So we make no attempts to treat the Elasticsearch containers as if they're stateless in the "no permanent storage" sense, but you can point your Elasticsearch data directory to a mounted volume and keep that state away from the core image.

But having an immutable image is a bigger deal. The expectation that users have of docker images is that you don't need to edit anything on the image files themselves, but that the configuration is provided from an external source, typically through environment variables.

Installing X-Pack is an image change. It stores new files in the plugins, bin, and config directories of Elasticsearch, and will typically require changes to your main cofiguration file. That's something docker users generally don't want.

Installing a license is a state change. The license is loaded with an HTTP API, and we store that data in the data directory alongside the stored documents. So for most users it's not a violation of their "immutable" expectations.

The other expectation is that you can simply start the container, and run it to completion and then terminate it - you don't want a process that expects to be started, and then reconfigured, and then restarted, etc. That affects other aspects of how you can configure Elasticsearch, but doesn't impact the licensing issue quite so much.

Given that the process was automagic, I can see the potential for confusion.

Seems you are handling the fallout nicely though.