I spent a day going over the production doc over and over thinking I missed some crucial step, regenerating and encrypting keys thinking I made a typo, pouring over logs, doing fresh installs of a full glu env on fresh VMs, digging into zookeeper data by hand, and pulling out my hair.
In the end, I managed to figure out that there was one file keeping my console from starting.
Removing it fixed the issue. Is that expected? If so, should that be documented in the production doc for people changing their keys? Is there a way to migrate that file using the new keys, or will people need to recreate any encryption keys they made in the console by hand?
The other issue I was running into at the same time is that the agents were holding on to their old passwords even when restarted. I ran into this same issue when changing the hostname of an agent. I had to remove the agent.properties and let the agent regenerate the file - I suppose I could have edited the file as well. Regeneration was easier.
Is there a way to tell the agent to regenerate its properties without having to log on to the boxes? Does the API support that? Of course, you'd need to be able to connect to the agent to use the API, so you'd have to have an agent-cli configured with the old keys to tell the agents to switch to the new keys.
Anyway, I would add the above to the production docs. If you have never started the agent before, the doc is fine. If you are changing keys for a fabric you already have running, this is an essential step.
The issue with the agent is clearly a bug. The purpose of the file data/config/agent.properties is to serve as a "cache" so that if you provide a value once (ex: you start the agent by giving an explicit fabric: -f foo) it is remembered when you start the agent again without providing it. Clearly if you provide it, it should pick it first. I create a ticket for this: http://glu.977617.n3.nabble.com/Console-would-not-start-when-using-new-keys-td3380251.html
In regards to the console it sounds like a bug as well. Can you describe a little more what you did exactly and what you mean by: "the console did not start". Are you getting an exception?
With the agent, I saw the same thing when changing the hostname. The hostname of the server changed, and the properties file still used the old hostname. I had to change it there (actually I deleted the file) after the hostname change to get the agent to work properly.
With the console, the default tar - in 3.3.0 - includes the console.secretkeystore in the keys directory. When you generate and install new keys, this file is still using the old keys so you get this error in the logs:
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
And the console in the browser says 'service unavailable'. Once I delete the secretkey file, everything works fine and at some point the console generates a new secretkey file.