Console would not start when using new keys.

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Console would not start when using new keys.

elutfallah
I spent a day going over the production doc over and over thinking I missed some crucial step, regenerating and encrypting keys thinking I made a typo, pouring over logs, doing fresh installs of a full glu env on fresh VMs, digging into zookeeper data by hand, and pulling out my hair.

In the end, I managed to figure out that there was one file keeping my console from starting.

console-server/keys/console.secretkeystore

Removing it fixed the issue. Is that expected? If so, should that be documented in the production doc for people changing their keys? Is there a way to migrate that file using the new keys, or will people need to recreate any encryption keys they made in the console by hand?

Thanks,

Eli
Reply | Threaded
Open this post in threaded view
|

Re: Console would not start when using new keys.

elutfallah
The other issue I was running into at the same time is that the agents were holding on to their old passwords even when restarted. I ran into this same issue when changing the hostname of an agent. I had to remove the agent.properties and let the agent regenerate the file - I suppose I could have edited the file as well. Regeneration was easier.

./bin/agent-server.sh stop
rm agent-server/data/config/agent.properties
./bin/agent-server.sh start

Is there a way to tell the agent to regenerate its properties without having to log on to the boxes? Does the API support that? Of course, you'd need to be able to connect to the agent to use the API, so you'd have to have an agent-cli configured with the old keys to tell the agents to switch to the new keys.

Anyway, I would add the above to the production docs. If you have never started the agent before, the doc is fine. If you are changing keys for a fabric you already have running, this is an essential step.

Thanks,

Eli
Reply | Threaded
Open this post in threaded view
|

Re: Console would not start when using new keys.

frenchyan
Administrator
The issue with the agent is clearly a bug. The purpose of the file data/config/agent.properties is to serve as a "cache" so that if you provide a value once (ex: you start the agent by giving an explicit fabric: -f foo) it is remembered when you start the agent again without providing it. Clearly if you provide it, it should pick it first. I create a ticket for this: http://glu.977617.n3.nabble.com/Console-would-not-start-when-using-new-keys-td3380251.html

In regards to the console it sounds like a bug as well. Can you describe a little more what you did exactly and what you mean by: "the console did not start". Are you getting an exception?

Thanks
Yan
Reply | Threaded
Open this post in threaded view
|

Re: Console would not start when using new keys.

elutfallah
With the agent, I saw the same thing when changing the hostname. The hostname of the server changed, and the properties file still used the old hostname. I had to change it there (actually I deleted the file) after the hostname change to get the agent to work properly.

With the console, the default tar - in 3.3.0 - includes the console.secretkeystore in the keys directory. When you generate and install new keys, this file is still using the old keys so you get this error in the logs:
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
And the console in the browser says 'service unavailable'. Once I delete the secretkey file, everything works fine and at some point the console generates a new secretkey file.

Is that enough to go on?

Thanks,

Eli
Reply | Threaded
Open this post in threaded view
|

Re: Console would not start when using new keys.

frenchyan
Administrator
I think this should be fine. I have created a ticket for it: https://github.com/linkedin/glu/issues/101

Thanks for reporting and sorry you wasted so much time on this issue.

Best
Yan