Remote agent showing up as installed on localhost?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Remote agent showing up as installed on localhost?

oldgluuser
Hi,

I have installed a remote agent on Host2.  Host1 is running the console.  I use the agentctl.sh script on the remote to start the agent.  The log file there has no errors and says the agent started.  The console seems to confirm this as it "sees" the agent too.  However, the console is indicating that the agent is running on 127.0.0.1, not the remote machine.  Anyone have any idea what I did wrong?

Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

oldgluuser
I stopped and restarted the agent, this time using the -c flag followed by the IP address.  This seemed to help as the agent now shows that IP address in the console.  However, I still can not "talk" to that remote agent from the console.  Likewise, the agent-cli on the console machine is unable to talk to either the secure or unsecure ports on the agent.  However, all of the config parameters of the agent are visible in the console and are correct...
Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

frenchyan
Administrator
I would suggest doing the following:

* download a fresh copy of glu but don't run the tutorial (if you run the tutorial, it does configure the agent bundled with glu for the tutorial)
* install this fresh agent on your host and try again. 

In general, the only thing that an agent needs to start is its zookeeper connection:
=> agent boots, determine its ip, determine its name (canonical host name by default)
=> connect to zookeeper to determine its fabric using its name (if you have configured which fabric this agent belongs to in zookeeper which is the recommended way)
=> using its name and fabric, use zookeeper to fetch the keys (for ssl connection)

Note that I am currently working on making production setup much easier and the issue with configuring the agent for the tutorial will go away when I release this new version.

Yan


On Mon, Jun 17, 2013 at 4:03 AM, joeg [via glu] <[hidden email]> wrote:
I stopped and restarted the agent, this time using the -c flag followed by the IP address.  This seemed to help as the agent now shows that IP address in the console.  However, I still can not "talk" to that remote agent from the console.  Likewise, the agent-cli on the console machine is unable to talk to either the secure or unsecure ports on the agent.  However, all of the config parameters of the agent are visible in the console and are correct...



If you reply to this email, your message will be added to the discussion below:
http://glu.977617.n3.nabble.com/Remote-agent-showing-up-as-installed-on-localhost-tp4025819p4025820.html
To start a new topic under glu, email [hidden email]
To unsubscribe from glu, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

oldgluuser
Yes, I am using a fresh copy of Glu 5.0.0 (NOT the tutorial).  I am running zookeeper on 3181 instead of 2181 (which is where I run the tutorial).   The Agent connects to zookeeper just fine, the log file has no exceptions and the console "sees" the agent, but nothing else works - still investigating.  I think it is the ssl configuration - very confusing - the documentation doesn't really say where the various files should be created, and the key definitions in the documentation don't seem to match with the files, so it is quite a guessing game.

Example: the documentation says:

this trustore will be used in the console and the password (AAAAAAA) will be assigned to console.truststorePassword   ("truststore" is mis-spelled many times as trustore, but not all the time)

However, in zookeeper-config/config.properties the only property with truststore is:
    glu.agent.truststorePassword

although this property seems to be in the section for console.truststore......
Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

oldgluuser
One thing that is completely unclear to me are the encrypted passwords that go into the clientConfig of the agent-cli - are those the same passwords that were put into the zookeeper config?   I think they are, but that test is failing for me - complaining that the passwords are bad....

Keystore was tampered with, or password was incorrect
Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

frenchyan
Administrator
In reply to this post by oldgluuser
I do know it is confusing and complicated but this is exactly what I am working on trying to fix right now... Should be a lot easier once I am done.

Yan

Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

oldgluuser
That sounds great - but is there anything I can do in the interim?

I can run without SSL for now, but that doesn't seem to work either!   I edited the config.properties file in zookeeper-config.  There I set glu.agent.sslEnabled=false.   Then I re-ran the setup-zookeeper.sh script for my fabric.  I deleted the agent on the remote host and recopied down the new one.   The agent seems to start just fine and the log even says:


2013/06/17 15:01:55.597 INFO [AgentMain] Started REST service on (NON SECURE) port: 12906


However, the agent-cli still fails:   bin/agent-cli.sh -H --url http://ipAddress:12906/

was the command I used, but it gives the same error with https...

Thanks for any help (can't wait for the updates you are doing!)
Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

frenchyan
Administrator
Yes running without ssl would bypass the headaches of setting it up.

Can you give more details in terms of "not working"? Are you getting an exception? An error message?

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

oldgluuser
When I reset everything I noticed that the regular build comes packaged with, I assume, a working setup.  So all I did was change the zookeeper port to 3181.  Started zookeeper.  Then I ran ./bin/setup-zookeeper.sh with -z and -f.

No errors, no problems.  Using "scp" I copied the entire agent-server down to the remote machine.  I started the agent with the same -z and -f flags.  I also gave it a name via -n.   The agent log file ends with:

2013/06/17 15:35:39.909 INFO [/] installed
2013/06/17 15:35:41.631 INFO [Server] jetty-7.6.5.v20120716
2013/06/17 15:35:41.850 INFO [AbstractConnector] Started SslSocketConnector@0.0.0.0:12906
2013/06/17 15:35:41.857 INFO [AgentMain] Started REST service on (secure) port: 12906
2013/06/17 15:35:41.869 INFO [zkMonitor] Started.
2013/06/17 15:35:41.869 INFO [AgentMain] Agent started.

So I go back to the main (console) server and into the agent-cli directory.  When I run the test with the -s flag I get:


2013/06/17 15:39:12.977 WARN [org.restlet] An error occurred during the communication with the remote HTTP server.
2013/06/17 15:39:14.994 WARN [org.restlet] An error occurred during the communication with the remote HTTP server.
2013/06/17 15:39:16.998 WARN [org.restlet] An error occurred during the communication with the remote HTTP server.
Exception in thread "main" java.util.concurrent.ExecutionException: org.linkedin.glu.agent.rest.client.RecoverableAgentException: Communication Error (1001) - The connector failed to complete the communication with the server


Does that help?
Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

frenchyan
Administrator
Hmmm, did you restart with the base setup including the keys bundled with glu? (I thought you were going with the idea of using no ssl at all...). I am kind of confused.

In order to debug, edit bin/agent-cli.sh and add this:

JVM_DEBUG="-Djavax.net.debug=all"

also edit conf/log4j.xml and change 

    <priority value="info"/>

to

    <priority value="debug"/>

Should generate more output...

Also you should make sure that you can actually talk from the machine where the agent-cli is running back to the agent machine (firewall, etc...). You can use the following (which removes the agent-cli from the equation):

curl -k https://<ipoftheagent>:12906/agent -E console.pem

You can get console.pem from there (https://raw.github.com/pongasoft/glu/master/dev-keys/console.pem). This is assuming that you are using the keys bundled with glu.

Yan






On Mon, Jun 17, 2013 at 10:24 AM, joeg [via glu] <[hidden email]> wrote:
When I reset everything I noticed that the regular build comes packaged with, I assume, a working setup.  So all I did was change the zookeeper port to 3181.  Started zookeeper.  Then I ran ./bin/setup-zookeeper.sh with -z and -f.

No errors, no problems.  Using "scp" I copied the entire agent-server down to the remote machine.  I started the agent with the same -z and -f flags.  I also gave it a name via -n.   The agent log file ends with:

2013/06/17 15:35:39.909 INFO [/] installed
2013/06/17 15:35:41.631 INFO [Server] jetty-7.6.5.v20120716
2013/06/17 15:35:41.850 INFO [AbstractConnector] Started SslSocketConnector@0.0.0.0:12906
2013/06/17 15:35:41.857 INFO [AgentMain] Started REST service on (secure) port: 12906
2013/06/17 15:35:41.869 INFO [zkMonitor] Started.
2013/06/17 15:35:41.869 INFO [AgentMain] Agent started.

So I go back to the main (console) server and into the agent-cli directory.  When I run the test with the -s flag I get:


2013/06/17 15:39:12.977 WARN [org.restlet] An error occurred during the communication with the remote HTTP server.
2013/06/17 15:39:14.994 WARN [org.restlet] An error occurred during the communication with the remote HTTP server.
2013/06/17 15:39:16.998 WARN [org.restlet] An error occurred during the communication with the remote HTTP server.
Exception in thread "main" java.util.concurrent.ExecutionException: org.linkedin.glu.agent.rest.client.RecoverableAgentException: Communication Error (1001) - The connector failed to complete the communication with the server


Does that help?


If you reply to this email, your message will be added to the discussion below:
http://glu.977617.n3.nabble.com/Remote-agent-showing-up-as-installed-on-localhost-tp4025819p4025829.html
To start a new topic under glu, email [hidden email]
To unsubscribe from glu, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

oldgluuser
Sorry about causing any confusion, turning off the SSL was not working for me either, so I tried starting all over from scratch.  Worst of all the underlying error was a crazy mistake on my part!   First and foremost, everything is working perfectly now - thanks for continuing to push me along!

The cause of the problem was something like just plain stupidity on my part!  The remote (agent) machine I am using is a linux virtual machine running via VirtualBox.   So while I knew I had to open the 12906 port via  the "iptables" command, I completely spaced on saving and restarting iptables.

So I did this command:
    sudo iptables -I INPUT -p tcp --dport 12906 -j ACCEPT

but not these two:
    sudo /etc/init.d/iptables save
    sudo /etc/init.d/iptables restart

as soon as I did the latter two everything starting working just fine.   Obviously not a sys admin!
Reply | Threaded
Open this post in threaded view
|

Re: Remote agent showing up as installed on localhost?

frenchyan
Administrator
Glad to see it is working now. Besides the keys generation and configuration (which I know is a big pain...), network connectivity is one of the major issue that people report when setting up glu (and this has nothing to do with glu), this is why I always tell people in the end to try with low level commands (telnet, curl, wget, etc...) and try to exclude glu from the picture as much as possible to diagnose the problem(s).

Yan