Re: Getting Started - Missing Agents [SOLVED]

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Getting Started - Missing Agents [SOLVED]

joseheitor
This post was updated on .
I have followed the Easy Production Setup instructions with 4 VMs (2x Agents, 1x Zookeeper and 1x Console). Everything appears to have started up correctly and all logs indicate mainly INFO entries and no errors.

I cannot however see any Agents listed in the Console...





The final entry in the Agents' logs indicate that they are connected to Zookeeper, and waiting for 'fabric'.

2016/11/17 12:21:50.132 INFO [ZooKeeper] Initiating client connection, connectString=192.168.1.245:2181 sessionTimeout=5000 watcher=org.linkedin.zookeeper.client.ZKClient$UniqueWatcher@1c39680d 2016/11/17 12:21:50.169 INFO [ClientCnxn] Opening socket connection to server 192.168.1.245/192.168.1.245:2181. Will not attempt to authenticate using SASL (unknown error) 2016/11/17 12:21:50.310 INFO [ClientCnxn] Socket connection established to 192.168.1.245/192.168.1.245:2181, initiating session 2016/11/17 12:21:50.325 INFO [ClientCnxn] Session establishment complete on server 192.168.1.245/192.168.1.245:2181, sessionid = 0x15871c926420001, negotiated timeout = 5000 2016/11/17 12:21:50.455 INFO [FabricTracker] Waiting for fabric @ zookeeper:/org/glu/agents/names/GLU-AGENT-1/fabric

If I run the [optional] command to test the agent (as described in the guide), I get this error in the terminal:

jose@GLU ~/glu-dist/dists/agent-cli/org.linkedin.glu.agent-cli-5.6.1 $ ./bin//agent-cli.sh -s https://192.168.1.247:12906/ -m /{"mountPoints":["/"]} 2016/11/17 12:48:15.201 WARN [org.restlet] An error occurred during the communication with the remote HTTP server. 2016/11/17 12:48:17.311 WARN [org.restlet] An error occurred during the communication with the remote HTTP server. 2016/11/17 12:48:19.391 WARN [org.restlet] An error occurred during the communication with the remote HTTP server. Exception in thread "main" java.util.concurrent.ExecutionException: org.linkedin.glu.agent.rest.client.RecoverableAgentException: Communication Error (1001) - The connector failed to complete the communication with the server at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at java_util_concurrent_Future$get$0.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114) at org.linkedin.glu.groovy.utils.concurrent.FutureTaskExecution.get(FutureTaskExecution.groovy:278) at java_util_concurrent_Future$get.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114) at org.linkedin.glu.agent.cli.ClientMain$_withAgent_closure23.doCall(ClientMain.groovy:77) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:122) at org.linkedin.glu.agent.rest.client.AgentFactoryImpl$_withRemoteAgent_closure2.doCall(AgentFactoryImpl.groovy:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:122) at org.linkedin.glu.agent.rest.client.RestClientFactoryImpl.withRestClient(RestClientFactoryImpl.groovy:85) at org.linkedin.glu.agent.rest.client.RestClientFactory$withRestClient.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:130) at org.linkedin.glu.agent.rest.client.AgentFactoryImpl.withRemoteAgent(AgentFactoryImpl.groovy:55) at org.linkedin.glu.agent.rest.client.AgentFactory$withRemoteAgent.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:130) at org.linkedin.glu.agent.cli.ClientMain.withAgent(ClientMain.groovy:65) at org.linkedin.glu.agent.cli.ClientMain$withAgent.callCurrent(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:163) at org.linkedin.glu.agent.cli.ClientMain.start(ClientMain.groovy:83) at org.linkedin.util.lifecycle.Startable$start.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114) at org.linkedin.glu.agent.cli.ClientMain.main(ClientMain.groovy:477) Caused by: org.linkedin.glu.agent.rest.client.RecoverableAgentException: Communication Error (1001) - The connector failed to complete the communication with the server at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:232) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:244) at org.linkedin.glu.agent.rest.client.AgentRestClient.handleRecoverableError(AgentRestClient.groovy:653) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:179) at org.linkedin.glu.agent.rest.client.AgentRestClient.handleError(AgentRestClient.groovy:626) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:179) at org.linkedin.glu.agent.rest.client.AgentRestClient.handleResponse(AgentRestClient.groovy:599) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:179) at org.linkedin.glu.agent.rest.client.AgentRestClient.handleResponse(AgentRestClient.groovy:574) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171) at org.linkedin.glu.agent.rest.client.AgentRestClient.getFullState(AgentRestClient.groovy:177) at org.linkedin.glu.agent.api.Agent$getFullState.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:122) at org.linkedin.glu.agent.cli.ClientMain$_closure22.doCall(ClientMain.groovy:412) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292) at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1246) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1209) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:807) at org.linkedin.glu.agent.cli.ClientMain.invokeMethod(ClientMain.groovy) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:428) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:367) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171) at org.linkedin.glu.agent.cli.ClientMain$_start_closure24.doCall(ClientMain.groovy:126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:122) at org.linkedin.glu.agent.cli.ClientMain$_withAgent_closure23$_closure29.doCall(ClientMain.groovy:68) at org.linkedin.glu.agent.cli.ClientMain$_withAgent_closure23$_closure29.doCall(ClientMain.groovy) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114) at org.linkedin.glu.groovy.utils.concurrent.FutureTaskExecution$_closure1.doCall(FutureTaskExecution.groovy:213) at org.linkedin.glu.groovy.utils.concurrent.FutureTaskExecution$_closure1.doCall(FutureTaskExecution.groovy) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114) at org.linkedin.glu.groovy.utils.concurrent.GluGroovyConcurrentUtils$1.call(GluGroovyConcurrentUtils.groovy:40) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java_util_concurrent_RunnableFuture$run.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114) at org.linkedin.glu.groovy.utils.concurrent.FutureTaskExecution.run(FutureTaskExecution.groovy:167) at java.lang.Thread.run(Thread.java:745)



By selecting 'Admin' on the console and 'View Agents Fabric' - I found the agents listed there, but unassigned. After assigning them, I could now see them in the Admin page listed by name, alongside the original entries, listed by IP:



In the Agents view - I now see them (listed by name), but with an IP of 127.0.0.1:



And although I can see the agent details (I assume, passed on through Zookeeper)...



I also cannot seem to communicate with the agents from the console:





(I have found a similar post that claimed to solve the same problem by adjusting iptables firewall rules...I have tried this, but it made no difference. BTW: I also cannot communicate between nodes by name - only by IP address)

Any suggestions?
Reply | Threaded
Open this post in threaded view
|

Re: Getting Started - Missing Agents

joseheitor
In an effort to confirm comms to the agent, I ran the following CURL command (aimed at the REST API of the agent):



...it reports a problem with the certificate.

Is this a valid operation to perform such a test?
Reply | Threaded
Open this post in threaded view
|

Re: Getting Started - Missing Agents

frenchyan
Administrator
In reply to this post by joseheitor
Seems that either you did not run step #6 or it did not work (http://pongasoft.github.io/glu/docs/latest/html/easy-production-setup.html#step-6-configuring-zookeeper-z). This is the step that configure the agents based on your meta model and should assign them a fabric.
Reply | Threaded
Open this post in threaded view
|

Re: Getting Started - Missing Agents

joseheitor
I have made some progress...

I started fresh - declared all hostnames and IPs in the hosts files of all the nodes, and in the meta-model script, declared all nodes by name, instead of IP.

The agents' REST API now starts up correctly (which wasn't happening) and the console now displays the agents in the Agents list immediately:



However, their IP (HOSTNAME) is still displayed as: 127.0.0.1 - and I think that I am still unable to communicate with the agents from the console:

Reply | Threaded
Open this post in threaded view
|

Re: Getting Started - Missing Agents

frenchyan
Administrator

‚ÄčI am not sure why the IP is 127.0.0.1 but that is clearly the issue... if you look at the tutorial ( http://pongasoft.github.io/glu/docs/latest/html/tutorial.html#view-the-agent ) you can see that the hostname is not 127.0.0.1 and it shouldn't be.

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Getting Started - Missing Agents [SOLVED]

joseheitor
Eureka! - Problem solved. (I found that all my hosts files had an entry of: 127.0.1.1 'HOSTNAME")