Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

sodul
I have a set of 3 machines freshly created and configured through a bootstrap script (consistent), but I have the third machine that is unable to complete the Install phase.

Most of the time the symptom is that glu.agent.scriptRootDir seems to be null (fails to mkdir null/mountPoint), but now I'm also seeing this error:
java.lang.IncompatibleClassChangeError: the number of constructors during runtime and compile time for java.lang.Exception do not match. Expected 4 but got 5

The java and blu versions are exactly the same on all 3 machines and I'm deploying the exact same app.
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
Can you check the agent log file during the boot sequence? It should display the version of java being used (this is something I added to be able to figure out which java is running). You should also see a line like this:

Running with jdk1.7: installed groovy Exception workaround

Can you confirm?

Thanks
Yan
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

sodul
2013/04/10 09:06:03.557 INFO [ExceptionJdk17Workaround] Running with jdk1.7: installed groovy Exception workaround
2013/04/10 09:06:03.686 INFO [AgentMain] java version "1.7.0_17" | Java(TM) SE Runtime Environment (build 1.7.0_17-b02) | Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
Can you post the exception stack trace that you are getting?

Seems that the workaround is not "always" working? Which OS?

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

sodul
In reply to this post by sodul
Oh it got better now ... I updated my bootstrap script to add GLU_AGENT_TAGS to my pre_master_conf.sh and the old agent process was not quit properly, while the agent-server directory was replaced by a freshly extracted copy. I guess the old agent got confused with the files under agent-server being swapped.

I manually killed the old agent, started a fresh one and deployments are smooth.

I need to improve that bootstrap script to run more than once while playing nice with the agent.
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
Good news! :)

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

sodul
Ugh, it came back. This time after a reboot of the OS so there was no prior running agent in memory:
I've included the full stack trace:
[+] java.lang.IncompatibleClassChangeError: the number of constructors during runtime and compile time for java.lang.Exception do not match. Expected 4 but got 5
at groovy.lang.MetaClassImpl.selectConstructorAndTransformArguments(MetaClassImpl.java:1407)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.selectConstructorAndTransformArguments(ScriptBytecodeAdapter.java:234)
at org.linkedin.groovy.util.rest.RestException.<init>(RestException.groovy:33)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:198)
at org.linkedin.groovy.util.rest.RestException$_fromJSON_closure1.doCall(RestException.groovy:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
at groovy.lang.Closure.call(Closure.java:412)
at groovy.lang.Closure.call(Closure.java:425)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1326)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1298)
at org.codehaus.groovy.runtime.dgm$148.invoke(Native Method)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:82)
at org.linkedin.groovy.util.rest.RestException.fromJSON(RestException.groovy:46)
at org.linkedin.groovy.util.rest.RestException$fromJSON.call(Native Method)
at org.linkedin.glu.agent.rest.client.AgentRestClient.handleError(AgentRestClient.groovy:589)
at org.linkedin.glu.agent.rest.client.AgentRestClient.this$2$handleError(Native Method)
at org.linkedin.glu.agent.rest.client.AgentRestClient$this$2$handleError.callCurrent(Native Method)
at org.linkedin.glu.agent.rest.client.AgentRestClient.handleResponse(AgentRestClient.groovy:553)
at org.linkedin.glu.agent.rest.client.AgentRestClient.this$2$handleResponse(Native Method)
at org.linkedin.glu.agent.rest.client.AgentRestClient$this$2$handleResponse$0.callCurrent(Native Method)
at org.linkedin.glu.agent.rest.client.AgentRestClient.waitForState(AgentRestClient.groovy:213)
at sun.reflect.GeneratedMethodAccessor1779.invoke(Native Method)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.linkedin.util.reflect.ObjectProxyInvocationHandler.invoke(ObjectProxyInvocationHandler.java:56)
at org.linkedin.glu.orchestration.engine.agents.RecoverableAgent.invoke(RecoverableAgent.java:56)
at com.sun.proxy.$Proxy37.waitForState(Native Method)
at org.linkedin.glu.agent.api.Agent$waitForState.call(Native Method)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl$_closure2_closure6.doCall(ActionExecutionFactoryImpl.groovy:127)
at sun.reflect.GeneratedMethodAccessor2165.invoke(Native Method)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl$_withAgent_closure5.doCall(ActionExecutionFactoryImpl.groovy:220)
at sun.reflect.GeneratedMethodAccessor2119.invoke(Native Method)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.linkedin.glu.agent.rest.client.AgentFactoryImpl$_withRemoteAgent_closure2.doCall(AgentFactoryImpl.groovy:64)
at sun.reflect.GeneratedMethodAccessor2111.invoke(Native Method)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.linkedin.glu.agent.rest.client.RestClientFactoryImpl.withRestClient(RestClientFactoryImpl.groovy:85)
at org.linkedin.glu.agent.rest.client.RestClientFactory$withRestClient.call(Native Method)
at org.linkedin.glu.agent.rest.client.AgentFactoryImpl.withRemoteAgent(AgentFactoryImpl.groovy:55)
at org.linkedin.glu.agent.rest.client.AgentFactory$withRemoteAgent.call(Native Method)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl.withAgent(ActionExecutionFactoryImpl.groovy:217)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl.this$2$withAgent(Native Method)
at sun.reflect.GeneratedMethodAccessor956.invoke(Native Method)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl$_closure2.doCall(ActionExecutionFactoryImpl.groovy:101)
at sun.reflect.GeneratedMethodAccessor1751.invoke(Native Method)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:848)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:831)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:164)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeClosure(ScriptBytecodeAdapter.java:570)
at org.linkedin.glu.orchestration.engine.core.action.execution.ClosureActionExecution.call(ClosureActionExecution.groovy:32)
at org.linkedin.glu.orchestration.engine.action.execution.ActionDescriptorStepExecutor.executeLeafStep(ActionDescriptorStepExecutor.java:54)
at org.linkedin.glu.provisioner.plan.impl.StepExecutionContext.executeLeafStep(StepExecutionContext.java:94)
at org.linkedin.glu.provisioner.plan.impl.LeafStepExecutor.doExecute(LeafStepExecutor.java:66)
at org.linkedin.glu.provisioner.plan.impl.AbstractStepExecutor$1.call(AbstractStepExecutor.java:164)
at org.linkedin.glu.provisioner.plan.impl.AbstractStepExecutor$1.call(AbstractStepExecutor.java:145)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

sodul
In reply to this post by frenchyan
This is on Ubuntu 12.04 LTS x86 running as a VM under Xen and with the latest JDK 7 from Oracle.
$ uname -a
Linux shn-us-v-qa-stable-dashboard 3.2.0-40-generic #64-Ubuntu SMP Mon Mar 25 21:22:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
The other machines are fine? This is quite weird... Have you tried to reboot the other machines as well? I may do a little test app to give you for troubleshooting...

Yan


On Wed, Apr 10, 2013 at 1:08 PM, sodul [via glu] <[hidden email]> wrote:
This is on Ubuntu 12.04 LTS x86 running as a VM under Xen and with the latest JDK 7 from Oracle.
$ uname -a
Linux shn-us-v-qa-stable-dashboard 3.2.0-40-generic #64-Ubuntu SMP Mon Mar 25 21:22:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux



If you reply to this email, your message will be added to the discussion below:
http://glu.977617.n3.nabble.com/Agent-flakyness-with-Java-1-7-0-17-and-Glu-4-7-0-tp4025596p4025606.html
To start a new topic under glu, email [hidden email]
To unsubscribe from glu, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
In reply to this post by sodul

Also, I am a little bit confused.. the exception you are seeing is in the console not in the agent. Are you sure that the agent is having a problem? Can you check the log file of the agent vs the log file of the console? 

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

sodul
That's odd, "grep 'IncompatibleClassChangeError' *' is not returning anything in the agent or the console log directories. I just did a couple of redeploys on that agent and everything seems to work fine at the moment.

The console log is showing some traces related to DB connectivity but I do not believe it is related. I'm restarting the console process to see if that

FYI the console is also running JDK 1.7.0_17, and the last JDBC errors are:
2013/04/11 10:17:09.499 ERROR [JDBCExceptionReporter] The last packet successfully received from the server was 65,645,000 milliseconds ago.  The last packet sent successfully to the server was 65,645,000 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
2013/04/11 10:17:09.502 ERROR [JDBCExceptionReporter] Already closed.
2013/04/11 10:17:09.502 ERROR [JDBCExceptionReporter] Already closed.

Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
The fact that you are not finding the IncompatibleClassChangeError in any agent log file (+ the stack trace you provided) clearly shows that it is actually the console triggering the error while trying to rebuild an exception generated on the agent side.

When an exception is generated on the agent side, it gets "serialized" into a json string. On the other side, the console deserializes the json string and create a RestException to contain the stack trace. This is this creation (according to your stack trace) which is triggering the IncompatibleClassChangeError error, hence it is in the console.

It does not explain why it is happening though :(

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

sodul
I'll keep an eye out for this. We have little Glu activity at the moment and do not have much to report yet. Once I have more apps under Glu control I'll let you know how stable things are.
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

sodul
I just experienced that once more. In this case we do have an exception in the python script that was called during the install phase (we log that outside the scope of the groovy script) which returned an non zero exit code, which trickled back to the groovy script and the console.

If this is the issue ... well not a big deal in our case since we do not use the stack trace in the console but rather's the log from our python layer. So, just FYI.

Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
5.0.0 is out :) Compiled with jdk1.7. Workaround removed... If you can upgrade, then the error should be gone :)

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

Laura
Hi Yan,

After I went over the problem on production setup Step 5 with your help on the other thread (http://glu.977617.n3.nabble.com/multi-node-glu-deployment-td2184865.html#a4025678), I proceeded with Step 6 & 7. The agent server seemed to have started successfully with a PID returned from agentctl.sh. So I proceeded to test it with agent-cli.sh. That's when I got the java.lang.IncompatibleClassChangeError Exception (See Log 1 below).

Following the advice of this thread, I went back to the the agent server starting log, and I find java.security.UnrecoverableKeyException (See Log 2 below). Since the agent server actually started and is still running, I wonder what the UnrecoverableKeyException is about and whether the IncompatibleClassChangeError that agent-cli.sh ran into is related.

Appreciate your any help!
Laura


------------------ Log 1--------------------------------
laura@ubuntu-poc-scc-1:/usr/local/thirdparty/org.linkedin.glu.packaging-all-4.6.2$ ./bin/agent-cli.sh -s https://ubuntu-poc-agent:12906
2013/05/02 10:25:08.234 WARN [org.restlet] An error occurred during the communication with the
 remote HTTP server.
2013/05/02 10:25:10.268 WARN [org.restlet] An error occurred during the communication with the
 remote HTTP server.2013/05/02 10:25:12.272 WARN [org.restlet] An error occurred during the communication with the
 remote HTTP server.
Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.IncompatibleClassChangeError: the number of constructors during runtime and compile time for java.lang.Excepti
on do not match. Expected 4 but got 5        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)        at java_util_concurrent_Future$get$0.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:12
0)
        at org.linkedin.glu.groovy.utils.concurrent.FutureTaskExecution.get(FutureTaskExecution.groovy:279)
        at java_util_concurrent_Future$get.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:4
0)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:12
0)        at org.linkedin.glu.agent.cli.ClientMain$_withAgent_closure23.doCall(ClientMain.groovy
:77)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        .....

---------------------Log 2--------------------------
2013/05/02 09:27:42.875 INFO [/] installed
2013/05/02 09:27:43.985 INFO [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.je
tty.util.log) via org.eclipse.jetty.util.log.Slf4jLog
2013/05/02 09:27:43.986 INFO [log] jetty-7.0.1.v200911252013/05/02 09:27:44.170 WARN [log] java.security.UnrecoverableKeyException: Cannot recover key
2013/05/02 09:27:44.170 WARN [log] FAILED HttpsServerHelper$2@0.0.0.0:12906: java.io.IOException: !JsseListener: java.security.UnrecoverableKeyException: Cannot recover key
2013/05/02 09:27:44.170 WARN [log] FAILED org.restlet.ext.jetty.JettyServerHelper$WrappedServer@761a89db: java.io.IOException: !JsseListener: java.security.UnrecoverableKeyException: Canno
t recover key
Exception in thread "main" java.io.IOException: !JsseListener: java.security.UnrecoverableKeyException: Cannot recover key
        at org.eclipse.jetty.server.ssl.SslSocketConnector.newServerSocket(SslSocketConnector.java:506)
        at org.eclipse.jetty.server.bio.SocketConnector.open(SocketConnector.java:75)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:357)
        at org.eclipse.jetty.server.bio.SocketConnector.doStart(SocketConnector.java:147)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.server.Server.doStart(Server.java:239)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.restlet.ext.jetty.JettyServerHelper.start(JettyServerHelper.java:395)
        at org.restlet.Server.start(Server.java:585)
        at org.restlet.Component.startServers(Component.java:600)
        at org.restlet.Component.start(Component.java:527)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:1
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
You cannot use agent-cli 4.6.2 with java 1.7. This is why you are getting the exception in log1.

In regards to log2, it seems that the keystore you are providing to the agent is not valid. Can you confirm first that the one that comes bundled with glu work properly? Then try with the one you generated. Also make sure that you use java 1.7 (when the agent boots it should display which VM it is using). I do know that the steps are not simple and confusing so it is easy to generate an invalid keystore.

Also you should not use 4.7.0 because it contains a critical issue. You should use 4.7.1.

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

Laura
Hi Yan,

Using 4.7.1 got rid of the exception in log 1 !
And the keystore that comes with the bundle work properly. I'll try to generate keys later.

Thank you very much,
Laura
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

Laura
Hi Yan,

Once I have gone through all the steps for production setup (http://pongasoft.github.io/glu/docs/latest/html/production-setup.html), I loaded the tutorial system model and tried to deploy it. In the first step of installing script for /sample/i003, it ran into the java.lang.IncompatibleClassChangeError exception (See log below).

Earlier I got the same exception because I was using agent-cli 4.6.2. But now I am using 4.7.1 all around.

Appreciate any help!
Thank you,
Laura


Deploy - Fabric [fabric1] - SEQUENTIAL
Deploy - Fabric [fabric1] - SEQUENTIAL - 1s
agent1 - /sample/i001 - 1s
Install script for [/sample/i001] on [agent1] - 1s
[+] java.lang.IncompatibleClassChangeError: the number of constructors during runtime and compile time for java.lang.Exception do not match. Expected 4 but got 5
at groovy.lang.MetaClassImpl.selectConstructorAndTransformArguments(MetaClassImpl.java:1407)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.selectConstructorAndTransformArguments(ScriptBytecodeAdapter.java:234)
at org.linkedin.groovy.util.rest.RestException.<init>(RestException.groovy:33)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:198)
at org.linkedin.groovy.util.rest.RestException$_fromJSON_closure1.doCall(RestException.groovy:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
at groovy.lang.Closure.call(Closure.java:412)
at groovy.lang.Closure.call(Closure.java:425)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1326)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1298)
at org.codehaus.groovy.runtime.dgm$148.invoke(Native Method)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:82)
at org.linkedin.groovy.util.rest.RestException.fromJSON(RestException.groovy:46)
at org.linkedin.groovy.util.rest.RestException$fromJSON.call(Native Method)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.linkedin.glu.agent.rest.client.AgentRestClient.handleError(AgentRestClient.groovy:589)
at org.linkedin.glu.agent.rest.client.AgentRestClient.this$2$handleError(Native Method)
at org.linkedin.glu.agent.rest.client.AgentRestClient$this$2$handleError.callCurrent(Native Method)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
at org.linkedin.glu.agent.rest.client.AgentRestClient.handleResponse(AgentRestClient.groovy:553)
at org.linkedin.glu.agent.rest.client.AgentRestClient.this$2$handleResponse(Native Method)
at org.linkedin.glu.agent.rest.client.AgentRestClient$this$2$handleResponse.callCurrent(Native Method)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
at org.linkedin.glu.agent.rest.client.AgentRestClient.installScript(AgentRestClient.groovy:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.linkedin.util.reflect.ObjectProxyInvocationHandler.invoke(ObjectProxyInvocationHandler.java:56)
at org.linkedin.glu.orchestration.engine.agents.RecoverableAgent.invoke(RecoverableAgent.java:56)
at com.sun.proxy.$Proxy37.installScript(Native Method)
at org.linkedin.glu.agent.api.Agent$installScript.call(Native Method)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl$_closure3_closure7.doCall(ActionExecutionFactoryImpl.groovy:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl$_withAgent_closure5.doCall(ActionExecutionFactoryImpl.groovy:220)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:69)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.linkedin.glu.agent.rest.client.AgentFactoryImpl$_withRemoteAgent_closure2.doCall(AgentFactoryImpl.groovy:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.linkedin.glu.agent.rest.client.RestClientFactoryImpl.withRestClient(RestClientFactoryImpl.groovy:85)
at org.linkedin.glu.agent.rest.client.RestClientFactory$withRestClient.call(Native Method)
at org.linkedin.glu.agent.rest.client.AgentFactoryImpl.withRemoteAgent(AgentFactoryImpl.groovy:55)
at org.linkedin.glu.agent.rest.client.AgentFactory$withRemoteAgent.call(Native Method)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl.withAgent(ActionExecutionFactoryImpl.groovy:217)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl.this$2$withAgent(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
at org.linkedin.glu.orchestration.engine.core.action.execution.ActionExecutionFactoryImpl$_closure3.doCall(ActionExecutionFactoryImpl.groovy:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:848)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:831)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:164)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeClosure(ScriptBytecodeAdapter.java:570)
at org.linkedin.glu.orchestration.engine.core.action.execution.ClosureActionExecution.call(ClosureActionExecution.groovy:32)
at org.linkedin.glu.orchestration.engine.action.execution.ActionDescriptorStepExecutor.executeLeafStep(ActionDescriptorStepExecutor.java:54)
at org.linkedin.glu.provisioner.plan.impl.StepExecutionContext.executeLeafStep(StepExecutionContext.java:94)
at org.linkedin.glu.provisioner.plan.impl.LeafStepExecutor.doExecute(LeafStepExecutor.java:66)
at org.linkedin.glu.provisioner.plan.impl.AbstractStepExecutor$1.call(AbstractStepExecutor.java:164)
at org.linkedin.glu.provisioner.plan.impl.AbstractStepExecutor$1.call(AbstractStepExecutor.java:145)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Reply | Threaded
Open this post in threaded view
|

Re: Agent flakyness with Java 1.7.0_17 and Glu 4.7.0

frenchyan
Administrator
Based on the stack trace, the error is not happening in the agent (same issue that Stephane had) but is happening in the console while trying to recreate an agent exception. Can you make sure that the console is indeed running with java1.7?

Please check the console.log file to see which VM is being used (there should be a line in the log file when you started the console that shows which VM is being use). There should also be a line about installing a workaround for jdk1.7. Both in the console log file.

Can you please copy/paste here?

Thanks
Yan
12