Urgent!!! Need help

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Urgent!!! Need help

zedfauji
After upgrading from 4.6.2 -> 4.7.2 , JDK7. console is giving this error :- An unexpected error has occured.

stacktrace :-

2013/10/16 03:14:51.472 ERROR [GrailsExceptionResolver] MissingPropertyException occurred when processing request: [GET] /console/agents/view/at-sigiri4.pv.at.nextag.com
No such property: parent for class: java.lang.String. Stacktrace follows:
groovy.lang.MissingPropertyException: No such property: parent for class: java.lang.String
        at org.linkedin.glu.agent.tracker.MountPointInfo.getParent(MountPointInfo.groovy:35)
        at org.linkedin.glu.orchestration.engine.agents.AgentsServiceImpl.createSystemEntry(AgentsServiceImpl.groovy:316)
        at org.linkedin.glu.orchestration.engine.agents.AgentsServiceImpl$_getCurrentSystemModel_closure12_closure22.doCall(AgentsServiceImpl.groovy:243)
        at org.linkedin.glu.orchestration.engine.agents.AgentsServiceImpl$_getCurrentSystemModel_closure12.doCall(AgentsServiceImpl.groovy:242)
        at org.linkedin.glu.orchestration.engine.agents.AgentsServiceImpl.getCurrentSystemModel(AgentsServiceImpl.groovy:237)
        at org.linkedin.glu.orchestration.engine.delta.DeltaServiceImpl.computeRawDelta(DeltaServiceImpl.groovy:181)
        at org.linkedin.glu.console.controllers.AgentsController$_closure6.doCall(AgentsController.groovy:182)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1448)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:276)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:103)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.apache.shiro.grails.SavedRequestFilter.doFilter(SavedRequestFilter.java:55)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:368)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:722)
-=Zed=- a.k.a -=Fauji=- More quieter you are , more you can hear
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

sodul
Yan might be able to answer. But can you specify: OS, java version, and are you sure you updated both JAVA_HOME and PATH so that the correct version of is used?
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

zedfauji

Os:- CentOS JDK7
And yes i have updated JAVA_HOME and path too

On Oct 16, 2013 9:36 PM, "sodul [via glu]" <[hidden email]> wrote:
Yan might be able to answer. But can you specify: OS, java version, and are you sure you updated both JAVA_HOME and PATH so that the correct version of is used?


If you reply to this email, your message will be added to the discussion below:
http://glu.977617.n3.nabble.com/Urgent-Need-help-tp4026195p4026197.html
To unsubscribe from Urgent!!! Need help, click here.
NAML
-=Zed=- a.k.a -=Fauji=- More quieter you are , more you can hear
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

sodul
Can you include the output of "java -version" so we get the exact version you use.

I looked at older posts in the forum and could not find anything matching what you are seeing.

Have you tried the 5.3.1 release? The directory structure has changed but can be recreated by running bin/setup-pre510.sh. This is what I use until I get time to get familiarized with the new setup procedure.

I've ran 4.7x agents with 5.x consoles for several months so there should not be any issue with upgrading the console to the latest version.
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

sodul
In reply to this post by zedfauji
I took a look at the source code and it seems that your stacktrace is happening on the agent, correct?

From what I can tell, you seem to either have an invalid groovy script and/or json model with regards to the 'parent' feature. It is possible that it used to not throw an exception with 4.6.2 but 4.7.2 is more picky about it.

If you could post the groovy script and your model I might be able to tell you what is wrong.

My best guess is that your json model does not provide a parent but your groovy script tries to reference it. It might also be that  your script is trying to access parent without checking if it is null.

http://pongasoft.github.io/glu/docs/latest/html/static-model.html
http://pongasoft.github.io/glu/docs/latest/html/agent.html#agent-capabilities-parent
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

frenchyan
Administrator
The stack trace shows that:

    return MountPoint.create(data?.scriptDefinition?.parent)

is failing. According to the error message, it seems that data.scriptDefinition is a String and I do not understand why as it should not be a String (but a map). This information is coming from ZooKeeper.

Can you run the following command (using the console-cli)?

./bin/console-cli.sh -f <fabric> -u <username> -x <password> -b -l status

(replace <fabric>, <username> and <password> by your own values).

Is this working? Can you post it here? (not sure if it contains private data...) 

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

zedfauji
Thanks Yan ,

Cant put live model status over here as it contain some sensitive data.  
-=Zed=- a.k.a -=Fauji=- More quieter you are , more you can hear
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

sodul
maybe you can anonymise the model by replacing your app names and version with generic names such as foo and bar. Be consistent when you replace the names so that we can get a proper idea of what is going on.
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

Nicholas Jasieniecki
We had this happen to us too when we updated from 4.3.1 to 4.7.2 on some box. Due to deployment system, we dont know which box (so we cant provide logs or anything), but after down grading back to 4.3.1 console service was restored, but large numbers of agents were not reporting correctly to the ZKs anymore. An upgrade to 4.7.1 restored service entirely.

At no point did we clear the glu-agent cache, just manipulated version numbers.

When we later upgraded everything from 4.7.1 to 4.7.2 everything went fine.
Reply | Threaded
Open this post in threaded view
|

Re: Urgent!!! Need help

frenchyan
Administrator
My understanding of the bug makes me believe that it happened because during the upgrade:

1) the agents stops and restarts.
2) When it restarts it needs to re-instantiate the glu script.
3) Prior to 4.6.2, the glu script was not stored locally and was being fetched from its original location (as defined in the glu model).
4) If the original location is not accessible, then glu cannot re-instantiate the glu script and simply ignores this entry (the agent itself is fine).
5) after booting, the agent synchronizes the filesystem with ZooKeeper (Syncing filesystem <=> ZooKeeper message in the agent log)
6) this step (was) blindly loading all the states from the filesystem (which are java serialized objects) and storing them in ZooKeeper as json object
7) the issue is that in 4.7.1 the format of the file has changed and so because of 4) and 6) you end up with the wrong format in ZooKeeper for those states that were ignored in 4)
8) the console then receives this invalid data and then fails

What I did to fix the issue:
1) during the boot process, the agent will upgrade old format to new format
2) if a state cannot be restored, it is moved to a separate location and a "dummy" InvalidStateScript is instantiated so that it will appear in the console with the proper stack trace so that you can identify what the problem is
3) on the console side, it no longer fails if one entry cannot be read but will also instantiate a dummy one so that it is not silent

Technically the console should not see this problem because of the fix in the agent, but if you use the new console with an old agent, then at least the console will "survive" and continue to be operable.

Both 4.7.3 and 5.5.1 have those fixes.

Yan