Quantcast

Glu Console Extremely Slow

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Glu Console Extremely Slow

rswiatek
    We are on an old version of Glu (4.7.2) and have an extremely large production environment (300+ different applications spread across 3200+ nodes) and our glu console is extremely sluggish for this environment (the dashboard has been known to take up to 2 minutes to load at times).  Is this sluggishness just due to the size of our model or is something else going on here.  We've been trying to track down the problem and found that proxyHandler.unwrapProxy is consistently taking 20 - 25 seconds to execute.

    Is upgrading going to help this situation or are we going to have pare down our production environment into smaller chunks in order to get better performance out of the console?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu Console Extremely Slow

frenchyan
Administrator
Although you are dealing with big numbers, I don't think that 2mns to load the dashboard is expected or normal for this kind of numbers.

* Did you check the GC log files? Sometimes if there is not enough memory you can end up doing way too much GC which totally kills the performance.
* you are using glu 4.7.2, but with which jdk? 1.6. or 1.7?
* when you say proxyHandler.unwrapProxy is taking over 20s to execute, how did you find that out? and do you have a stack trace that shows which part of the code is doing that (I don't believe that this is in glu's code but in groovy code)?
* I do not know if upgrading will help this specific situation since I do not know what the actual problem is, but it should be relatively easy for you to test it out: simply download the latest version of glu, start the console and load your 300+/3200+ model in it and go to the dashboard... 
* if you are willing to send me your big model I can take a look at it (you can send me privately and sanitize the info if you are uncomfortable in sending it)

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu Console Extremely Slow

yvyas
Hi Yan,

1. Yes we have done some investigation of the GC logs and will be making changes to GC options based on output of some diagnostic options. Currently we know that this is one of the factors in the performance degradation

2. we are using jdk 1.7

3. We have found that information using AppDynamics in the production envrionment to investigate performance bottlenecks.
The stack trace is

Thread.run:722 (0ms self time, 26313 ms total time)
 CharacterEncodingFilter.doFilterInternal:88 (0ms self time, 26313 ms total time)
  AbstractFilter.doFilter:63 (0ms self time, 26281 ms total time)
   PageFragmentCachingFilter.doFilter:195 (0ms self time, 26281 ms total time)
    HttpServlet.service:848 (0ms self time, 26281 ms total time)
     HttpServlet.service:735 (0ms self time, 26281 ms total time)
      GrailsDispatcherServlet.doDispatch:306 (0ms self time, 26281 ms total time)
       SimpleControllerHandlerAdapter.handle:48 (0ms self time, 3923 ms total time)
        SimpleGrailsController.handleRequest:72 (0ms self time, 3923 ms total time)
         DashboardController.delta:unknown (0ms self time, 3923 ms total time)
          DashboardController$_closure4.doCall:unknown (0ms self time, 3923 ms total time)
           DashboardController$_closure4.doCall:64 (0ms self time, 3923 ms total time)
            MetaClassImpl.invokeMethod:906 (0ms self time, 3923 ms total time)
             ExpandoMetaClass.invokeMethod:1106 (0ms self time, 3923 ms total time)
              MetaClassImpl.invokeMethod:1027 (0ms self time, 3923 ms total time)
               MetaClassImpl.invokeMethod:906 (0ms self time, 3923 ms total time)
                ExpandoMetaClass.invokeMethod:1106 (0ms self time, 3923 ms total time)
                 MetaClassImpl.invokeMethod:1082 (0ms self time, 3923 ms total time)
                  MetaMethod.doMethodInvoke:233 (0ms self time, 3923 ms total time)
                   DashboardController.this$3$doComputeDelta:unknown (0ms self time, 3923 ms total time)
                    DashboardController.doComputeDelta:185 (0ms self time, 3923 ms total time)
                     DeltaServiceImpl.computeCustomGroupByDelta:593 (0ms self time, 3923 ms total time)
                      DeltaServiceImpl$computeCustomGroupByDelta.callCurrent:unknown (0ms self time, 3013 ms total time)
                       DeltaServiceImpl.computeCustomGroupByDelta:402 (0ms self time, 3013 ms total time)
                        DeltaServiceImpl$computeDelta.callCurrent:unknown (0ms self time, 2645 ms total time)
                         DeltaServiceImpl.computeDelta:192 (52ms self time, 2645 ms total time)
                          DeltaMgrImpl.computeDelta:87 (108ms self time, 2593 ms total time)
                           Closure.call:412 (0ms self time, 731 ms total time)
                            MetaClassImpl.invokeMethod:906 (0ms self time, 731 ms total time)
                             ExpandoMetaClass.invokeMethod:1106 (0ms self time, 731 ms total time)
                              MetaClassImpl.invokeMethod:1082 (0ms self time, 731 ms total time)
                               MetaMethod.doMethodInvoke:233 (0ms self time, 731 ms total time)
                                Closure.call:412 (0ms self time, 679 ms total time)
                                 MetaClassImpl.invokeMethod:906 (0ms self time, 679 ms total time)
                                  ExpandoMetaClass.invokeMethod:1106 (0ms self time, 679 ms total time)
                                   MetaClassImpl.invokeMethod:1082 (0ms self time, 679 ms total time)
                                    MetaMethod.doMethodInvoke:233 (53ms self time, 679 ms total time)
                                     Closure.call:412 (0ms self time, 433 ms total time)
                                      MetaClassImpl.invokeMethod:906 (0ms self time, 433 ms total time)
                                       ExpandoMetaClass.invokeMethod:1106 (0ms self time, 433 ms total time)
                                        MetaClassImpl.invokeMethod:1082 (0ms self time, 433 ms total time)
                                         MetaMethod.doMethodInvoke:233 (0ms self time, 433 ms total time)
                                          Reference.<init>:35 (0ms self time, 433 ms total time)
                                           GroovyObjectSupport.<init>:32 (433ms self time, 433 ms total time)


Regarding the model, we will get back to you soon, early next week. Thanks for your help!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu Console Extremely Slow

frenchyan
Administrator
Can you explain how to read this kind of stack trace? I am not familiar with AppDynamics nor what it reports and how to interpret it. Also in your previous post you talked about unwrapProxy but I don't seem to see it in the stack trace.

Since you are already using jdk1.7, I am not entirely convinced that upgrading to glu 5.x would help as it is using the same version of groovy and grails. The only real big difference is that glu 4.7.x is compiled with java 6 but can run with java 7 and glu 5.x is compiled directly with java 7 (and does not run with java 6), but it should not really have any impact in performance. But if you can still give it a try in an isolated environment it would definitely be a valid data point.

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu Console Extremely Slow

frenchyan
Administrator
In reply to this post by yvyas
Also at this stage, glu does not use any of the new features that come with groovy 2.0 mainly


both features should (theoretically) make glu more responsive.

That being said it requires to be turned on during compilation and also requires java 1.7 which means it could not be back-ported to the 4.7.x line.

Still at this stage, I would like to have a reproducible case otherwise I will have no clue what is the actual problem nor if any changes I am making is having a positive (or negative) impact.

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu Console Extremely Slow

sodul
In reply to this post by frenchyan
AppDynamics is a java tool that tracks all your function call and how long they each take. Think traceroute but for java stack.

I don't recall what the first number means but here:
  DeltaMgrImpl.computeDelta:87 (108ms self time, 2593 ms total time)

It means DeltaMgrImpl.computeDelta took 2.6s to run with 108ms in the function itself (vs what it is calling).

What is interesting is to look where the times are jumping:
  DeltaServiceImpl.computeCustomGroupByDelta:402 (0ms self time, 3013 ms total time)
    DeltaServiceImpl$computeDelta.callCurrent:unknown (0ms self time, 2645 ms total time)

Then:
  DeltaServiceImpl.computeCustomGroupByDelta:593 (0ms self time, 3923 ms total time)
    DeltaServiceImpl$computeCustomGroupByDelta.callCurrent:unknown (0ms self time, 3013 ms total time)

And mostly the biggest one 4s to 26s:
  GrailsDispatcherServlet.doDispatch:306 (0ms self time, 26281 ms total time)
    SimpleControllerHandlerAdapter.handle:48 (0ms self time, 3923 ms total time)


rswiatek, how many zookeeper instances do you use for the cluster of your glu fabric? 3 to 5 are ideal. If you use more, things will get much much slower due to the nature of zookeeper.
Loading...