Quantcast

Instance variable 'serverRoot' value does not retain on agent start

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Instance variable 'serverRoot' value does not retain on agent start

BhagatSingh
The  'serverRoot' variable value does not retain on agent start. Here are the steps which I am executing:
1- Deploy application from glu console on an agent (Agent is on Windows in my case). Application is in running state.
2- Kill glu agent where application has been deployed(Windows agent)
3- Application is still running ( checked in Windows process explorer)
4- Start glu agent. Console is showing application is running on Windows agent but when I click on stop/bounce/un-deploy etc then I get exception because serverRoot is null.

Yan:
Which class loads groovy script on agent start-up or where can I debug this to find out what is going wrong in my case?

Please assist.

Groovy Script which I am using:
import org.linkedin.glu.agent.api.ShellExecException

class SampleGluScript
{
  def version = '@script.version@'
  def serverRoot
  def logsDir
  //def serverCmd
  def pid
 
        def install = {
                log.info('############### Enter in Install Phase ###############');
                def jettySkeleton = shell.fetch(params.skeleton)
                def distribution = shell.untar(jettySkeleton)
                //shell.rmdirs(mountPoint)
                serverRoot = shell.mv(shell.ls(distribution)[0], mountPoint)
                if(serverRoot != null){
                        log.info('serverRoot is not Null: '+serverRoot)
                }
                logsDir = serverRoot.'logs'
                log.info 'Log File Location : '+ logsDir.file
                log.info('############### Exit from install Phase ###############');
        }
 
        def configure = {
                log.info('############### Enter in configure Phase ###############');
                log.info "Configure (${params.message})"
                def configFile = serverRoot.'scripts/config.properties'.file
                log.info 'Configuration Location: '+ configFile
                Properties props = new Properties()
                props.setProperty('popcorn.jmx.name',"${params.'popcorn.jmx.name'}")
                props.setProperty('http.port',"${params.'http.port'}")
                props.setProperty('spring.http.port',"${params.'spring.http.port'}")
                props.setProperty('queue.manager',"${params.'queue.manager'}")
                props.setProperty('property1',"${params.property1}")
                props.setProperty('property2',"${params.property2}")

                configFile.withOutputStream{ os->
                        props.store(os,'Config Properties')
                }

                //check for PID
                pid = isProcessUp()
                log.info( 'Process Id: '+pid )

                //Configure Timer for ServerMonitor
                timers.schedule(timer: serverMonitor, repeatFrequency: params.serverMonitorFrequency ?: '15s')
                log.info('############### Exit from configure Phase ###############');
        }

        def start = {
                log.info('############### Enter in Start phase ###############');
                log.info " Application Root: "+ serverRoot.file
               
                shell.exec(redirectStderr:true, pwd:serverRoot.file,command:params.startCommand)
                shell.waitFor(timeout: '5s', heartbeat: '250') {
                        pid = isProcessUp()
                }
        log.info('############### Exit from Start Phase ###############');
        }

        def stop = {
                log.info " ########## Stop phase started . StopCommand (${params.stopCommand}) #################"
                doStop()
                log.info("######### Stopped Phase Completed ####################")
        }
       
        def unconfigure = {
                log.info "########### Enter in unconfigure###########"
                timers.cancel(timer: serverMonitor)
                log.info "########### Exit From unconfigure###########"
         }

        def uninstall = { }

        private def doStop = {
                log.info "########### Enter in doStop ###################"
                if(isProcessDown()){
                        log.info "########### Application is  Down ###################"
                }else{
                        shell.exec(redirectStderr:true,pwd:serverRoot.file,command: "${params.stopCommand}")
                        //Wait  for the process to be down
                        try{
                                log.info 'Going to sleep.....'
                                Thread.sleep(1000)
                                log.info 'I am back from sleep.....'
                                shell.waitFor(timeout: params.stopTimeout ?: '3s', heartbeat: '1s') { duration ->
                                        log.info "########### ${duration}: Waiting for server to be down ###########"
                                        isProcessDown()
                                }
                        }catch(Exception e){
                                log.info("Exception : ${e.message}")
                        }
                        log.info("########### Process Stopped ###########")
                }
                log.info "########### Exit From doStop ###################"
        }
       
        private Integer isProcessUp() {
                log.info('################### Enter in isProcessUp ####################');
                try{
                        def processId = shell.exec(redirectStderr:true,pwd:serverRoot.file,command:"${params.pingCommand}")
                        if(processId != null && !processId.equals("") && !processId.equals("0")){
                                log.info("ProcessId :"+processId)
                                return processId as int
                        }else{
                                log.info(" ########### Application is not running. ###########")
                                return null
                        }
                }catch(ShellExecException e){
                        log.info("isProcesUP Exception :${e.message}")
                        return -1
                }
        }

        private boolean isProcessDown()
        {
                isProcessUp() == null
        }

        def serverMonitor = {
                try{
                        log.info '############ Enter Server Monitor ############'

                        // check if process is up
                        def pidTemp = isProcessUp()
                        def currentState = stateManager.state.currentState
                        log.info 'Application Current State :'+ currentState
                        if(pidTemp == null || pidTemp == 0){
                                pid = null
                        }else{
                                pid = pidTemp
                        }

                        if(pid == null){
                                log.info("############### Server is down. Please check the logs ###############")
                                def newState = 'stopped'
                                def newError = 'Server is down. Please check the logs'
                                if(currentState != 'stopped'){
                                        stateManager.forceChangeState(newState,newError);
                                }else{
                                        log.info ' @@@ELSE@@@ Application Current State :'+ currentState
                                }
                        }
                }catch(Throwable th){
                        log.warn ("In serverMonitor ############### Exception while running serverMonitor: ${th.message}   ###############")
                        log.debug('In serverMonitor ############### Exception while running serverMonitor (ignored) ###############' , th)
                }
                log.info '############ Exit From Server Monitor ############'
        }
}



Thanks!
BS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

BhagatSingh
This post was updated on .
serverRoot is missing in zookeeper. please see blow:
 
"scriptState":{
      "script":{
         "version":"@script.version@",
         "pid":8272
      },
      "stateMachine":{
         "currentState":"running"
      },
      "timers":[
         {
            "timer":"serverMonitor",
            "repeatFrequency":"15s"
         }
      ]
   }

Yan: any clue where to look (for debugging ) to find out why zk client is missing this variable while writing to zk?

Thanks!
BS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

frenchyan
Administrator
In reply to this post by BhagatSingh
At the end of the day, the class that is involved is https://github.com/pongasoft/glu/blob/master/agent/org.linkedin.glu.agent-impl/src/main/groovy/org/linkedin/glu/agent/impl/storage/FileSystemStorage.groovy

This is the one that writes the state of the script (hence the attributes of the class) to the file system and reads it back on bootstrap. 

Whatever is written in ZooKeeper is never read back/used by the agent... it is only for other clients/listener to see the values. So you should investigate the loadState / storeState methods on FileSystemStorage

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

BhagatSingh
Thanks Yan, sure I will have a look at this class.

Thanks!
BS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

BhagatSingh
This is working fine after making a small change in org.linkedin.util.io.resource.internal.AbstractResource Constructor, added code is in bold:

/**
   * Constructor
   */
  protected AbstractResource(InternalResourceProvider resourceProvider, String path)
  {
    if(path != null && !path.isEmpty() && path.indexOf("\\") > 0){
        path = path.replace("\\", "/");
    }

    if(!path.startsWith("/"))
      throw new IllegalArgumentException(path + " must start with /");
    if(resourceProvider == null)
      throw new NullPointerException("resource provider cannot be null");
   
    _resourceProvider = resourceProvider;
    _path = path;
  }
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

frenchyan
Administrator
This change may "fix" your issue but seems to not be the proper place to be done. The path provided to the constructor should not contain \ in the first place, so whoever is providing this path is providing an already wrong path... for all intent and purposes it would be better to change the constructor to reject invalid path instead:

    if(path != null && !path.isEmpty() && path.indexOf("\\") > 0){
        throw new IllegalArgumentException("invalid path ${path}")

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

BhagatSingh
Hi Yan,
We are setting all locations with '/' but some of them are getting converted to '\\' at runtime in Windows. Therefore we can not reject it.

Here is what I am supplying:

VM Options:
-Dorg.linkedin.app.name=org.linkedin.glu.agent-server -Dorg.linkedin.app.version=5.4.1 -Xmx128m -XX:+PrintGCDateStamps -Xloggc:file:file:///projects/glu/agent/org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1/data/logs/gc.log -Dlog4j.configuration=file:///projects/glu/agent/org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1/5.4.1/conf/log4j.xml -Djava.util.logging.config.file=file:///projects/glu/agent/org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1/5.4.1/conf/logging.properties -Djava.io.tmpdir=/projects/glu/agent/org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1/data/tmp -Dsun.security.pkcs11.enable-solaris=false -Djava.awt.headless=true -Dglu.agent.zkConnectString=devpmapp1.pimco.imswest.sscims.com:2188 -Dglu.agent.name=bhagat-agent-2 -Dglu.agent.port=12910 -Dglu.agent.homeDir=/projects/glu/agent/org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1 -Dglu.agent.apps=/projects/glu/agent/apps -Dglu.agent.zookeeper.root=/org/glu

Program arguments:
file:///projects/glu/agent/org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1/5.4.1/conf/agentConfig.properties

Here is how it got changed to '\' in agent.properties:
glu.agent.scriptRootDir=C\:\\projects\\glu\\agent\\apps
glu.agent.logDir=C\:\\projects\\glu\\agent\\org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1\\data\\logs


Don't you think this code with not impact anything, it will only change '\' to '/' in path if it exists. Thereafter existing code is taking care...

if(path != null && !path.isEmpty() && path.indexOf("\\") > 0){
        path = path.replace("\\", "/");
    }
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

frenchyan
Administrator
There is a chain of events between the command line and the call to the constructor. It seems to me that the chain of events is not doing what it should be doing. This is what needs to be investigated.

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

BhagatSingh
It make 100% sense to me, but that was a fix for the problem in short time frame here in my project  . I will investigate it in my down time.

Thanks!
Bhagat
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Instance variable 'serverRoot' value does not retain on agent start

frenchyan
Administrator
Agreed

Yan
Loading...