implement time lag between each complete steps while deployment.

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

implement time lag between each complete steps while deployment.

zedfauji
Yan,

I need to know how could i implement some time interval between deployment on each node. Let say when i start SEQUENTIAL deployment , so i need that glu will take some time after it completes deployment on a mount point. Just wait for a given time and then move to next mount point.
-=Zed=- a.k.a -=Fauji=- More quieter you are , more you can hear
Reply | Threaded
Open this post in threaded view
|

Re: implement time lag between each complete steps while deployment.

frenchyan
Administrator
At this stage, glu does not let you create your own plan and/or inject your own logic in the middle. That being said, a way you could achieve what you want to do would be to use the rest api and do the following:

1) issue a query to generate a plan (not execute it.. just generate it) as can be seen in the tutorial (http://pongasoft.github.io/glu/docs/latest/html/tutorial.html#using-the-console-cli step 7) => you get back an xml representation of the (entire) plan

2) parse the xml to extract each agent/mountPoint in the order defined by the plan

3) now for each one do:
   execute plan filtered by agent/mountPoint
   wait given amount of time
   repeat

The filtering syntax is in the documentation as well.

This should definitely work and accomplish what you want to do.

That being said, I would like to understand your use case a bit. Why do you need to wait between mountPoint?

Yan
Reply | Threaded
Open this post in threaded view
|

Re: implement time lag between each complete steps while deployment.

sodul
In reply to this post by zedfauji
If you do not mind taking the hit, you could add a sleep to your start step.

If what you are trying to achieve is to make sure application Foo is ready before application Bar is ready you have several options:
 - ask the developers to update the startup behavior of application Bar so that it will retry to connect Foo for x seconds
 - add that logic in the start step of Bar: check that the Foo port is listening for x seconds, and eventually fail the start of Bar.

Like Yann asked, if you tell us what problem you are trying to solve by adding a delay we might be able to give you a proper solution.
Reply | Threaded
Open this post in threaded view
|

Re: implement time lag between each complete steps while deployment.

zedfauji
Hi Yan & Sodul,

By mean of Deployment i mean execution of jar file. Each mount point is a name of jar file. in deployment what we do is to kill the execution of current , upgrade the version of jar and then start it again. Now before moving to deployment of next mount point i need glu to wait for sometime so that jar we have started earlier should start processing and it takes some time to report to zookeeper. Thing is that by the time that jar has started talking to zookeeper , Glu has already been moved on to next deployment. So by this our pools are getting failed. Thats y i need to implement this. I checked, SequentialStepExecutor.java can we put some time in that loop.?
-=Zed=- a.k.a -=Fauji=- More quieter you are , more you can hear
Reply | Threaded
Open this post in threaded view
|

Re: implement time lag between each complete steps while deployment.

zedfauji
+ could you tell how could i add some sleep or delay in start command or start phase.
-=Zed=- a.k.a -=Fauji=- More quieter you are , more you can hear
Reply | Threaded
Open this post in threaded view
|

Re: implement time lag between each complete steps while deployment.

frenchyan
Administrator
With glu, each phase (install, configure, start, etc...) is supposed to be blocking, meaning it should not terminate until the action that is supposed to happen in the phase is complete.

Example (in pseudo code!):

def start = {
  1) execute jar file
  2) wait for boot procedure of jar file to be complete
}

If you do not have the wait, then the phase will exit before your jar is actually booted. The definition of started depends on what you are starting. For example, if you look at the start phase for the JettyGluScript (https://github.com/pongasoft/glu/blob/master/scripts/org.linkedin.glu.script-jetty/src/main/groovy/JettyGluScript.groovy#L177), you can see:

1) shell.exec ... which starts the jetty server (which contains some webapps)
2) a) wait for the jetty process to be up
    b) wait for jetty app to be up (only when the entire boot process of jetty is complete, will the socket on port "port" will respond)

glu has the shell.waitFor convenient call to wait until a condition becomes true. In the first waitFor, the timeout is set to 5s and the heartbeat to 250ms, which means that the condition (true/false) (which is what the closure returns) will be checked every 250ms for no longer than 5s. This is fine because forking a process should not really take that long in any case. If the timeout is reached, an exception is raised. The second waitFor has a heartbeat of 1s and a timeout provided in initParameters, and by default it is null, which means it will wait until the condition is true: it is very hard to predict how long every webapp included in the container will take to boot (I have seen 20mns for some webapps refreshing their cache on boot!) => it just waits until the condition turns true (but if the boot procedure fails, then shell.fail (L195) will abort it).

You can replace all this by a simple "sleep" (which is your second question): Thread.sleep(5000) which will simply wait for 5s. BUT the issue with this is that there is still no guarantee that your app will be booted within 5s, and if it takes 1s, you are wasting 4s, which may not seem like a lot but across many instances start to add up.

It is highly recommended you implement a proper "wait for boot procedure of jar file to be complete" in your glu script.

Yan

Reply | Threaded
Open this post in threaded view
|

Re: implement time lag between each complete steps while deployment.

sodul
+1 make your start step blocking until you jar is ready. In my web apps I actually wait for the port the app listen to to return before the start step completes, or even better checks for a status page to be ready. It could be the same page that your load balancer might check for a 200 response.

If the start does not return 'OK' within a determined time (say 30s) then the start step is flagged as failed, since it means something is wrong, no need to resume with the next component.

Note that if you do not want to write groovy code you do not have to write much of it. All my components share a single 200 lines groovy script and all my steps logic is written in python or shell.

I've shared my 'abstraction' groovy script in this thread:
http://glu.977617.n3.nabble.com/Need-Help-NewBie-td4025426.html#a4025428

This groovy script has been used to deploy all sort of apps without modification: python apps, C apps, shell scripts, binary artifacts (non executable), tomcat webapps, java apps, etc...