Hybrid Serial/Parallel Execution Plan

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Hybrid Serial/Parallel Execution Plan

tsukit
I have been trying to find the answer to my question below but I couldn't so I'm posting the question here. Not sure if it's been asked before. Sorry in advance if it has, and in that case please point me to it.

Currently Glu only supports serial and parallel execution plan. If I, say, have 10 entries for 10 different agents/machines. Now I want to perform an action (e.g. deploy, bounce). If choose serial execution plan, then Glu will perform the action one action/machine at a time. And if choose parallel execution, then Glu will perform the action on all 10 agents/machines at once.

However, is there a hybrid option that I can have Glu perform the action in parallel on N (say 2) number of agents/machines, then move on to another N again in parallel, until the action have been performed on all agents/machines?

Thanks.
Ed
Reply | Threaded
Open this post in threaded view
|

Re: Hybrid Serial/Parallel Execution Plan

frenchyan
Administrator
Technically glu can execute any kind of plan, so the one you describe which is essentially something like (see http://pongasoft.github.io/glu/docs/latest/html/tutorial.html#using-the-console-cli for the xml representation of the plan"):

<sequential>
  <parallel>
    <sequential>  agent A1 / MountPoint M1  </sequential>
    <sequential>  agent A2 / MountPoint M2  </sequential>
  </parallel>
  <parallel>
    <sequential>  agent A3 / MountPoint M3  </sequential>
    <sequential>  agent A4 / MountPoint M4  </sequential>
  </parallel>
   ... ...
</sequential>

can totally be executed by the engine.

The only catch is that at this point, glu does not generate this kind of plan nor is there a way to create your own plan.

So unfortunately it is not supported directly at this time. That being said you can achieve what you want by doing the following (which involves using the rest api / console-cli):

1) generate the sequential or parallel plan without executing it (option -n of the console-cli tool)
2) parse the xml back to extract the full list of agent/mountPoint that belong to the plan
3) generate a parallel plan with the first N elements of the list by using a system filter (-s option), and execute the plan
4) repeat at 3 until list is empty

=> you will execute sequentially a set of parallel plans containing N entries which is what you want.

There might be another way (which would be global): https://github.com/pongasoft/glu/issues/147 was implemented a while ago: it basically allows you to limit the parallelism of parallel plan (which may be what you are after) (https://github.com/pongasoft/glu/blob/master/packaging/org.linkedin.glu.packaging-setup/src/cmdline/resources/config-templates/console-server/conf/glu-console-webapp.groovy.gtmpl#L89). You can set the property in the meta model to whatever value you want. Let's say 10. It is different in the sense that it is not going to split your full parallel plan into buckets of 10: it will not execute all in parallel but only 10 at a time, once one completes, another one starts.

I actually like your idea/concept of the scale of hybrid plan: on one end of the scale it is fully parallel, on the other end of the scale it is fully sequential. I am just wondering what would be the best way to visually represent it (parallel and sequential are easy... hybrid requires an extra parameter N)... Any suggestion would be appreciated (created a ticket for this feature: https://github.com/pongasoft/glu/issues/260)

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Hybrid Serial/Parallel Execution Plan

tsukit
Thank you Yan for the information. We in fact completely control Glu via REST and build lots of abstractions on top it to generate proper static models. However, giving the complexity and the fact that it's not a showstopper for us right now, I'll wait until this feature comes out. Some apps of our have lots of hosts. This feature will significantly speed up the entire process.
Reply | Threaded
Open this post in threaded view
|

Re: Hybrid Serial/Parallel Execution Plan

frenchyan
Administrator
I have implemented the feature and it is now part of 5.5.0

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Hybrid Serial/Parallel Execution Plan

tsukit
That's awesome! Thanks Yan for very quick turnaround.