Quantcast

All processes view not parsing very long command lines

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

All processes view not parsing very long command lines

sodul
Hi,

We just upgraded our storm installation to the latest release and we ended up with a much longer classpath than before (over 4k character). Now the process view in the console end up with the command line option trimmed in the middle and the console is not able to find the app.

This does not break any critical functionality but does make the view less useful when troubleshooting.

The reason we are getting this is because the kernel will trim the command line to 4096 bytes and this is hardcoded in the kernel itself:



I've found a workaround by using jps -v:
/usr/java/1.7/bin/jps -v | grep ui.log
15887 core -Dstorm.options= -Dstorm.home=/shn/apps/storm/apache-storm-0.9.2-incubating -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -Xmx768m -Djava.net.preferIPv4Stack=true -Dlogfile.name=ui.log -Dlogback.configurationFile=/shn/apps/storm/apache-storm-0.9.2-incubating/logback/cluster.xml -Dorg.linkedin.app.name=storm_ui

It could be used to solve the missing app.name in the All Processes view.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

frenchyan
Administrator
Do you mind posting a screenshot of the issue? I am not entirely sure I know what you are talking about.

Also, when you talk about a fix, is it a fix that needs to be integrated in glu? or is it something that you do when you start your app?

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

sodul
This is what I get on the actuall process view:


Note how the last command line argument reported is -Do, which would have been -Dorg.linkedin.app.name=storm_ui, but is cut off since the Linux kernel only keeps the first 4096 bytes of a command line.

The Storm management scripts are the ones crafting the command line, so I have very limited control on what get passed and in which order (I add the -Dorg.linkedin.app.name=storm_ui since I use it to monitor the running processes, but also for convenience when debugging from the console). So far I've had no problems with the 3rd party apps we use but I do expect to have this issue pop once in a while with Java, especially with apps that still explicitly list every single class file inside a lib directory on their command line.

The jps command ($JAVA_HOME/bin/jps) helps here since it can show the command line options, bypassing the Linux kernel limitation:
$ /usr/java/1.7/bin/jps -v | grep 20601
20601 core -Dstorm.options= -Dstorm.home=/shn/apps/storm/apache-storm-0.9.2-incubating -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -Xmx768m -Djava.net.preferIPv4Stack=true -Dlogfile.name=ui.log -Dlogback.configurationFile=/shn/apps/storm/apache-storm-0.9.2-incubating/logback/cluster.xml -Dorg.linkedin.app.name=storm_ui

Unfortunately it takes 0.3s to run on average, so you don't want to run it often, but the agent could cache the output, at least until a new java process with 4096 bytes command lines is found.

The logic would be:
Agent list all the processes like it does right now. If a java process is listed with a 4096 bytes command line, and -Dorg.linkedin.app.name is not found (or the trailing one, could be a name split in half in such case), then call jps to try to better guess the app name.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

sodul
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

frenchyan
Administrator
I know it is not ideal, but have you tried to put the -Dorg.linkedin.app.name right upfront (after -server)? That way it won't be cut off...

The issue is that the "ps" command is actually using Sigar under the cover and implementing your suggestion is quite heavy (right now it is not forking to get the information). Especially since it can have a pretty significant cost on performance.

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

sodul
I would if I could, but this is a third party app it does not take environment variables and get the java opts from a yaml file: I have no control on the position.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

sodul
One option could be to extend the groovy script to support pid tracking similar to the way we can customize the list of logs through the 'logs' dictionary. Here we would have an optional pids variable that the agent sends back to the console through zookeeper. Monitor would be in charge of updating pids.

The format would be "pid[123] = {"org.linkedin.app.name": "my.app.name", "command": "java"}.
From there the "All Processes" view could populate the missing entries and names. An added feature is that this view would now be able to represent processes running as a different user, which is useful to track sometimes, especially since console/agents/ps is able to handle pid=xxx for an different user than the agent's.

This should require a lot less code on your part, not affect performance for those that do not want it, and allow glu to be much more flexible on that part.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

frenchyan
Administrator
I will think about this... the issue with jps, besides a lot of code and performance, is that it would work only for java processes. I know that -Dorg.linkedin.app.name=xxx is an option for java, but you could start any process with this option (obviously if the process ignores it...) and it would be picked up. But jps would not...

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

frenchyan
Administrator
In reply to this post by sodul
Created a ticket to keep track of it: https://github.com/pongasoft/glu/issues/274

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

sodul
In reply to this post by frenchyan
Yes -Dorg.linkedin.app.name=xxx works well for non-java. As you can see from the screenshot above I use it for my python scripts ;-) I explicitly support this in my python scripts so that they allow and ignore it.

Augmenting the discovery of the pids and app names to the groovy script is probably the least intrusive way to work around the limitation of the existing implementation and without implementing OS specific workaround in Glu itself. I also believe it to be more flexible.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

frenchyan
Administrator
I have implemented what I think you had in mind

Agent View Page:


All Processes Page:


The way it works is this way:

* in the glu script you can still  declare the "pid" entry like before ("main" in the Processes list)
* in addition you can have a map called pids

This is the glu script that generated this output:

class HelloWorldGluScript
{
  def pid = 0
  def pids = [:]

  def start = {
        pid = 1234
        pids[5555] = [command: 'c1xxx', cpu: 12, 'org.linkedin.app.name': "olan-1-${mountPoint}"]
        pids[5556] = [command: 'c2xxx', 'org.linkedin.app.name': "olan-2-${mountPoint}"]
  }
}

The ps command is run normally and then merged with whatever comes from the glu script. As you can see in my example I have used bogus pids, but it still get picked up in the ui and that is fine.

I have added a column MOUNT POINTS which ties back the process to the mount point (only works if pid or pids is defined obviously...) and made it filterable (see little magnifying glass...)

Let me know what you think...

Yan

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

sodul
That looks great!!!

When you say the data is merge does that mean the data is preserved when you have the same pid and no new value. For example if the groovy script does not provide the cpu value, will it be preserved from the data of the current behavior?

No rush on this new feature, it is not blocking but will definitely add more flexibility on Glu.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: All processes view not parsing very long command lines

frenchyan
Administrator
Yes you are correct. I am actually not really expecting the glu script to be providing this value. It was kind of a way for me to test if I could send any data. But it will indeed be merged. 

Since it is mostly all implemented it will make it into the next release.

Yan
Loading...