Glu truncates script output

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Glu truncates script output

xyang
Hi Yan,

We have Glu script calls out some python script on the agent box, we have seen issues that when the python script fails, the output of the python script gets truncated on the Glu deployment page. Is there a way to not let Glu truncate the output?

For example:
org.linkedin.glu.agent.api.ScriptExecutionCauseException: [org.linkedin.glu.agent.api.ScriptFailedException]: Install failed: Error while executing command [SOME_PYTHON_COMMAND] res=1 - output=[
TRUNCATED
OUTPUT
FROM
PYTHON
SCRIPT]... - error=
[
STACKTRACE
]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu truncates script output

frenchyan
Administrator
I do not remember exactly but if you follow the link to the mountPoint and go the agent view, there is a link that states: view full stack trace. When you click there, is it still truncated?

Yan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu truncates script output

xyang
This post was updated on .
Sorry Yan, I clicked on the mountPoint which directed me to the agent view. I did not see view stack trace link there. All I can see under the agent is Logs: main | gc | more... I am running Glu 5.5.5

And the agent log has the same truncated error output as it is shown on the page.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu truncates script output

frenchyan
Administrator
When you are on the agent page, you ​click on "View Details" for the mountPoint (not the agent)... down at the end there you should be a error section with "View Full Stack Trace" link.

See screenshot.

Yan

Inline image 1
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu truncates script output

frenchyan
Administrator
In reply to this post by xyang

I dug a little in the code and the line that generate the exception is here: https://github.com/pongasoft/glu/blob/master/utils/org.linkedin.glu.utils/src/main/groovy/org/linkedin/glu/groovy/utils/shell/ShellExec.groovy#L312

      ShellExecException exception =
      new ShellExecException("Error while executing command ${commandLineAsString}: res=${exitValue} - output=${shell.toLimitedStringOutput(bytes[StreamType.stdout], 512)} - error=${shell.toLimitedStringOutput(bytes[StreamType.stderr], 512)}".toString())
      exception.res = exitValue
      exception.output = bytes[StreamType.stdout]
      exception.error = bytes[StreamType.stderr]

      throw exception


The output (and error) are purposely truncated because they are part of the message in the exception. They could be quite large obviously, and since they end up in ZooKeeper it would not make sense to store megabytes of data this way.

The (full) output is also stored in the output variable of the exception but at this stage it is not being used or extracted anywhere.

I created a feature ticket to see if we can display it somehow: https://github.com/pongasoft/glu/issues/293

What I would suggest you to do is when you fork a process, redirect output yourself: the shell.exec api offers many ways to do just this (https://github.com/pongasoft/glu/blob/master/agent/org.linkedin.glu.agent-api/src/main/groovy/org/linkedin/glu/agent/api/Shell.groovy#L579)

you set failOnError to false and handle output the way you want in your glu script.

Another way would be that the forked process can be a wrapper script which redirects output and error into a file and display on its output the file that has the redirect... that way if there is an error, the location of the output and error file will be directly in the error message of the exception (provided it is less than 512 bytes :)

Yan

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glu truncates script output

xyang
Thank you Yan, for the help!
Loading...