Consistently getting - Unable to rename ++tmp.agent.properties.2910d7c8-619e-4b2a-8e69-6e38240186cc.tmp++ to agent.properties

[BhagatSingh]

If agent.properties already exists at c:\..\..\data\config\ then agent doesn't start and throws this exception. Therefore I manually delete it before I start agent.
Once agent is started and running then I did see this exception keep on coming in log but doesn't hurt anything.
Question here is why agent is updating this properties file again and again and how is it working in non-windows env?

Exception:
java.io.IOException: Unable to rename C:\projects\glu\agent\org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1\data\config\++tmp.agent.properties.2910d7c8-619e-4b2a-8e69-6e38240186cc.tmp++ to C:\projects\glu\agent\org.linkedin.glu.agent-server-bhagat-agent-2-12910-zkc1-5.4.1\data\config\agent.properties
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
        at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:71)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:81)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
        at org.linkedin.groovy.util.io.GroovyIOUtils.safeOverwrite(GroovyIOUtils.groovy:299)
        at org.linkedin.groovy.util.io.GroovyIOUtils$safeOverwrite$0.callStatic(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:53)
        at org.linkedin.groovy.util.io.GroovyIOUtils$safeOverwrite$0.callStatic(Unknown Source)
        at org.linkedin.groovy.util.io.GroovyIOUtils.safeOverwrite(GroovyIOUtils.groovy:262)
        at org.linkedin.groovy.util.io.GroovyIOUtils$safeOverwrite.call(Unknown Source)
        at org.linkedin.glu.agent.impl.storage.AgentProperties.save(AgentProperties.groovy:136)
        at org.linkedin.glu.agent.impl.storage.AgentProperties$save.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at org.linkedin.glu.agent.impl.storage.FileSystemStorage.saveAgentProperties(FileSystemStorage.groovy:160)
        at org.linkedin.glu.agent.impl.storage.WriteOnlyStorage$saveAgentProperties.callCurrent(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
        at org.linkedin.glu.agent.impl.storage.FileSystemStorage.updateAgentProperty(FileSystemStorage.groovy:173)
        at org.linkedin.glu.agent.impl.storage.WriteOnlyStorage$updateAgentProperty.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at org.linkedin.glu.agent.impl.storage.FilteredStorage.updateAgentProperty(FilteredStorage.groovy:83)
        at org.linkedin.glu.agent.impl.storage.DualWriteStorage.super$2$updateAgentProperty(DualWriteStorage.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
        at org.linkedin.glu.agent.impl.storage.DualWriteStorage.updateAgentProperty(DualWriteStorage.groovy:74)
        at org.linkedin.glu.agent.impl.storage.WriteOnlyStorage$updateAgentProperty.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at org.linkedin.glu.agent.server.AgentMain$_closure2.doCall(AgentMain.groovy:820)
        at org.linkedin.glu.agent.server.AgentMain$_closure2.doCall(AgentMain.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:848)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:831)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:164)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeClosure(ScriptBytecodeAdapter.java:570)
        at org.linkedin.glu.agent.server.AgentMain.onConnected(AgentMain.groovy:808)
        at org.linkedin.zookeeper.client.ZKClient$StateChangeDispatcher.run(ZKClient.java:183)

[frenchyan]

The agent uses a create/move strategy in order to write a file instead of creating the file directly: so it creates a file whose name is ++tmp.xxxxx.tmp++ in the same directory where the final file needs to be located and if the creation is successful, simply rename the file to its target name (since it is in the same folder, no actual copy or space allocation is required... only a pointer change... at least on Unix...)

Why this is failing on Windows, I do not know. There is a test which test for this feature (https://github.com/pongasoft/utils-misc/blob/master/org.linkedin.util-groovy/src/test/groovy/test/util/io/TestGroovyIOUtils.groovy#L95) I would advise to try running it on Windows.

Yan

[BhagatSingh]

Hi Yan,

I am not sure if this is the correct fix for Windows, but it is working for me. I have added few lines of code in org.linkedin.groovy.util.io.GroovyIOUtils class, please see the added code below in the bold:

static def safeOverwrite(File toFile, Closure tempFileFactory, Closure closure)
  {
    if(toFile == null)
      return null

    mkdirs(toFile.parentFile)

    File newFile = tempFileFactory(toFile)

    try
    {
      def res = closure(newFile)

       if(toFile.isFile()){
           try{
               toFile.delete()
           }catch(Exception exp){
               exp.printStackTrace()
           }
       }
      if(newFile.exists() && !newFile.renameTo(toFile))
      {
        // somehow the rename operation did not work => delete new file (will happen in the finally)
        // and throw an exception thus effectively leaving the file system in the same state as
        // when the method was called
        throw new IOException("Unable to rename ${newFile} to ${toFile}")
      }

      return res
    }
    finally
    {
      // always deleting the newFile in the finally ensuring that the filesystem remain in
      // the same state as when enterred
      GroovyLangUtils.noExceptionWithMessage(toFile) {
        deleteFile(newFile)
      }
    }
  }

[frenchyan]

It may work but the way you are implementing it is changing the meaning of the method: safe overwrite is supposed to "leave the filesystem in the same state if something bad happens".

In your implementation you delete the target file then there is the check for newFile.exists()... if that check fails then the toFile is gone... 

Yan

[BhagatSingh]

Thanks Yan, I will implement it correctly, right now my target is to complete deployment complete cycle in Windows agent and I am almost there.

[BhagatSingh]

Hi Yan,

Here is what I have done for "Unable to rename ++tmp.agent.properties.2910d7c8-619e-4b2a-8e69-6e38240186cc.tmp++ to agent.properties" issue:

 if(newFile.exists() && !isFileRenameSuccessFul(newFile, toFile))
      {
        // somehow the rename operation did not work => delete new file (will happen in the finally)
        // and throw an exception thus effectively leaving the file system in the same state as
        // when the method was called
        throw new IOException("Unable to rename ${newFile} to ${toFile}")
      }

//New Method
/**
     * isFileRenameSuccessFul method creates a backup copy if file already exists and then
     * rename temp file. If rename operation fails then it moves the backup file to original one.
     *
     * @param newFile
     * @param toFile
     * @return true when file rename operation was successful else false
     */
    static boolean isFileRenameSuccessFul(File newFile, File toFile)
    {
        boolean isFileMoveSuccess = true
        String fileName
        String backFileName
        Path toFilePath
        if (toFile.exists()) {
            try {
                fileName = toFile.canonicalPath
                backFileName = fileName + ".backup"
                toFilePath = Paths.get(fileName).toRealPath()
                Files.move(toFilePath, toFilePath.resolveSibling(backFileName), StandardCopyOption.REPLACE_EXISTING)
            }catch (IOException ioExp) {
                isFileMoveSuccess = false
            }
        }else{
            isFileMoveSuccess = false
        }

        //rename temp config file to original config file
        boolean isRenameSuccess =  newFile.renameTo(toFile)

        /**
         * Move backup copy to original if move was success and rename file was unsuccessful.
         */
        if(isFileMoveSuccess && !isRenameSuccess){
            Path backFilePath = Paths.get(backFileName).toRealPath();
            Files.move(backFilePath, backFilePath.resolveSibling(fileName), StandardCopyOption.REPLACE_EXISTING)
        }
       return isRenameSuccess;
    }


***** while debugging I realized that the method safeOverwrite gets called very frequently .. I am wondering why it is ? what gets changed so frequently?

[frenchyan]

It is called every time a property gets changed in a script because the state is persisted right away. This ensures that if the agent crashes and restarts, the most recent values known for a given property will be recovered. It is also available in ZooKeeper right away during the processing of a phase (which can take a long time, for example waiting for a server to be up or down could take minutes...).

Yan

[BhagatSingh]

Are you OK with the code I added/modified to support this issue?

[frenchyan]

It is hard to do a code review on portions of non properly formatted/indented code. Do you have it on a fork in github? That would be easier...

Yan

[BhagatSingh]

Sure, I will submit my changes on fork.

Thanks!
Bhagat

[BhagatSingh]

frenchyan wrote

It is hard to do a code review on portions of non properly
formatted/indented code. Do you have it on a fork in github? That would be
easier...

Yan

Please see the changes at this link:

https://github.com/bhagatsingh/utils-misc/commit/9a3d3009ee45c90c114d9fe8ec932e115478a0d6