Problem when moving files to other hotfolders from Switch

[gabrielp]

Hi guys,
We've encountered a pretty annoying issue and I just wanted to mention it here in case anyone else comes across it. When processing jobs in Switch, if you have Switch deliver those files to another hotfolder workflow, you may find that the other system watching the hotfolder does not pick the files up, and in some cases, may error out.

In the case of our BCC integration, the error was something like this:

Code: Select all

08/05/2015 10:31:01.124 >> .~#~247_Z-2969011-2969661_R-uni_1438782349_BCC.csv : File Found
08/05/2015 10:31:01.359 >> .~#~247_Z-2969011-2969661_R-uni_1438782349_BCC.csv : File no longer available

In the case of our Creo color server, there was no error we could find. The Creo would just not pick up the files.

In both cases, you can get the files to process by stopping and starting the service that watches those hotfolders (restart BCC or Creo) and it will pick them up the existing files. And in both cases, if you manually drag a file into those hotfolders from Finder or Explorer, it works fine. The issue is just with Switch passing the files off. And of course, "strip unique name" was set to "Off" in the destination folder and we also tried archiving which was also unsuccessful.

I think a clue to what is going on can be found in the BCC error:

08/05/2015 10:31:01.124 >> .~#~247_Z-2969011-2969661_R-uni_1438782349_BCC.csv : File Found

It seems that Switch moves the file to the hotfolder with some sort of internal naming convention prefix ".~#~" and then, in an instant, removes the prefix with a rename. It seems that within the time of the move and the rename, Switch is locking up the file so that these external hotfolder systems can't process the file and they just get stuck.

The solution for us was to write a script that would make Switch, instead of moving the files regularly, would execute a cp command to move the files to the destination via CLI. Here is the script for anyone interested: https://github.com/dominickp/SwitchCopy ... yUnixCp.js

[dkelly]

Switch creates the oddly named file and renames it after writes are complete. This has caused many problems for me both with Prinergy and with NAS hardware from SNAP.

I wrote this copy script to solve the problem. It's pedantic in an attempt to make sure the copy succeeded 100% of the time.

Code: Select all

// Copy job
// Copyright 2013 by Dwight Kelly <dkelly@apago.com>
// All rights reserved

function jobArrived( s : Switch, job : Job )
{
	var destDir = s.getPropertyValue("destDir");
	var destFN = destDir + "/" + job.getName();
	
	if (File.exists(destFN)) {
		// remove any exist file or directory
		if (File.isFile(destFN)) {
			try {
				File.remove(destFN);
			} catch( e ) { 
			    job.fail("Unable to delete existing file " + destFN);
				return;
			}
		} else {
			var dir = new Dir(destFN);
			var entries = dir.entryList("*", Dir.Files, Dir.Name);
			for (var i=0; i<entries.length; i++) {
				var fn;
				try {
					fn = destFN + "/" + entries[i];
					File.remove(fn);
					if (File.exists(fn)) {
						//verify directory/file was deleted
						job.fail("Failed to delete " + fn);
						return;			
					}
				} catch( e ) { 
					job.fail("EXCEPTION: Unable to delete existing file " + fn);
					return;
				}
			}
			dir.rmdir(destFN);
			delete dir;
			if (File.exists(destFN)) {
				//verify directory/file was deleted
				job.fail("Failed to delete " + destFN);
				return;			
			}
		}
	}
	
	try {
		var file = new File(destDir);
		if (!file.exists) {
			job.fail("Destination directory doesn't exist");
			return;
		}
		if (!file.writable) {
			job.fail("Destination directory isn't writable");
			return;
		}
		
		var bCopied = false;
		var nTries = 0;
		for (nTries = 0; nTries < 2; nTries++) {
			s.copy(job.getPath(), destFN);
			//verify it exists after copy
			if (File.exists(destFN)) {
				bCopied = true;
				break;
			}
			s.sleep(2);
		}
		if (bCopied == false) {
			//job.log(3, "File wasn't copied to destination after " + nTries + " attempts");
			var args = new Array;
			args.push("cp");
			args.push(job.getPath());
			args.push(destFN);
			var exitCode = Process.execute(args);
			if (exitCode || !File.exists(destFN)) {
				job.fail("File wasn't copied via cp");
				return;
			}
		}
		if (nTries > 0)
			job.log(2, "File was copied to destination after " + nTries + " attempts");
		
	} catch( e ) { 
		job.fail("EXCEPTION: Unable to copy " + destFN + " error " + e);
		return;
	}
	job.sendToNull(job.getPath());
}

[gabrielp]

Switch creates the oddly named file and renames it after writes are complete. This has caused many problems for me both with Prinergy and with NAS hardware from SNAP.

I wrote this copy script to solve the problem. It's pedantic in an attempt to make sure the copy succeeded 100% of the time.

Impressive work as always, Dwight. Thanks for sharing your solution and I'm glad I'm not the only one who has run into this.

[sander]

Ouch, I encountered the same problem before. My MIS XML import did two imports due of this, I thought it was my MIS problem so decided to use SQL to dump the entire XML in the database.

Today I noticed this job in Production Center, which failed;


So, thanks for your script Dwight, extremely useful for me!

[LasseThid]

We have some problems with copying files to our file server so that our Indigo machines can pick them up and I'm starting to think this might be needed to solve our problem as well, however my code reading skills aren't good enough to figure our how to use this script.

Could anyone please be kind and explain it to me?

Thanks

[gabrielp]

We have some problems with copying files to our file server so that our Indigo machines can pick them up and I'm starting to think this might be needed to solve our problem as well, however my code reading skills aren't good enough to figure our how to use this script.

Could anyone please be kind and explain it to me?

Thanks

I haven't used Dwight's script, but looks like his is pretty contained. The script I shared in the original post relies on the Switch server having access to the CLI program 'cp' which is standard on most unix machines like Macs: http://man7.org/linux/man-pages/man1/cp.1.html

[sander]

Just noticed I only gave my thanks to Dwight but offcourse thank you too Gabriel.

Switch on Windows over here, that's why I skipped the cp script.

I just felt I had to correct myself since it seems you have a solution for a lot of issues

[gabrielp]

Just noticed I only gave my thanks to Dwight but offcourse thank you too Gabriel.

Thanks man, but Dwight's is much better.

It looks like his relies on 'cp' as well. Perhaps we could merge these scripts into a single script that would work for Mac or Windows. If the 'cp' command fails, then we could attempt 'copy' (the DOS version).

In the mean time, you can probably alter mine or Dwight's scripts and simply replace cp with copy since it appears that the basic usage syntax is nearly identical.

[dkelly]

or change

Code: Select all

 args.push("cp");

to

Code: Select all

args.push( s.isWindows() ? "move" : "cp");

[ArielRauch]

Is the "save move" option to a network folder a "workaround"?

I would guess it is first copying to a temp folder removing the prefix and then copying the file without prefix into network destination folder.

Your thoughts?


Ariel

[enfocus]

This is from the documentation:

This option is only relevant for transferring job files between folders on different volumes (drives/partitions).
If set to Yes (default value), Switch will first copy the job to a temporary file before deleting the job from the original location. This is useful to avoid that files get lost in case of network problems.

If set to No, Switch immediately moves the job files to the Archive hierarchy. This may be necessary in case of network permission issues.

Support told me that disabling safe move removes the .~#~ prefix, so I think the temporary file mentioned in the documentation actually is the file with the .~#~ prefix.