Only allow one file to process through a flow

[JimmyHartington]

I have a flow, which uses some external applications for creating bookmarks and merging files, while keeping the bookmarks intact.



The flow recieves a zip-file, which is uncompressed and all files are converted to pdf.

Bookmarks are added and the job is merged.



Sometimes the zip-files entering the flow, contains the same files. In the end there a distributed by another process to different addresses. That is why the same files enters the flow.



The problem is that when the same files are processed, then the external process merges the files wrong.



Now I am looking into only allowing one file zip-file to process to the end before another zip-file must be processed.



How should I go about accomplishing this?

[freddyp]

There is a very easy way and a somewhat less easy way.



The very easy way is to add a Hold element at the beginning of the flow and to set it to "Space jobs apart". If the processing of the ZIP file is guaranteed to finish within say 20 seconds you could space the jobs 30 seconds apart (a bit of safety margin) and that solves your problem. The downside is that you will not be able to process more than 120 files per hour, but perhaps that is not a problem?



The less easy way involves creating e.g. 5 flows, a master flow for distribution and 4 identical flows to process the jobs. The master flow uses the Hold element to distribute the incoming files to the 4 process flows. The output folders of the master flow are the input folders of the identical flows (tip: do not strip unique name in the output folder). By having the ZIP files in different flows you know they will not stumble over each other's files. The advantage here is that you have concurrent processing. But take care: it is not guaranteed that the 4 flows only have 1 file in them! That depends on how quickly the ZIP files arrive. To get around that you can increase the number of flows and/or combine it with the first method.



I can also think of a method using two scripts in a flow, one at the beginning, one at the end, that set a global variable on or off depending on whether a file is processing in the flow or not, but I think one of the two methods above will probably do.



Freddy

[JimmyHartington]

Hi Freddy



Did not know about the "Space jobs apart". Will use this.



Thanks for your help.

[JimmyHartington]

By the way.



A flow setting a global variable, is that set using a script element?

And how do I set a variable as global?

[freddyp]

You could also use a script expression, but an element would make more sense in this case. To see what capabilities there are for using global variables I suggest you consult the online help under Scripting Module - Scripting Reference - Flow element module - Environment class. It is easy enough.



Freddy

[dkelly]

to set



setGlobalData( "myscript", "key", "value" );



to read



var myValue = getGlobalData( "myscript", "key" );

[JimmyHartington]

Hi dkelly



I have tried to create a Script Element with your code.



The code is:

// Is invoked each time a new job arrives in one of the input folders for the flow element.

// The newly arrived job is passed as the second parameter.

function jobArrived( s : Switch, job : Job )

{

setGlobalData( "Processing", "key", "process" );

}



// Is invoked at regular intervals regardless of whether a new job arrived or not.

// The interval can be modified with s.setTimerInterval().

function timerFired( s : Switch )

{

}





But I get an error saying:

Error in line 5 of script : TypeError. 'setGlobalData' undefined or not a function



What I am trying to achieve is this.

1. A zip-file enters flow.

2. A hold element only passes a file through if a global variable (ProcessStatus) equals done.

3. A script element sets a global variable to processing.

4. All kinds of uncompress, convert to pdf, add bookmarks, merge to one pdf happens.

5. A script element sets the global variable (ProcessStatus) to done.

6. The next zip-file is processed.



And I am a novice in creating a script element. All previous attempts have been more copy-paste.



Hope you understand what I am trying to achieve.



Kind regards Jimmy Hartington

[dkelly]

Sorry, I left out the "s." prefix.





function jobArrived( s : Switch, job : Job )

{

s.setGlobalData( "Processing", "key", "process" );

}

[freddyp]

Does this mean that the "Space jobs apart" method does not solve your problem? Why not?



I am the last person to say that a script cannot be extremely useful, but I always try to find a solution that does not include a script, because that makes it more generic, and easier for those who have no scripting experience (or who do not have the scripting module).



Freddy

[JimmyHartington]

freddyp wrote: Does this mean that the "Space jobs apart" method does not solve your problem? Why not?


It does work. But this flow process a zip-file containing Word, PowerPoint and pdf-files, which are all converted to pdf, processed to add a chapter page and bookmarks. Then this file is merged.

Depending on the number and types of files in the zip, then this can take 5-25 minuttes.

So I have set "Space jobs apart" to 30 min.

But this also slows the next (if there are any) if the first file only took 5 minuttes.



So it is working now, but I am also looking to refine it.

[dkelly]

Here's one way to do it







The first Hold Job has the following release condition



var status = s.getGlobalData( "Processing", "ProcessStatus" );

var bRelease = "false";

s.log(1, "status: " + status);

if (status.length == "") {

s.setGlobalData( "Processing", "ProcessStatus", "working" );

bRelease = "true";

}

bRelease;



The 2nd Hold Job just simulates a long processing time and is not required. The following folder clears the ProcessStatus flag allowing another job to run.



s.setGlobalData( "Processing", "ProcessStatus", "" );





You can download the flow from https://dl.dropboxusercontent.com/u/621 ... 0run.sflow



Learn advanced Javascript for Switch

Full day seminar during Print 2013 in Chicago, September 10th

http://www.brownpapertickets.com/event/449541



Dwight Kelly

Apago, Inc.

dkelly@apago.com

[JimmyHartington]

Hi Dwight



Sorry for the very late reply.



Your flow works as described.

If I process 10 files it holds as excepted.

But if there is a period with no files running through, then it stops working.



I have made a solution, which works but could be described as a bit of a hack.



I use the hold configurator to only allow one file in a folder. This folder has 2 outgoing connections. One sends the zip file to processing and the other one sends to a folder. But the last connection is set on hold.

This means that a copy of the file stays in the folder, where only one file is allowed.

Then at the end of the flow a delete command deletes the file in the folder.

Now the folder is empty and the hold configurator accepts the next file.



Thanks for all the help on this topic.



Here is the first and last part of the flow:

[mclarke]

I have a similar need. I need to sort XML files by an internal variable one at a time with a set amount of time between each file. I looked at the Hold Job tool and do not see an option for "Space Jobs Apart". I tried using Delay with a time of 30 seconds, but all the files in the incoming folder were sent at the same time.



I am using Switch 11 update 5.



What am I missing?

[dkelly]

My suggestion would be a simple script that implements timerFired() every 30 seconds. It would wake up and single one file.





function timerFired( s : Switch )

{

if (s.getTimerInterval() == 0)

s.setTimerInterval(30);



var jobs = s.getJobs();

if (jobs.length) {

var job = jobs.getItem(0);

job.sendToSingle(job.getPath());

}

}

[freddyp]

@mclarke: the reason why you did not see "Space jobs apart" is because the interesting options of "Hold job" are not on the element itself, but on the outgoing connection(s). Look at the properties of the arrows; there you will find "Space jobs apart".



Didactical side note: "Inject job" is another element that has the interesting options on the outgoing connection.



Freddy