Records limit with VDP flows?

[LasseThid]

Is there a limit regarding how many records you can have in a database when creating a VDP flow?
I have a database with 3100 records that keep failing with the error "Communication was aborted: ScriptExecutor termination was requested" and then "Configurator job was aborted because the operation timed out".

[freddyp]

May I assume that you are using the HP Smartstream Designer VDP configurator? There is no limit for the number of lines in a CSV that you input to the configurator, but there is a setting in the Switch Preferences - Error handling that determines after how minutes of processing Switch thinks that a process is hanging and kills it. My guess is that the amount of time needed to process the 3100 lines is longer than the time configured in your Preferences.

So, there are two possible approaches: increase the setting in the Preferences or split the CSV in smaller parts.

[LasseThid]

Thanks for the information Freddy.

Yes, I am using the SmartStream Designer VDP Configurator for the flow.
The time out was set to 20 minutes, so I just increased it to 120 minutes and we'll see what happens.

I did trim the csv down to 1500 records and still it failed, so I trimmed it to 1000 records and it's currently processing.

Update: I just had a chat with the manager of the Indigos and he said I shouldn't create a pdf from the SmartStream Designer VDP, but rather a JLYT... perhaps that's the reason it's taking so long...
PDF was great for me though as I could verify that the output was correct...

[gabrielp]

Update: I just had a chat with the manager of the Indigos and he said I shouldn't create a pdf from the SmartStream Designer VDP, but rather a JLYT... perhaps that's the reason it's taking so long...
PDF was great for me though as I could verify that the output was correct...

Honestly, I think the JLT format is problematic. PDF workflows are much better because we can use tools like PDF readers to verify what we're printing before it rips and gets locked into the Indigo ecosystem. It allows you to proof from your production file, archive it for later auditing, and the possibility of reprinting the same pre-built production file at a later time on any device that supports PDF.

Creating PDFs in SmartStream can be a huge pain depending on the rules and size of the elements within the template. We've had success downsampling images and removing unneeded elements. Another interesting bit is how SmartStream processes, it blows through the first 20% and then comes to a crawl on our environments. So sometimes we split up databases into smaller chunks. I don't think folks who use InDesign server have this problem.

The configurator relying on the global timeout is a huge nuisance and the vague (and frequently incorrect) log errors/warnings it produces are incredibly frustrating. It would be much better is the configurator had a built in timeout that didn't rely on the global timeout. It's the reason that our global timeout is currently 90 minutes, which means when any of our other flows lock up, we can't service that flow for an hour and a half or until we force quit Switch Server.

[LasseThid]

I restarted the developer server since my Mac would create a JLT-file from the original csv-file with 3117 in 69 seconds.
Now the file is processed much faster than before, even though it takes almost twice as long for the SmartStream Designer VDP configurator to create the JLT.
Just for fun I'm gonna try to create a PDF.