Unexpected behavior with file names in Switch - reproducible bug

[gabrielp]

Hi guys,

We're noticing some odd behavior with Switch with different named files. We've tracked it down to some of our filenames containing underscores which are changing the names in unexpected ways. Switch uses a filename prefix to keep track of jobs as they move through Switch and it appears that if you're not careful, it can cause some problems.

Any file Switch is watching is in the following format: _SWITCHID_YourFilename.pdf. So obviously, you shouldn't put any information starting in the file name between two underscores or Switch might think that that information is the Switch ID. I think we all understand that. The problem becomes if you use additional underscores and pass the job between other configurators, the Switch ID can be parsed again, causing odd behavior. I'll explain...

For this test, I've created the following files:
- somefile1.log
- _HELLO_somefile2.log
- __HELLO_somefile3.log
- _12345__HELLO_somefile4.log
- _12345_HELLO_somefile5.log
- 123456_12345_HELLO_somefile6.log
- _12345_HELLO_1234_somefile7.log
- _12345__HELLO_1234_somefile8.log

I'm using _HELLO_ to simulate me putting my information in the file name and _12345_ as a simulated Switch ID.

Now we drag those files into a folder that Switch watches and we see the following transformation:
- somefile1.log
- somefile2.log
- __HELLO_somefile3.log
- _12345__HELLO_somefile4.log
- _12345_HELLO_somefile5.log
- 123456_12345_HELLO_somefile6.log
- _12345_HELLO_1234_somefile7.log
- _HELLO_1234_somefile8.log

The ones in red have changed and I have excluded the prefix that Switch has added to the actual files (I'm showing you how it appears in Designer). It's interesting to note that several files (4, 5, and 7) didn't have the file name parse as we would expect -- so Switch is taking some steps to avoid this problem.

Now, if we pass those jobs through a "Pack job" and into another folder, watch what happens:
- somefile1.log.zip
- somefile2.log.zip
- __HELLO_somefile3.log.zip
- somefile4.log.zip
- HELLO_somefile5.log.zip
- 123456_12345_HELLO_somefile6.log.zip
- HELLO_1234_somefile7.log.zip
- 1234_somefile8.log.zip

The ones in blue have changed since the last step.

You can see now that between every step, Switch parses the file name to see what the job is supposed to be which can cause unintended results. Even scarier, since Switch is parsing a new unique prefix between these steps, it's looking up job information for which is doesn't have -- essentially forgetting any private data and datasets it knew about previously. I believe this is how some of us have encountered "impossible" scenarios in Switch where Switch seems to forget information that we know should have been written.

The long and short is, you should avoid using leading underscores anywhere near the start your file file names when processing jobs through Switch. But this example shows that depending on what configurators you're using, even underscores within the filename can cause bits to later be parsed.

[loicaigon]

Obviously something to be really careful of
Thanks for the effort.