Framework jobs operate on a "file set". A file set can be thought of as a group of files that a job step is currently working on. A particular job step may cause files to be added to the current file set, removed from the file set, or a combination of the two. In order to implement this aspect of the framework, the files belonging to the file set are stored locally on disk, in the Data\ directory for the corresponding job. This allows the Retry Run feature to work correctly without inadvertently duplicating steps for a given file. To best use the concept of file sets to your advantage, it is worthwhile to review a few examples to illustrate how a job interacts with a file set.
Example 1
First, let us consider a simple job that:
(1) downloads all files from a particular remote directory on a managed site
(2) delivers all *.txt files to a particular directory on the local filesystem; and
(3) places all other files in a separate directory on the local file system.
Initially, before any job step has executed, the file set is empty. Let's assume that we have the following files in the root directory of our chosen managed site, "myftpsite":
Files in remote directory / on site "myftpsite"
log_20180410-143307.txt
log_20180410-170122.txt
report_11201.csv
report_11327.csv
chart_11201.svg
chart_11327.svg
In order to download files from our chosen managed site, we will use a PULL step:
File set before executing the PULL step
[Empty]
Since our PULL step file filter was set to "*", all files that were found on the root directory of the "myftpsite" site will be added to our file set, and each such file will be downloaded to a local working directory managed by Robo-FTP.
File set after executing the PULL step
log_20180410-143307.txt
log_20180410-170122.txt
report_11201.csv
report_11327.csv
chart_11201.svg
chart_11327.svg
Our next task is to place all *.txt files in a particular local directory. We will use a DELIVER step to accomplish this:
File set after executing the DELIVER step (no change)
log_20180410-143307.txt
log_20180410-170122.txt
report_11201.csv
report_11327.csv
chart_11201.svg
chart_11327.svg
Here, the DELIVER step uses the "File Filter" to select all *.txt files from the current file set and places all such files in the given destination folder (C:\myftpsite\logs). The file set is not altered by the DELIVER step. This leaves us free to do other things with these files in subsequent steps, such as upload them to a separate server or any number of possibilities. However, if you recall, in this case our next task is to place all other files (non- *.txt) in a separate directory. In this case we would want to use another DELIVER step, with File Filter set to "*". Remember though, that the *.txt files are still in our file set at the end of the first DELIVER step, so we will want to make some explicit changes to the file set in order to ensure that our second DELIVER step does not place any *.txt files in that separate directory.
Specifically, we wish to remove all *.txt files from the file set so that they are no longer available to any future steps. In order to achieve this we make use of the WHITELIST step. The WHITELIST step allows us to manipulate the file set by removing files that match a particular pattern, or by only including files that match another pattern. Since here we will want to remove all *.txt files, we use the following step definition:
File set after executing the WHITELIST step
report_11201.csv
report_11327.csv
chart_11201.svg
chart_11327.svg
Finally, we place the remaining, non-*.txt files in the other local directory with a second DELIVER step. The DELIVER step never modifies the file set so if we, hypothetically, had any additional tasks to perform, they would still be available in subsequent steps. In this example we do not, and our job ends here.
File set after executing the DELIVER step (no change)
report_11201.csv
report_11327.csv
chart_11201.svg
chart_11327.svg
Example 2
In this example, our goal is to:
(1) grab a set of files from a local directory;
(2) place a copy of the files in a special archive directory on the local filesystem;
(3) compress the files we found into a .zip file;
(4) PGP encrypt the file; and then
(5) upload the PGP-encrypted file to a managed site
While there is a lot going on here, this is an extremely common use case for the Framework and is a snap to implement. Unlike the previous example, no explicit manipulation of the file set is needed. We will use the following steps in this order: MONITOR, DELIVER, COMPRESS, ENCRYPT, and PUSH. Each such step corresponds to the tasks listed above. Let us examine how the file set changes as each step is executed. We start by assuming that there are 3 files present in our C:\outgoing directory:
outgoing_file1.txt
outgoing_file2.txt
outgoing_file3.txt
The MONITOR step waits for files to appear in the specified source directory ("File Path"). After executing the MONITOR step, the file set is populated with those three files above.
File set before executing the MONITOR step
[Empty]
File set after executing the MONITOR step
outgoing_file1.txt
outgoing_file2.txt
outgoing_file3.txt
Before uploading these files, we want to place a copy of them in an archive directory in our local filesystem. The DELIVER step makes use of the "File filter" parameter to select a subset of the files in the file set and places them in the Destination path (C:\outgoing\archive). In this case, "File filter" is set to the * wildcard, so all files from the file set are selected and placed in the C:\outgoing\archive directory. As always, the DELIVER step does not make any modification to the file set itself.
File set after executing the DELIVER step (no change)
outgoing_file1.txt
outgoing_file2.txt
outgoing_file3.txt
Next, we want to place all files found in the original directory and place them in a zip file. The COMPRESS step takes all files that match the File filter (*) wildcard and zips them into the filename specified in the Target File field. Moreover, after the compression has taken place, all files that matched the File Filter pattern are removed from the file set and replaced with the new zip file. Because they have been removed from the file set, the original files are no longer accessible to any subsequent steps.
File set after executing the COMPRESS step
outgoing_18-04-10_1337.zip
Our next task is to PGP encrypt the zip file before sending it out. The ENCRYPT step first selects the files that match the given file filter. In this case, it is the single zip file that was created in the previous COMPRESS step. A PGP key previously imported into the Configurator's PGP Keys menu is selected and PGP encryption is performed on the file, which results in a new file named "outgoing_18-04-10_1337.zip.pgp". The ENCRYPT step functions similarly to the COMPRESS step in that files that match the File filter are replaced with the output of the step. In this case, the zip file in the file set is removed and replaced with our new PGP file. As before, the original zip file is no longer accessible to any subsequent steps because it is no longer in the file set.
File set after executing the ENCRYPT step
outgoing_18-04-10_1337.zip.pgp
Lastly, the PUSH step logs in to the "myftpsite" managed site and uploads all files in the file set (which now consists solely of the PGP file created in the previous step) to the remote "/incoming" directory. The PUSH step does not modify the file set, so upon completion, this PGP file remains in the file set, and would be accessible to any further steps. In this case however, there are no such steps and the job has run to completion.
File set after executing the PUSH step (no change)
outgoing_18-04-10_1337.zip.pgp