Note: First see Working with File Sets for a discussion on how data flows through step to step.
The Monitor step adds local files to a job's file set.
Friendly Name |
INI Variable |
Required? |
Description |
File Filter |
specification |
optional |
specification for source file to match |
File Path |
path |
required |
directory to monitor for a source file |
Timeout |
timeout |
optional |
Timeout in minutes looking for source file and send error notification |
Leave Files on Host |
persistent_source |
optional |
leave processed files in source. default: false |
Include Directories |
include_directories |
optional |
import directories matching specification. default: false |
File Order |
file_order_first |
optional |
Start processing files starting from the oldest or newest files. Especially significant when multiple_files is something other than all. Possible values are "oldest" or "newest", without the quotes. |
Multiple Files |
multiple_files |
optional |
Number of files matching spec to import. Any is zero or more making the absence of a file for this step to import not an error. Possible values are "#", "all", or "any", defaulting to "all", without the quotes. |
Minimum Age |
older_than |
optional |
Files must be older than the given relative value, which has a resolution of seconds, minutes, hours, days, or months; for example: "1 month" or "7 days". |
Maximum Age |
newer_than |
optional |
Files must be newer than the given relative value, which has a resolution of seconds, minutes, hours, days, or months; for example: "1 month" or "7 days". |
Minimum Size |
min_size |
optional |
Specifies that a file must be at least this many bytes, defaulting to 0. |
N/A |
newer_than |
optional |
A relative value with a resolution of seconds, minutes, hours, days, or months; for example: "1 month" or "7 days". |
N/A |
errors_directory |
optional |
put files that error out here and send notification |
N/A |
archive_directory |
optional |
put files that succeed here |
Archive Files |
archive_files |
optional |
keep archival copies of created or imported files |
The Monitor step defines both the beginning and ending processing for jobs that operate on local files. Most framework jobs begin with either a Monitor step or a Pull step.
Beginning Processing
The Monitor step searches the source path for file names that match the configured specification. Matching files are added to the job's file set. Subsequent job steps operate on this file set. If no matching files are found within the timeout period this steps returns an error and the remaining job steps are not processed.
If the value of the multiple_files option is a number then exactly that number of files are added to the file set. If the expected number of files are not available the step will error out and job processing will halt. If the value is "all" then all matching files are added to the file set. If no matching files are found the step reports an error and processing stops. If the value is "any" then all matching files are added and the absence of any matching files is not treated as an error and processing continues to the next step.
Ending Processing
After all the subsequent job steps have completed, control returns to the Monitor step and the files are removed from the current file set. If the job was successful the processed files are deleted unless the archive_directory and/or persistent_source options are configured. The archive_directory option is used to store successfully processed files and the persistent_source option causes the Framework to leave the source files in their original place and track their processing status in a database. The errors_directory option is used to keep files when job processing fails. A configuration error is returned if the end-of-processing move or delete operation fails.
The include_directories option does not work with the persistent_source option.
Example Usage:
See also: Scheduling and the Timeout Parameter