Processing Dynamic Folders

If you need to process all of the files in a folder without changing any of them then you should use the GETFILE or FTPGETFILE command. Likewise, you can use either the GETNEXTFILE or the GETSITEFILE command without the /next option to process all of the files in a folder when you plan to move or delete every file. If you need to process the contents of a folder and conditionally move, rename, or delete only some of the files then you'll need to use either the DIFF or the FTPDIFF command.

The trick to processing folders with dynamic contents is to recreate the snapshot database during each execution of your command script. By deleting and recreating the database, every file in the target folder is considered "new" and so they all trigger the DIFF_FILE_IS_NEW difference. The GETDIFF or FTPGETDIFF command is then used to cycle through all the files using the snapshot database instead of a directory listing.  This allows you to change the contents of the directory without risk of skipping files or processing an existing file twice.

The following example script builds a database containing all the files under the source folder where the file name ends in the ".txt" extension, including any files in sub-folders. Then it reads the database in a loop and the text files are uploaded to a remote site.  If the local file was in a sub-folder, a corresponding sub-folder is created on the remote site. The local source files are moved to an archive folder after they are successfully uploaded. Files not ending in ".txt" are ignored.

SET source = "c:\data\original"

SET archive = "c:\data\archive" ;; archive must NOT be a sub-folder of source

SET snapshot = %installdir + "\files.sql"

DELETE snapshot                 ;; get rid of existing snapshot database

WORKINGDIR source

IFERROR!= $ERROR_SUCCESS GOTO done

ARCHIVEDIR archive

IFERROR!= $ERROR_SUCCESS GOTO done

SET source = source + "\*.txt"   ;; process all TXT files in source folder

DIFF source snapshot /incldirs /append

IFERROR!= $ERROR_SUCCESS GOTO done

IFNUM= %difffiles 0 GOTO done

FTPLOGON "ftp.mydomain.com" /user="UserID" /pw="secret"

IFERROR!= $ERROR_SUCCESS GOTO done

:loop_top

GETDIFF snapshot /commitlast

IFERROR= $ERROR_READ_EOF GOTO done

IFNUM= %difffileid $DIFF_FILE_NOT_FOUND GOTO loop_top ;; won't ever happen

SENDFILE %difffilepath /copydirs /archive

GOTO loop_top

:done

FTPLOGOFF

EXIT

If you wanted to develop a similar script for downloading files you would use the FTPDIFF command to create a database containing the remote directory listing and FTPGETDIFF to loop through each remote file.

Related command(s): DIFF, GETDIFF, FTPDIFF, FTPGETDIFF

See also: %difffileid, %difffilename, %difffilepath, and %difffiletext, %ftpdifffileid, %ftpdifffilename, %ftpdifffilepath, and %ftpdifffiletext, Using the built-in database engine