SecureTransport - File Retention


"Out of the box" SecureTransport (ST) does not provide a means to apply any data retention policies to delete files that have been transferred after a specified period of time. While you could request users delete files after they have transferred them, it’s preferable to implement an automated policy. Axway’s custom development group can provide this functionality at an additional cost. However, an alternative is to roll your own script for this. 

As I have deployed ST on Windows the two scripts are based on a Windows based deployment although the first could also be used on Linux. You should modify and test these scripts to suit your own environment and use case.

Option 1 - Cygwin Shell Script

SecureTransport on Windows relies on a number of components which run under Cygwin – Axway don’t seem to have fully ported the application to Windows yet. ) The script provided below can be executed as a cron job every day which will delete all files with a modification time older than 30 days (or whatever retention policy you wish to implement).

Update:
I realised I forgot to post the link to the shell script (oops), but the code is now on GitHub at https://github.com/vijayjt/SecureTransportScripts

You will want to edit the file to change the TARGET_DIR variable to point to where the user home folders reside and the LOG_PATH variable to SecureTransport's var\logs directory. In doing so you'll have to enter Cygwin style paths i.e. /drives/c/some/path/ rather than Windows paths C:\some\path.

The file should be copied to the following directory (where D:\ should be changed to the drive where ST is installed)

D:\Program Files (x86)\Tumbleweed\SecureTransport\STServer\bin\

Next the script needs to be scheduled to run by creating a cron job entry:
  1. Open a Windows command prompt
  2. Type “D:\Program Files (x86)\Tumbleweed\SecureTransport\cygwin\”
  3. Type Cygwin.bat to open a Cygwin shell
  4. Type cd /var/cron/tabs
  5. Type chmod 677 SYSTEM
This will change the permissions of the file so you can save your changes. While you can do this from windows, it's simpler to do it from a shell because in Windows you'll have to take ownership before you can give the administrator user write permissions.

  1. Type “vim SYSTEM” to edit the file
After changing the permissions you can edit the file with wordpad / notepad but it is preferable to use vim from within the shell to avoid any DOS vs UNIX file format issues that lead to cron being unable to read the file.

  1. Type “shift-g” to go to the last line of the file
  2. Type the letter ‘o’ to enable editing mode and enter a new line
  3. Type the following line to the file to schedule the script to run on the 28th day of every month at 11:30 pm:
 30 23 * 28 * "/drives/d/Program Files (x86)/Tumbleweed/SecureTransport/STServer/bin/fileretention.sh" >> /tmp/ fileretention.out 2>&1  
       
        This should all be on one line.

  1.  Type ‘:wq’ to save the file and exit
  2.  Type exit


Restart the cron service after making changes to this file. To do this, start Task Manager, select the Services tab and find the cygwin_cron service in the list, right-click it, and select Stop Service and then select it again and select Start Service.


Option 2 - PowerShell Script

The advantage of PowerShell over the Cygwin shell script is that it is simpler to modify and extend as necessary because it does not rely on the Cygwin environment used by SecureTransport.

The script provided at the end of this post can be added as a Windows Scheduled Task that runs every day. The script is a modified version of a script written by Marcus Lerch - so all credit goes to Marcus. The only modification being the addition of a log file parameter that allows you to log all files that have been deleted to a file.

The script uses a number of parameters as shown below (refer to the comments within the script for details of the meaning / purpose of the parameters). The example below assumes the user home folders are under the path "E:\PathToHomeFolders".
 .\Remove-Files.ps1 -Path “E:\PathToHomeFolders\” –Recurse -KeepDays 30 –LogFile D:\scripts\remove-files-log.txt  

The scheduled task will need to run with appropriate permissions to allow the script to delete file. The following should be entered in the Program field of the scheduled task creation wizard:

 C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe  

The following should be entered in the in the Add argument (optional) field:

  -NoProfile -ExecutionPolicy RemoteSigned -Command "& d:\scripts\Remove-Files.ps1 -Path “E:\PathToHomeFolders\” –Recurse -KeepDays 30 –LogFile d:\scripts\remove-files-log.txt; exit $LASTEXITCODE"  

The –NoProfile is used to make sure that nothing in the PowerShell profile interferes with the task. If the Execution Policy does not allow running scripts the -ExecutionPolicy parameter allows an exception to be made for this task only. The –Recurse parameter will cause the script to recursively inspect sub-folders. The –KeepDays parameter specifies only to keep files for up to 30 days if a file is older than that it is deleted. The –LogFile parameter causes a record of the files that are deleted to be kept in the specified log file, in this case remove-files-log.txt, this file must be created before executing the script. After 60 days the entries in the log file will be overwritten to prevent he file from becoming too large. The last portion “exit $LASTEXITCODE”, causes the exit code from the script to be reported to the Windows Task Scheduler.

PowerShell Code



  1. <#  
  2. .SYNOPSIS   
  3. Script to delete old and/or obsolete files  
  4.   
  5. .DESCRIPTION  
  6. The script Remove-Files is used to remove old and/or obsolete files from the   
  7. target directory. It can delete files that are older than a specified amount of   
  8. time or older than a specified date or it can delete the oldest files in the   
  9. directory keeping any number of recent files.  
  10. A filter for specific files can be applied, so it is possible to delete only a  
  11. certain type of files.  
  12.   
  13. .PARAMETER  KeepFiles  
  14. Sets the number of files to keep. In combination with KeepDays or Date it  
  15. preserves at least the number of files set.  
  16.   
  17. .PARAMETER  Path  
  18. Directory path where the files should be deleted. If the path contains blanks it  
  19. needs to be enclosed in double quotation marks  
  20.   
  21. .PARAMETER  Date  
  22. Removes the files created before the specified date  
  23.   
  24. .PARAMETER  KeepDays  
  25. Removes the files older than specified days before actual date  
  26.   
  27. .PARAMETER  Recurse  
  28. Removes the files in the specified location and in all subdirectories  
  29.   
  30. .PARAMETER  Extension  
  31. Removes only files that have the specified extension, for instance *.log  
  32.   
  33. .PARAMETER  LogFile  
  34. Writes the full path to the the files deleted in the specified log file  
  35.   
  36. .Example  
  37. .\Remove-Files.ps1 -Path .\Testing -KeepFiles 3 -Recurse  
  38.   
  39. Description  
  40. -----------  
  41. Removes all but the newest 3 files from the directory .\Testing and all subdirectories  
  42.   
  43. .Example  
  44. .\Remove-Files.ps1 -Path .\Testing -KeepDays 7  
  45.   
  46. Description  
  47. -----------  
  48. Removes all files older 7 days from the directory testing  
  49.   
  50. .Example  
  51. $date = (get-Date).AddMonth(-1)  
  52. .\Remove-Files.ps1 -Path .\Testing -Date $date -Extension *.txt  
  53.   
  54. Description  
  55. -----------  
  56. Removes all textfiles older than one month from directory .\Testing  
  57.   
  58. .NOTES    
  59.         File Name       : Remove-Files.ps1   
  60.         Author          : Marcus Lerch  
  61.         Source          : http://gallery.technet.microsoft.com/scriptcenter/Remove-old-files-from-053499f9)  
  62.         Modified by     : Vijay Thakorlal  
  63.         Modifications   : Added LogFile parameter and logging capability  
  64.           
  65.   
  66. #>  
  67.   
  68. #region Parameter  
  69. [cmdletBinding(SupportsShouldProcess=$true)]  
  70. param(  
  71.     [Parameter(Position = 0, Mandatory = $true)]  
  72.     [ValidateScript({  
  73.         $vr = Test-Path $_  
  74.         if(!$vr){Write-Host "The provided path $_ is invalid!"}  
  75.         $vr  
  76.     })][String]$Path,  
  77.     [String][ValidatePattern("\.[a-z]{2,5}")]$Extension,      
  78.     [Int]$KeepFiles,  
  79.     [ValidateScript({  
  80.         if($Date){$vr = $false; write-host "Parameter error see get-help .\Remove-Files.ps1"}  
  81.         else{$vr = $true}  
  82.         $vr  
  83.     })][Int]$KeepDays,  
  84.     [ValidateScript({  
  85.         if($KeepDays){$vr = $false; write-host "Parameter error see get-help .\Remove-Files.ps1"}  
  86.         else{$vr = $true}  
  87.         $vr  
  88.     })][DateTime]$Date,  
  89.     [ValidateScript({  
  90.         $vr = Test-Path $_ -PathType leaf  
  91.         if(!$vr){Write-Host "The provided logfile $_ is does not exist!"}  
  92.         $vr  
  93.     })][String]$LogFile,  
  94.     [int]$FileSize=0,  
  95.     [Switch]$Recurse  
  96. )  
  97. #endregion  
  98.   
  99. ## Clear out the log file after say 60 days to prevent the file from getting too big  
  100. $LogFileProperties = Get-Item $LogFile  
  101. $OlderThanXDays = 60  
  102. $NumDaysOld = ( (Get-Date) - $LogFileProperties.CreationTime).Days  
  103. if ( $NumDaysOld -gt $OlderThanXDays )  
  104. {  
  105.     Write-Host "Log file is $NumDaysOld days old, overwriting contents of the log file..."  
  106.     Write-Output "" | Out-File -FilePath $LogFile  
  107.     # Reset file creation time, otherwise the file will be overwritten on the next run of the script  
  108.     $LogFileProperties.CreationTime = Get-Date  
  109. }  
  110.   
  111. Write-Output "$(Get-Date) : Starting file retention script run" | Out-File -FilePath $LogFile -Append  
  112.   
  113. #region Functions  
  114. function Remove-BeforeDate  
  115. {  
  116.     param(  
  117.         [Parameter(Mandatory = $true)][DateTime]$TargetDate,  
  118.         [Parameter(Mandatory = $true)][Int]$Keep,  
  119.         [Parameter(Mandatory = $true)][Object[]]$Files  
  120.     )  
  121.     $Files = $Files | sort-object -Property PSParentPath,LastWriteTime -Descending | Group-Object -Property PSParentPath  
  122.     $FileList = @()  
  123.     foreach ($Group in $Files){  
  124.         $i=1  
  125.         Foreach ($item in $Group.Group){  
  126.             if($Item.LastWriteTime -lt $TargetDate -and $i -gt $Keep){  
  127.                 $FileList += $Item  
  128.             }  
  129.             $i++  
  130.         }  
  131.     }  
  132.     return $FileList  
  133. }  
  134. function Remove-Oldest  
  135. {  
  136.     param(  
  137.         [Parameter(Mandatory = $true)][Int]$Keep,  
  138.         [Parameter(Mandatory = $true)][Object[]]$Files  
  139.     )  
  140.     #Select all files except the most recent specified number of files  
  141.     $Files = $Files | sort-object -Property PSParentPath,LastWriteTime -Descending | Group-Object -Property PSParentPath | Where-Object {$_.count -gt $KeepFiles}  
  142.     $FileList = @()  
  143.     foreach ($Group in $Files){  
  144.         $i=1      
  145.         foreach ($Item in $Group.Group){  
  146.             if ($i -gt $Keep){  
  147.                 $FileList += $Item  
  148.             }  
  149.             $i++  
  150.         }  
  151.     }  
  152.     return $FileList  
  153. }  
  154. function Remove-Files   
  155. {  
  156.     foreach ($File in $Files2Delete){  
  157.         if($pscmdlet.ShouldProcess($File.FullName, "Delete File")){  
  158.             Remove-Item -LiteralPath $File.FullName  
  159.             Write-Output "$(Get-Date) : Deleting file $($File.FullName)" | Out-File -FilePath $LogFile -Append  
  160.         }  
  161.     }  
  162. }  
  163. #endregion  
  164.   
  165. #region Main  
  166. #Get the files  
  167. if ($Recurse){$RecOption = "-Recurse"}  
  168. if ($Extension){$ExtOption = "-Filter *$Extension"}  
  169. $strGetChild = "Get-ChildItem -Path `"$Path`" $RecOption $ExtOption | Where-Object {`$_ -is [System.IO.FileInfo]}"  
  170. $cbGetChild = [scriptblock]::Create($strGetChild)  
  171.   
  172. $Files = @(Invoke-Command -ScriptBlock $cbGetChild)  
  173. $Files = $Files | Where-Object {$_.Length -ge $FileSize}  
  174.   
  175. if($KeepDays){  
  176.     #Select the files before the specified date  
  177.     $Files2Delete = @(Remove-BeforeDate -TargetDate ((Get-Date).AddDays(-$KeepDays)) -Files $Files -Keep $KeepFiles)  
  178. }  
  179. elseif($Date){  
  180.     #Select the files before the specified date  
  181.     $Files2Delete = Remove-BeforeDate -TargetDate $Date -Files $Files -Keep $KeepFiles  
  182. }  
  183. else{  
  184.     $Files2Delete = Remove-Oldest -Keep $KeepFiles -Files $Files  
  185. }  
  186. if ($Files2Delete -eq $null -or $Files2Delete.Length -eq 0){  
  187.     #Write-Host "Nothing to delete!"  
  188.     Write-Output "$(Get-Date) : No files found to delete!" | Out-File -FilePath $LogFile -Append  
  189. }  
  190. else {  
  191.     Remove-Files $Files2Delete  
  192. }  
  193.   
  194. Write-Output "$(Get-Date) : Completed file retention script run" | Out-File -FilePath $LogFile -Append  
  195.   
  196. #$Files2Delete  
  197. #endregion  

Comments