SecureTransport - File Retention

December 06, 2012

SecureTransport - File Retention

"Out of the box" SecureTransport (ST) does not provide a means to apply any data retention policies to delete files that have been transferred after a specified period of time. While you could request users delete files after they have transferred them, it’s preferable to implement an automated policy. Axway’s custom development group can provide this functionality at an additional cost. However, an alternative is to roll your own script for this.

As I have deployed ST on Windows the two scripts are based on a Windows based deployment although the first could also be used on Linux. You should modify and test these scripts to suit your own environment and use case.

Option 1 - Cygwin Shell Script

SecureTransport on Windows relies on a number of components which run under Cygwin – Axway don’t seem to have fully ported the application to Windows yet. ) The script provided below can be executed as a cron job every day which will delete all files with a modification time older than 30 days (or whatever retention policy you wish to implement).

Update:
I realised I forgot to post the link to the shell script (oops), but the code is now on GitHub at https://github.com/vijayjt/SecureTransportScripts

You will want to edit the file to change the TARGET_DIR variable to point to where the user home folders reside and the LOG_PATH variable to SecureTransport's var\logs directory. In doing so you'll have to enter Cygwin style paths i.e. /drives/c/some/path/ rather than Windows paths C:\some\path.

The file should be copied to the following directory (where D:\ should be changed to the drive where ST is installed)

D:\Program Files (x86)\Tumbleweed\SecureTransport\STServer\bin\

Next the script needs to be scheduled to run by creating a cron job entry:

Open a Windows command prompt
Type “D:\Program Files (x86)\Tumbleweed\SecureTransport\cygwin\”
Type Cygwin.bat to open a Cygwin shell
Type cd /var/cron/tabs
Type chmod 677 SYSTEM

This will change the permissions of the file so you can save your changes. While you can do this from windows, it's simpler to do it from a shell because in Windows you'll have to take ownership before you can give the administrator user write permissions.

Type “vim SYSTEM” to edit the file

After changing the permissions you can edit the file with wordpad / notepad but it is preferable to use vim from within the shell to avoid any DOS vs UNIX file format issues that lead to cron being unable to read the file.

Type “shift-g” to go to the last line of the file
Type the letter ‘o’ to enable editing mode and enter a new line
Type the following line to the file to schedule the script to run on the 28^th day of every month at 11:30 pm:

 30 23 * 28 * "/drives/d/Program Files (x86)/Tumbleweed/SecureTransport/STServer/bin/fileretention.sh" >> /tmp/ fileretention.out 2>&1

This should all be on one line.

Type ‘:wq’ to save the file and exit
Type exit

Restart the cron service after making changes to this file. To do this, start Task Manager, select the Services tab and find the cygwin_cron service in the list, right-click it, and select Stop Service and then select it again and select Start Service.

Option 2 - PowerShell Script

The advantage of PowerShell over the Cygwin shell script is that it is simpler to modify and extend as necessary because it does not rely on the Cygwin environment used by SecureTransport.

The script provided at the end of this post can be added as a Windows Scheduled Task that runs every day. The script is a modified version of a script written by Marcus Lerch - so all credit goes to Marcus. The only modification being the addition of a log file parameter that allows you to log all files that have been deleted to a file.

The script uses a number of parameters as shown below (refer to the comments within the script for details of the meaning / purpose of the parameters). The example below assumes the user home folders are under the path "E:\PathToHomeFolders".

 .\Remove-Files.ps1 -Path “E:\PathToHomeFolders\” –Recurse -KeepDays 30 –LogFile D:\scripts\remove-files-log.txt

The scheduled task will need to run with appropriate permissions to allow the script to delete file. The following should be entered in the Program field of the scheduled task creation wizard:

 C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

The following should be entered in the in the Add argument (optional) field:

  -NoProfile -ExecutionPolicy RemoteSigned -Command "& d:\scripts\Remove-Files.ps1 -Path “E:\PathToHomeFolders\” –Recurse -KeepDays 30 –LogFile d:\scripts\remove-files-log.txt; exit $LASTEXITCODE"

The –NoProfile is used to make sure that nothing in the PowerShell profile interferes with the task. If the Execution Policy does not allow running scripts the -ExecutionPolicy parameter allows an exception to be made for this task only. The –Recurse parameter will cause the script to recursively inspect sub-folders. The –KeepDays parameter specifies only to keep files for up to 30 days if a file is older than that it is deleted. The –LogFile parameter causes a record of the files that are deleted to be kept in the specified log file, in this case remove-files-log.txt, this file must be created before executing the script. After 60 days the entries in the log file will be overwritten to prevent he file from becoming too large. The last portion “exit $LASTEXITCODE”, causes the exit code from the script to be reported to the Windows Task Scheduler.

PowerShell Code

<#
.SYNOPSIS
Script to delete old and/or obsolete files
.DESCRIPTION
The script Remove-Files is used to remove old and/or obsolete files from the
target directory. It can delete files that are older than a specified amount of
time or older than a specified date or it can delete the oldest files in the
directory keeping any number of recent files.
A filter for specific files can be applied, so it is possible to delete only a
certain type of files.
.PARAMETER KeepFiles
Sets the number of files to keep. In combination with KeepDays or Date it
preserves at least the number of files set.
.PARAMETER Path
Directory path where the files should be deleted. If the path contains blanks it
needs to be enclosed in double quotation marks
.PARAMETER Date
Removes the files created before the specified date
.PARAMETER KeepDays
Removes the files older than specified days before actual date
.PARAMETER Recurse
Removes the files in the specified location and in all subdirectories
.PARAMETER Extension
Removes only files that have the specified extension, for instance *.log
.PARAMETER LogFile
Writes the full path to the the files deleted in the specified log file
.Example
.\Remove-Files.ps1 -Path .\Testing -KeepFiles 3 -Recurse
Description
-----------
Removes all but the newest 3 files from the directory .\Testing and all subdirectories
.Example
.\Remove-Files.ps1 -Path .\Testing -KeepDays 7
Description
-----------
Removes all files older 7 days from the directory testing
.Example
$date = (get-Date).AddMonth(-1)
.\Remove-Files.ps1 -Path .\Testing -Date $date -Extension *.txt
Description
-----------
Removes all textfiles older than one month from directory .\Testing
.NOTES
File Name : Remove-Files.ps1
Author : Marcus Lerch
Source : http://gallery.technet.microsoft.com/scriptcenter/Remove-old-files-from-053499f9)
Modified by : Vijay Thakorlal
Modifications : Added LogFile parameter and logging capability
#>
#region Parameter
[cmdletBinding(SupportsShouldProcess=$true)]
param(
[Parameter(Position = 0, Mandatory = $true)]
[ValidateScript({
$vr = Test-Path $_
if(!$vr){Write-Host "The provided path $_ is invalid!"}
$vr
})][String]$Path,
[String][ValidatePattern("\.[a-z]{2,5}")]$Extension,
[Int]$KeepFiles,
[ValidateScript({
if($Date){$vr = $false; write-host "Parameter error see get-help .\Remove-Files.ps1"}
else{$vr = $true}
$vr
})][Int]$KeepDays,
[ValidateScript({
if($KeepDays){$vr = $false; write-host "Parameter error see get-help .\Remove-Files.ps1"}
else{$vr = $true}
$vr
})][DateTime]$Date,
[ValidateScript({
$vr = Test-Path $_ -PathType leaf
if(!$vr){Write-Host "The provided logfile $_ is does not exist!"}
$vr
})][String]$LogFile,
[int]$FileSize=0,
[Switch]$Recurse
)
#endregion
## Clear out the log file after say 60 days to prevent the file from getting too big
$LogFileProperties = Get-Item $LogFile
$OlderThanXDays = 60
$NumDaysOld = ( (Get-Date) - $LogFileProperties.CreationTime).Days
if ( $NumDaysOld -gt $OlderThanXDays )
{
Write-Host "Log file is $NumDaysOld days old, overwriting contents of the log file..."
Write-Output "" | Out-File -FilePath $LogFile
# Reset file creation time, otherwise the file will be overwritten on the next run of the script
$LogFileProperties.CreationTime = Get-Date
}
Write-Output "$(Get-Date) : Starting file retention script run" | Out-File -FilePath $LogFile -Append
#region Functions
function Remove-BeforeDate
{
param(
[Parameter(Mandatory = $true)][DateTime]$TargetDate,
[Parameter(Mandatory = $true)][Int]$Keep,
[Parameter(Mandatory = $true)][Object[]]$Files
)
$Files = $Files | sort-object -Property PSParentPath,LastWriteTime -Descending | Group-Object -Property PSParentPath
$FileList = @()
foreach ($Group in $Files){
$i=1
Foreach ($item in $Group.Group){
if($Item.LastWriteTime -lt $TargetDate -and $i -gt $Keep){
$FileList += $Item
}
$i++
}
}
return $FileList
}
function Remove-Oldest
{
param(
[Parameter(Mandatory = $true)][Int]$Keep,
[Parameter(Mandatory = $true)][Object[]]$Files
)
#Select all files except the most recent specified number of files
$Files = $Files | sort-object -Property PSParentPath,LastWriteTime -Descending | Group-Object -Property PSParentPath | Where-Object {$_.count -gt $KeepFiles}
$FileList = @()
foreach ($Group in $Files){
$i=1
foreach ($Item in $Group.Group){
if ($i -gt $Keep){
$FileList += $Item
}
$i++
}
}
return $FileList
}
function Remove-Files
{
foreach ($File in $Files2Delete){
if($pscmdlet.ShouldProcess($File.FullName, "Delete File")){
Remove-Item -LiteralPath $File.FullName
Write-Output "$(Get-Date) : Deleting file $($File.FullName)" | Out-File -FilePath $LogFile -Append
}
}
}
#endregion
#region Main
#Get the files
if ($Recurse){$RecOption = "-Recurse"}
if ($Extension){$ExtOption = "-Filter *$Extension"}
$strGetChild = "Get-ChildItem -Path `"$Path`" $RecOption $ExtOption | Where-Object {`$_ -is [System.IO.FileInfo]}"
$cbGetChild = [scriptblock]::Create($strGetChild)
$Files = @(Invoke-Command -ScriptBlock $cbGetChild)
$Files = $Files | Where-Object {$_.Length -ge $FileSize}
if($KeepDays){
#Select the files before the specified date
$Files2Delete = @(Remove-BeforeDate -TargetDate ((Get-Date).AddDays(-$KeepDays)) -Files $Files -Keep $KeepFiles)
}
elseif($Date){
#Select the files before the specified date
$Files2Delete = Remove-BeforeDate -TargetDate $Date -Files $Files -Keep $KeepFiles
}
else{
$Files2Delete = Remove-Oldest -Keep $KeepFiles -Files $Files
}
if ($Files2Delete -eq $null -or $Files2Delete.Length -eq 0){
#Write-Host "Nothing to delete!"
Write-Output "$(Get-Date) : No files found to delete!" | Out-File -FilePath $LogFile -Append
}
else {
Remove-Files $Files2Delete
}
Write-Output "$(Get-Date) : Completed file retention script run" | Out-File -FilePath $LogFile -Append
#$Files2Delete
#endregion

Search This Blog

Vijay Thakorlal's Blog