If a filename is specified in the 'XML transform file' field on the Output tab of the NUnit action, that value is passed in the nunit-console /transform command-line parameter. If that is not working as expected, you would need to contact the NUnit authors to troubleshoot the problem (you can check the 'Log command-line' field on the Advanced tab and rebuild the step to determine the command-line being used).
http://www.nunit.org/index.php?p=con...ndLine&r=2.4.8