In a Transactional Execution if any test fails the execution is aborted and all remaining tests are marked as blocked.

For example, let's suppose we have tests A, B, C that are run in this order. If Transactional Execution option is marked and test A fails then A is marked with result status 'Failed'. And the tests B and C are not execute but have their result status updated to 'Blocked'.

This option is useful in situations where you need to save resource if any test fails or when you have a scenario where the results of a test execution provide information for the others tests.

For further information see TestLink plug-in Hudson Wiki page http://wiki.hudson-ci.org/display/HUDSON/TestLink+Plugin.