There is quick guide to limit SQL Memory on TMG or any other MSDE/SQL Express instance:
Open CMD and start osql -E -S localhost\MSFW
Paste that commands for 512MB limit:
USE master
GO
EXEC sp_configure ‘show advanced options’, 1
RECONFIGURE WITH OVERRIDE
GO
EXEC sp_configure ‘max server memory (MB)’, 512
GO
EXEC sp_configure ‘show advanced options’, 0
RECONFIGURE WITH OVERRIDE
GO
More info can be find on: