Security tests are black or white box tests meant to test possible attack vectors on a given system.
Security tests are deterministic and mostly realized as system tests.
An simple example would be to test for an SQL injection on an input field exposed to the user.