Every hash on the Hashing & Message Authentication page is built so that a single changed bit produces a completely unrelated digest. TLSH (Trend Micro Locality Sensitive Hash) is built to do the opposite: similar inputs produce similar digests, and sgcTLSH_Diff scores how far apart two digests are. It answers "how alike are these two files" rather than "are these two files identical", which is what makes it useful for clustering and near duplicate detection, malware family triage, and finding documents that were copied and lightly edited.
TLSH is not a security hash. Do not use it for integrity or authentication.
It is not collision resistant, and an attacker who controls the input can deliberately steer the digest. Every function is in sgcCrypto_TLSH.
Computing a digest
The input must be at least 50 bytes (cTLSH_MinDataLength) and varied enough to fill more than half the internal buckets; a digest is 72 characters (cTLSH_DigestLength), starting with the version prefix T1. An input that is too short or too uniform (a long run of one repeated byte, for example) does not have enough signal to characterize, and the functions below return an empty string rather than a meaningless digest.
function sgcTLSH(const aData: TBytes): string;
function sgcTLSH_Stream(aStream: TStream): string;
function sgcTLSH_IsValid(const aDigest: string): Boolean;
var
vDigest: string;
begin
vDigest := sgcTLSH(oFileBytes);
if vDigest = '' then
{ too short or too uniform to characterize }
else
{ vDigest is a 72-character T1... string };
end;
A large file does not need to be held in memory at once; feed it through the streaming interface in blocks, or pass a stream directly to sgcTLSH_Stream:
procedure sgcTLSH_Init(var aContext: TsgcTLSHContext);
procedure sgcTLSH_Update(var aContext: TsgcTLSHContext; const aData: TBytes);
function sgcTLSH_Final(var aContext: TsgcTLSHContext): string;
Comparing two digests
function sgcTLSH_Diff(const aLeft, aRight: string; aLengthDiff: Boolean = True): Integer;
Zero (cTLSH_IdenticalDiff) means the same digest; the score grows with how different the two inputs are, with no fixed upper bound. There is no universal threshold, calibrate it against your own corpus, but as a starting point a distance under roughly 30 usually indicates the same or nearly the same content, while higher values indicate progressively less relation. aLengthDiff folds the difference in input length into the score; turn it off when comparing content that is legitimately padded or truncated, such as log files that were rotated at different sizes. The function returns -1 if either digest is malformed.