Now, there are a plethora of tricks which have been documented

Checking NTGlobalFlag

The NtGlobalFlag is located in the Process Environment Block (PEB) around offset 0x68 on x86 Windows, and at offset 0xBC on x64 Windows.

The default value is always 0, and doesn’t change when a debugger is attached to the process. There are several methods in which the NtGlobalFlag can be changed to detect the presence of a debugger. The NtGlobalFlag contains many flags which affect the running of a process.

The most common flags which are set with NtGlobalFlag when a debugger creates a process are the following -

NtSetInformationThread() - Thread Hiding:
The NtSetInformationThread() and ZwSetInformationThread() function has a hidden undocumented parameter called ThreadHideFromDebugger (0x11), which can be used to disallow the program from sending any debugging event to the debugger. Can easily detect this function using PE studio/explorer etc.

Scanning System Artifacts
This is the usual bread and butter of malware and even legitimate programs (read: PC games), where it constantly check for software interrupts such as INT3 in memory
Performing code checksum - Another run of the mill technique is doing a checksum of the whole code and comparing it with a predefined value stored somewhere in memory.

TLS callback
Instead of writing about these i thought i’d link to already written quality articles below -

GetLocalTime, GetSystemTime, timeGetTime, NtQueryPerformanceCounter - typical timing functions which while debuggers run they there would be a delay

Now one cool anti-debugging trick seen in a couple of strains of malware is that it would check at some specific interval is the mouse pointer is moving, which would signify that it might not be running in a sand boxed automated environment/

The guy at infosecinstitute has already blogged about this piece of code which does all the above mentioned things in the post (mostly). So I will just leave it here-

Further references can be obtained from the following -

[2] [See pages 43-47]

The second link is a must read as well as enlightening.