Preventing executable analysis - Part 1, Static Analysis

In this series of posts, I’m going to discuss executable analysis, the methods that are used and mechanisms to prevent them. There are three types of analysis that can be performed on executables:

I’m going to look at each type in detail, giving examples of techniques used in each and ways to make analysis difficult.

In this first post, I’ll look at static analysis. This type of analysis involves parsing the executable and disassembling the code and data contained within it, without ever running it. The benefit of this is that it’s safe, since it’s impossible for the code to cause any damage. The downside is that static analysis can’t really make assumptions about high-level behaviours.

Entry Point Check The first method used to perform static analysis is simple header checks. If the entry point (EP) of the executable resides outside of a section marked as code, it is safe to assume that the application isn’t “normal”. In order to prevent recognising this from being a simple task, the executable should have its BaseOfCode header pointing at the same section the EP is in, even when packed.

Packing Executables are often packed - i.e. their code is encrypted in some way. We can analyse this using entropy calculations on each section, to discover how “random” the data looks. It’s often tempting for authors to try to create a good cipher for encrypting packed sections, but this often leads to a few problems. Firstly, entropy calculations will very quickly spot sections that look too random to be normal code or data. Secondly, there are many applications out there that will look for sequences of data and instructions that match known cryptographic algorithms. It’s relatively easy to spot magic numbers and S-box arrays

In order to prevent this, a packing algorithm should be used that preserves the statistical signature of the original data. A good way to do this is to flip only the lowest two bits of each byte, or to simply shuffle the data rather than encrypting it with xor or a similar operation. By definition, a sample of data will have the same Shannon entropy regardless of how much you shuffle it. The usual way that analysis tools work is to split each section into blocks and compute an entropy graph across the file. By using a cipher that only shuffles bytes that are close, you can achieve an almost identical entropy graph:

Entropy Graph

Since instructions are multi-byte, shuffling completely destroys the code, making it impossible to read. It’s relatively simple to perform half-decent shuffling, given a reasonably large key:

for each byte k in key
{
    tmp = data[0]
    data[0] = data[k]
    data[k] = tmp
}

Simply loop the above over a sequence of data, you’ll get reasonable shuffling within each 256-byte block. OllyDbg doesn’t recognise this as packed, since it works on counts of particularly common bytes in code sections.

Jump Tables Static analysis tools such as IDA Pro work by mapping sequences of jumps together. Some enhance this by performing heuristic analysis of jumps, for example turning jmp [file.exe+0x420c0] into an assumed jump based on the data at file offset 0x420c0. We can try to defeat this type of analysis by using jump tables. These are pointer tables generated at runtime, which are encrypted or obfuscated on disk. Jumps in the code are done by pointing to offsets in the jump table. Often this is further obfuscated by using jumps to register pointers, or stack jumps:


; ecx = function ID
mov eax, [ptrToTable+ecx*4] ; load the encrypted pointer into eax
xor eax, [ptrToKey+ecx*4]   ; xor with the key
push eax                    ; push address to stack
ret                         ; return (jump) to it, obfuscates the jump

Obviously there’s more we can do here - better encryption, values generated at runtime, more obfuscation, etc.

Control Flow Obfuscation Some analysis tools focus on artifacts of compilers - i.e. the signatures of how common high level language constructs translate into assembly language. For example, some loops may be translated into a dec/jg loop, whereas some others might use rep mov. It all depends on the high level construct in use. By altering these constructs and using them in situations where they are unusual, this can confuse heuristics. One example for short loops is using a switch:

for(int i=0; i<5; i++)
{
    if(i%2==0) printf("%i is even\n", i);
    else printf("%i is odd\n", i);
    if(i==4) printf("done");
}

We can turn this into a switch statement that flattens out the flow, instead of being an obvious loop:

for(int i=0; i<5; i++)
{
    switch(i)
    {
        case 0: printf("%i is even\n", i); break;
        case 1: printf("%i is odd\n", i); break;
        case 2: printf("%i is even\n", i); break;
        case 3: printf("%i is odd\n", i); break;
        case 4: printf("%i is even\n", i); printf("done"); break;
    }
}

Since this uses a switch, we can use a jump table that is easy to obfuscate.

Conclusion There are many ways to break static analysis, some of which are simple, some of which are more complex. By employing these, it makes it very difficult for any analyst to decode and understand. Such methods can also prevent automated tools from performing in-depth analysis of the code. Understanding these methods helps both implement them and circumvent them. In the next part, I’ll be looking at virtualised and emulated analysis, which uses virtual hardware to analyse and fingerprint software without actually executing the real application code live on a hardware processor.

Further Reading