Detecting packed/crypted executables with Snort

As a result of some research into various trojans it was identified that majority of them rely on packers, crypters, and anti debugging tricks. Nothing new here. However, what was interesting is that some of them were completely missed by Snort. Rulebase included default signatures that come with base installation, community rules, and Emerging Threats (ET).

It turns out that analyzed trojans modified the executable's header which did not fall within signature's patterns. This can be partly fixed by creating a signature to detect the PE header's magic value within a certain distance from the start of the DOS header. However, the trick here is to pick a distance which will not produce false negatives. If it's too short then it's doomed to miss lots of stuff. Higher values may produce false positives, more research into this is needed.

It's best to detect executables by reading their structure. PE/COFF format states that the offset to start of PE header is found 0x3C bytes from start of DOS header. I was not able to achieve desired results using the regular Snort rule syntax but using the dynamic rules feature worked perfectly. It's written C and must be compiled. Download it here.