When it comes to detecting OS X malware, the future may not be rooted in machine learning algorithms, but patterns and heatmap visualization, a researcher posits.
In an academic paper published by Virus Bulletin on Monday, Vincent Van Mieghem, a former student at the Delft University of Technology in the Netherlands, describes how a recurring pattern he observed in OS X system calls can be used to indicate the presence of malware.
Van Mieghem wrote the paper, “Behavioral Detection and Prevention of Malware on OS X,” (.PDF) while interning at Fox-IT but has since moved on to become a pentester for PricewaterhouseCoopers’ cybersecurity clients.
Thanks to @staatsgeheim @patrickwardle @osxreverser and @claud_xiao. https://t.co/Jzfj8AbRxM
— Vincent Van Mieghem (@_vivami) September 12, 2016
By the numbers, the detection method Van Mieghem concocted is a success; it detected infections from 100 percent of malware samples found on OS X systems at the time. The method apparently leaves little room for error too; it resulted in a scant 0 percent to 20 percent false positive rate, depending on the user, according to the paper.
The method is based on the analysis of system calls. By using manual sequential analysis and heat map visualizations to keep track of call traces Van Meighem saw a malicious pattern emerge. The researcher said the detection scheme relies on looking for the difference between malicious and benign call traces.
“The goal is to extract recurring patterns that are present in malicious system call traces, but absent from benign system call traces. Such patterns may then be used to identify a malicious process,” he writes in the paper.
Using a fully patched virtual machine running El Capitan, OS X 10.11.3, Van Mieghem tested 23 functional malware samples and logged the resulting raw system calls. He wrote the paper for his master thesis from November 2016 till July 2016 and tried to find as many malware samples as he could; many were donated by Synack’s director of research and known OS X hacker Patrick Wardle and Palo Alto Networks’ Claud Xaio.
Wardle released a generic detector earlier this summer, but for ransomware – RansomWhere? – that monitors machines for untrusted processes.
After logging calls, Van Mieghem input data points associated with each malware’s calls and processes figures into the x- and y-axis of a heatmap. The number of calls per system call per process helps form a pattern. A glaring, recurring pattern, a high number of execution calls to shells carried out by malicious processes, materialized.
“The most prominent malicious pattern derived from the analysis phase to detect presence of malware on a system is the extraordinary number of execution calls (to shells) performed by malicious processes,” he writes.
Van Mieghem looked at some of the earliest malware to hit OS X, such as Flashback, the Trojan from 2011 that targeted OS X, along with 2014’s WireLurker, for the first half of his research. After he identified the execution call pattern in February, Van Mieghem looked at OS X malware uncovered this year, such as the Trojan KeRanger and the adware Pirrit, and confirmed the same malicious pattern.
While not all of the samples elicited an execution call – OSX.Crisis didn’t because Van Mieghem didn’t have a complete and functional sample of the Trojan – most of the ones he looked at did. 
The pattern was clear, according to the researcher.
“The large number of execution system calls to shell processes in all malware samples can clearly be seen,” Van Mieghem says, later adding “The use of execution system calls and interactions with shells and auto-run services appears to be an accurate indication of malware on a system.”
The false positive largely stems from so-called “power users” who may run compilers and interprets such as Python and JavaScript. Since those platforms typically allow for calls to binaries, and executions to shell processes, they may trip Van Mieghem’s detection pattern.
“The only drawback is the false positive rate,” Meighem told Threatpost, “On power users/developer systems, this detection method implicates false positives. This is the weak spot of the detection method.”
While he acknowledges the importance of algorithms, for his research Van Mieghem wanted to get away from them and find a better “real time” detection method.
“Algorithms will always play a huge role in behavioral detection,” Van Mieghem told Threatpost Tuesday, “The goal of my work was to create a method that in real time detects malware and does not rely on machine learning algorithms, since research has shown this to be not feasible in real time client side.”
He refers to the his techniques as “durable and reusable” and believes that in the future, they could be used to extract other patterns from system call traces.
“I think there is a lot of low hanging fruit in OS X malware detection,” Van Meighem said, “Characteristics which are easily detectable, but are not yet found due to the absence of researchers investigating in OS X malware.”
“Signature checking alone to detect malware samples will not be sufficient. In the future, I think AV system will increase to move towards algorithmic/heuristic detection methods.”
Wardle, who helped provide Van Mieghem with malware samples, discussed the glib state of OS X malware detection in a talk at Black Hat 2015, calling the majority of OS X malware “amateur and prosaic,” urging Apple to be more proactive with its security features.
A year later it can be argued that OS X malware still isn’t as robust as Windows malware families, but it is on the upswing. As Van Meighem points out in the paper, five times more OS X malware appeared last year than the previous five years combined.
Last week researchers with Kaspersky Lab identified an OS X variant of a family of backdoors. The variant, Mokes, can capture audio, take screenshoots, monitor machines for certain files and allow attackers to execute arbitrary commands.
