Time-Based CAN Intrusion Detection Benchmark

AURC-PR comparison
Performance comparison based on PRC. (a) with (b) without outliers. The Binning method performs best overall.


Modern vehicles are complex cyber-physical systems made of hundreds of electronic control units (ECUs) that communicate over controller area networks (CANs). This inherited complexity has expanded the CAN attack surface which is vulnerable to message injection attacks. These injections change the overall timing characteristics of messages on the bus, and thus, to detect these malicious messages, time-based intrusion detection systems (IDSs) have been proposed. However, time-based IDSs are usually trained and tested on low-fidelity datasets with unrealistic, labeled attacks. This makes difficult the task of evaluating, comparing, and validating IDSs. Here we detail and benchmark four time-based IDSs against the newly published ROAD dataset, the first open CAN IDS dataset with real (non-simulated) stealthy attacks with physically verified effects. We found that methods that perform hypothesis testing by explicitly estimating message timing distributions have lower performance than methods that seek anomalies in a distribution related statistic. In particular, these “distribution-agnostic” based methods outperform “distribution-based” methods by at least 55% in area under the precision-recall curve (AUC-PR). Our results expand the body of knowledge of CAN time-based IDSs by providing details of these methods and reporting their results when tested on datasets with real advanced attacks. Finally, we develop an after-market plug-in detector using lightweight hardware, which can be used to o deploy the best performing IDS method on nearly any vehicle.

Significance and Impact

  • We benchmark some intuitive, straightforward approaches to time-based detection of fabrication attacks. The methods chosen aim to identify and compare the statistical concepts foundational to time-based CAN IDS methods on a high fidelity dataset. To the best of our knowledge, an analysis benchmarking several different time-based IDS methods has not yet been thoroughly performed.
  • Our results suggest that counting methods are perhaps slightly better than those that consider inter-message timing, and more dramatically, the heuristic/distribution agnostic methods far outperform those that explicitly seek tails of an estimated distribution. Inspecting the testing data reveals an artifact of flam attacks (that send one illegitimate message 5 just after a legitimate message); i.e., they force the attack inter message times to be a bi-modal distribution.

Research Details

We used the ROAD dataset (https://0xsam.com/road/) that involve a fully compromised ECU introduced to the CAN bus using the OBD-II port. The ROAD Dataset is the first open dataset with real (non-simulated), stealthy (using flam delivery) fabrication attacks that have physically verified effects on the vehicle. These characteristics make the ROAD dataset ideal for benchmarking time-based IDS methods. The ROAD dataset consists of 12 ambient captures (log files) containing about three hours of ambient (non-attack) data and 33 attack captures that last in total about 30 minutes. Table I lists contents of attacks and logs in the ROAD dataset, and indicates the subset of logs used in this paper. The ROAD dataset contains several types of attacks (see Table I, bottom): (1) fabrication attacks, including fuzzing attacks and several different targeted ID attacks using flam delivery; (2) masquerade attacks and (3) an advanced “Accelerator” attack. Note that (2) and (3) do not alter timing characteristics and are thus out of scope for this paper. For thorough testing, several of the attacks are run and logged more than once. The ROAD data was obfuscated to ensure anonymity of the vehicle while preserving aspects that are needed for testing IDS methods.

We tested four detection methods that exploit the timing regularities of CAN messages: Mean Inter-Message Time, Binning, Fitting a Gaussian Distribution, and Kernel Density Estimation (KDE). While the first two rely on heuristics based on the inter-message times, the latter two follow previous anomaly detection works by fitting a continuous distribution to the inter message times and detecting time gaps with low p-value.


D. Blevins*, P. Moriano*, R. Bridges, M. Verma, M. Iannacone, and S. Hollifield. Time-Based CAN Intrusion Detection Benchmark. Workshop on Automotive and Autonomous Vehicle Security (AutoSec), 2021. DOI: https://dx.doi.org/10.14722/autosec.2021.23013

*Equal contributors.

Last Updated: February 26, 2021 - 9:02 am