After I shipped trafilo I finally had a working streaming spine. Raw log lines could flow through buckets and get windowed and flush to my sink callback in real time. The pipeline was alive.
It was also completely useless.
My capstone project revolves around behavioral anomaly detection. I needed to build Markov transition matrices and calculate Kullback-Leibler divergence against a baseline. The problem is that Markov chains learn transitions between discrete events rather than raw strings. If a log says “Failed password for alice” and the next one says “Failed password for bob” the Markov model sees two entirely different inputs. The transition matrix explodes into pure noise. The anomaly detector just collapses into a random number generator.
I needed something to sit between trafilo and the Markov module. Something that could look at a raw log line and hand back a stable ID. ID then are handed out by the similarity of their states.
I called it dRain. It is a strict implementation of the Drain algorithm by He et al from ICWS 2017.
Templates Before Sequences
Drain mines log templates online in a single pass using a fixed-depth parse tree. For each incoming line it produces a template ID plus the variable parameters it extracted along the way.
So a log for alice from one IP and a log for bob from another IP both become a single template ID pointing to a string with wildcards. Once you have template IDs those sequences actually mean something. The stream from a single SSH session might look like four failed passwords followed by an accept. That is exactly the shape a Markov chain can learn from.
The architectural principle I locked early is the part of this project I am most proud of:
dRain owns structure while Markov owns behavior while the sampler is the one who defines meaning.
Every time I felt tempted to add a semantic event type layer to group templates by what they mean I reminded myself that meaning is what Markov infers from the transitions.
Meaning is not something the parser is supposed to hand it. Drain does not know template 0 is a failure event. It just sees a shape and assigns it a number.
That separation looks pedantic on a whiteboard.
The Rust Detour
C was the obvious choice. Trafilo is C and the FFI surface is C and the capstone runs on a Raspberry Pi where every megabyte matters. The path of least resistance was to write dRain in C and link it as a static library and be done with it.
I wrote it in Rust instead.
I had three reasons.
The first is that the work is string heavy. Tokenizing and regex preprocessing and template comparison make this kind of code significantly less miserable with Rust string machinery than C buffer juggling.
The second reason is that the compiler does what Helgrind does but way earlier. Getting trafilo threading clean cost me a lot of Helgrind runs and a nasty debugging session I still resent. The Rust borrow checker rejects most of those races before the program runs and I wanted to know what that looked like.
The third reason is I wanted the FFI to hurt. This is a multi language portfolio piece. A clean Rust to C boundary with a cdylib crate and a hand written header file and ownership rules that span the language gap is exactly the kind of skill I cannot fake having. Building the friction in on purpose was the point.
The cost was real. I lost about three days fighting box strings versus static strings on the template slot type. I lost another day getting my mutex laid out so the lock was not held across regex matches.
How dRain Actually Works
The internals are simple enough to draw in one diagram:
Each leaf holds a list of templates. A template is a sequence of slots where each slot is either a literal token or a wildcard. When a new tokenized line lands at a leaf dRain scores it against every template using a similarity ratio. The best match above the threshold wins. Any tokens that disagree with the winning template literal slots get promoted to wildcards. Templates get strictly more general over their lifetime and never more specific.
If nothing matches a fresh template is born from the input. That is the whole algorithm. The real work was making it fast and correct under concurrent writes from the trafilo workers.
Preprocessing Is Load Bearing
The first version I ran against the loghub Linux 2k corpus produced 214 templates. The Drain ground truth is around 30.
My first instinct was to drop the similarity threshold and let templates merge more aggressively. That was the wrong instinct. When you are seven times over budget the threshold is rarely the first suspect. I dumped fifty unique templates and started reading.
The timestamp was not in my regex set. Every minute of the corpus was sprouting fresh branches in the tree because the seconds kept changing and looked like distinct first tokens after preprocessing. The same thing happened with hostnames I missed and file paths under the var log directory.
Once I added regex patterns for timestamps and paths the Linux 2k run dropped to 34 templates. Perfectly inside the band.
Then I ran OpenSSH 2k expecting around 50 templates. I got 19.
I had the opposite problem. Preprocessing was now too aggressive. It was eating numeric fields that carried real distinguishing information for OpenSSH lines so structurally different events were collapsing into the same template. That is a known tension where the preprocessor is a global hammer and corpora respond to it differently. I filed it as an issue and tagged version 0.1 anyway. The capstone needs good enough templating rather than perfect templating. The job of the Markov model is to be robust to noisy template IDs.
The deeper lesson is one the Drain paper makes explicit and I underestimated until I felt it. Preprocessing is where domain knowledge enters the system. The algorithm itself is generic but the regex set is corpus specific. There is no universally correct preprocessing. There is only what is tuned for this kind of log.
Where Rust Stopped Helping
I expected to never run Helgrind on this project because the borrow checker was supposed to make that unnecessary.
Helgrind ran clean on my own threading. It also produced about thirty pages of warnings about data races inside the regex crate lazy DFA construction. The crate is correct and does its own synchronization but Helgrind cannot read Rust atomic ordering semantics and cannot tell. It was just structural false positives.
I wrote a suppression file targeting the regex internals and moved on. My code was race free and the dependency was race free. The race detector just could not see what the type system had already proved.
The real lesson is more useful. Rust guarantees do not propagate across the FFI boundary and they do not help your runtime tooling see what your compiler sees. The C side of dRain still needed me to think about ownership manually. If you forget to call the free function for the heap allocated params array that dRain hands back through FFI you get a Rust allocated leak that is invisible to C tooling and invisible to the borrow checker. Static guarantees on one side do not magically fix the other side. The interface is a contract written in English at the top of the header file rather than a property the compiler can enforce.
That is the FFI tax. It is not a bug. It is the whole reason this project was a learning win.
What Is Next
Markov is next. dRain hands template IDs over the FFI and Markov consumes them per bucket and learns transition probabilities against a baseline window. That module is also going to be in Rust because I am not undoing the language choice halfway through the pipeline.
After Markov comes the sampler. After the sampler the capstone is just a Raspberry Pi running the full thing in a Docker harness watching itself.
dRain is on GitHub at Basliel25/dRain If you want the algorithm itself read the Drain paper by He et al from ICWS 2017.