I spent four months building a systems framework in C because I essentially backed myself into a corner.
It started with my capstone project. I was building an adaptive metrics daemon for a Raspberry Pi, and I had all these grand, theoretical plans mapped out in my head. We’re talking Kullback-Leibler divergence triggers, Shannon entropy sampling budgets, and Markov transition matrices to act as behavioral baselines. I wanted to monitor system events in real-time and dynamically adjust how much data we were keeping based on how weird the system was acting. It was going to be elegant.
But before you can do any of that fancy math, you have to actually ingest and process the data.
So, I started where basically everyone starts: batch processing. I read the MapReduce chapter in Operating Systems: Three Easy Pieces, looked at some log frequency analyzers, and built a batch pipeline. And to be fair, it worked beautifully—if your logs were just sitting quietly in a static file.
But when I tried to point it at a live system, everything fell apart.
MapReduce is brilliant for offline analytics because you can gather all your data, wait a bit, map it, reduce it, and get a neat little answer. But telemetry doesn’t wait. System logs aren’t static files; they’re a firehose. My poor Raspberry Pi was choking because batch processing creates massive latency and forces you to completely reset your state every time a window closes. You lose all the context between the seconds, and when you’re trying to do anomaly detection, that context is literally the only thing that matters.
I didn’t need a batch processor. I needed a streaming spine.
I went looking for one in the C ecosystem and found… well, almost nothing. Everything out there was either a massive JVM behemoth like Kafka or a heavy Python wrapper. So, as a second-semester student who barely knew what a socket was a few months ago, I decided to just build it myself.
I called it Trafilo.
Global Locks Are a Trap
When you start writing concurrent C for the first time, you quickly realize that your biggest enemy isn’t the logic itself—it’s thread contention.
My first instinct was to just slap a global mutex on the hashmap. One lock to rule them all. But think about it: if every single worker thread has to fight for the exact same lock just to update a counter, your performance actually scales in reverse. You add more threads, and the system just gets slower while they all wait in line. I briefly considered locking every individual node in the hashmap, but that turned into a complexity nightmare of finding and unlocking pointers that I really didn’t want to debug.
The actual sweet spot ended up being per-bucket sharding.
I used the FNV-1a algorithm to hash incoming keys into buckets, and I gave each bucket its own specific mutex. This changed the entire feel of the project.
You see, most telemetry follows a power law. A couple of really noisy services (like your web server or database) produce 90% of the logs, while everything else is pretty quiet. With per-bucket sharding, those hot, noisy keys get their own serialized lane. Meanwhile, the cold keys share the tiny overhead of finding an uncontended bucket. It was the first time the system actually felt “right”—the architectural design finally matched the physical shape of the data.
// The "find or create" pattern that keeps Trafilo fast.
// We lock only the specific bucket we need, not the entire world.
bucket_node *hashmap_find_or_create(hashmap_t *hashmap, const char *key) {
size_t bucket_idx = bucket_index(hashmap, key);
pthread_mutex_lock(&hashmap->locks[bucket_idx]);
bucket_node *current = hashmap->buckets[bucket_idx];
while(current != NULL) {
if(strcmp(key, current->key) == 0) {
return current; // Lock held; the worker proceeds in its own lane
}
current = current->next;
}
// ... creation logic ...
}The Thread Architecture
Before we talk about what went wrong, let’s look at how the data actually flows through the system. Here is the entire concurrency model in action:
The network listener dumps raw strings into the bounded queue. The workers pull from the queue, hash the keys, and acquire the locks for their specific buckets. The buckets manage their own sliding windows, which eventually flush data out to the user’s sink callback. Simple, right?
Well, not exactly.
Things That Go Bump in the Threads
Building the bounded queue was where the real pain started. I needed a ring buffer to sit between my UDP listener pushing data in and my worker pool pulling data out. I used condition variables—not_empty and not_full—to manage all the signaling.
It sounds super simple on a whiteboard, right? In practice, it’s an absolute minefield.
For instance, I spent days chasing down a bug where the shutdown sequence would just hang forever. It turned out I was broadcasting the not_full signal to my worker threads instead of not_empty. The workers would dutifully wake up, check the queue, see that it was empty, and go right back to sleep. The system wouldn’t die; it just hovered there in a zombie state indefinitely.
Then there was the strtok disaster. I was parsing incoming log strings in my worker threads using the standard C strtok function. Out of nowhere, I started seeing total garbage tokens showing up in my output. After a lot of digging, I learned that strtok actually maintains an internal static state pointer. So in a multi-threaded environment, one worker’s parsing routine was actively corrupting another worker’s string. Switching to the thread-safe strtok_r was a one-line fix, but tracking it down cost me about six hours of my sanity.
Honestly, I wouldn’t have found half of these issues if it weren’t for Helgrind. If you’re writing threaded C and you aren’t using Valgrind’s race detector, you’re basically just flying blind. Helgrind caught stupid mistakes—like allocating the size of a pointer instead of the size of a struct—and it validated my lock ordering so I could actually sleep at night.
Sliding Windows (Because Tumbling is for Gymnasts)
The most debated part of the design in my own head was the windowing logic. Most of the simpler frameworks use “tumbling” windows, fixed time intervals that just reset when the clock strikes.
But for my KL divergence triggers, tumbling windows were way too jagged. Think about it: if a massive burst of error events happens right at the 59-second mark, a tumbling window splits that burst in half. Part of it goes into the old window, part goes into the new one, and your anomaly detector completely misses the spike.
So, I moved to a sliding window, which I implemented as a linked-list deque of timestamps. Every time a new event arrives, the framework evicts the old timestamps from the head and appends the new one to the tail.
This gave the system a continuous retention horizon. The signal smoothed out completely, and all those weird boundary artifacts vanished. Because each bucket in the hashmap owns its own sliding logic, the framework became totally decoupled from some rigid, global clock. It finally acted like a living, real-time system.
Proving It Actually Works
I couldn’t just claim the framework worked, I had to prove it. So I rebuilt with it HiveParser.
It’s an ncurses-based TUI that acts as a top-N log analyzer, using Trafilo as its internal engine. I wrote a quick parse callback to extract service names, a handle callback to check the payload for error keywords like “fatal” or “denied”, and a sink callback to update the TUI when the windows emitted.
The moment it all finally clicked was when I fired up my Docker Compose testing harness. I had four dummy daemons blasting fake logs at Trafilo over UDP. Instantly, I saw sshd auth attempts and kernel logs filling up the dashboard in real-time. I flipped a local environment variable to artificially spike the error rate on one of the background services, and the dashboard row flashed red in less than a second.
Trafilo just stayed out of the way, quietly handling the threads and the sliding windows, while HiveParser did all the actual analysis.
I’m moving back to my capstone now. Trafilo is the exact streaming spine I needed, and it completely unblocked the control loop logic for my sampling budgets. It’s no longer just a theory paper, it’s a running, breathing system on my Raspberry Pi.
- GitHub: Basliel25/trafilo
- Documentation: Trafilo Docs