LogMerge

A distributed job leaves its logs spread across a dozen files. There are driver logs, executor logs and sidecars, plus rotated files and their .gz archives. LogMerge reads all of them at once and writes every line in timestamp order. Each line can still be traced to its file, and stack traces stay under the line that raised them. On an M1 Max it runs at about 1 GB/s from disk.

The core of it is a min-heap with one entry per open file, keyed by the timestamp of that file's next line. LogMerge pops the earliest file and streams it until its next line would come after the earliest timestamp left in the heap, then pushes it back. Before the merge starts, a goroutine per file parses the first timestamp, so the merge begins with a full heap.

Archives are read in place, and an entry inside one is addressed as archive.tar.gz!/name.log. Timestamps come in many shapes, such as ISO 8601, ctime and date-time-millis, and they all come out at the same width. Nothing is allocated per line.

It is written in Go and has no runtime dependencies. Releases are binaries for Linux, macOS and Windows on amd64 and arm64, built with profile-guided optimisation.