At Intouch Insight our logging infrastructure is our holy grail. The engineering team relies on it every day, so we need to keep it up to snuff. I was lucky enough to be able to update our ELK cluster this week to 5.6 - a huge upgrade from our previous stack running ES 2.3 and Kibana 4.

One of the features I was most looking forward to was the dead letter queue that was introduced in Logstash 5.5.

The Problem

The documentation surrounding the usage of the dead letter queue mostly revolves around re-processing rejected events. I wasn’t particularly interested in that use case; I just wanted to be able to easily see when events were rejected.

The Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

input {
    dead_letter_queue {
        path => "/usr/share/logstash/data/dead_letter_queue"
    }
}

filter {
    # First, we must capture the entire event, and write it to a new
    # field; we'll call that field `failed_message`
    ruby {
        code => "event.set('failed_message', event.to_json())"
    }

    # Next, we prune every field off the event except for the one we've
    # just created. Note that this does not prune event metadata.
    prune {
        whitelist_names => [ "^failed_message$" ]
    }

    # Next, convert the metadata timestamp to one we can parse with a
    # date filter. Before conversion, this field is a Logstash::Timestamp.
    # http://www.rubydoc.info/gems/logstash-core/LogStash/Timestamp
    ruby {
        code => "event.set('timestamp', event.get('[@metadata][dead_letter_queue][entry_time]').toString())"
    }

    # Apply the date filter.
    date {
        match => [ "timestamp", "ISO8601" ]
    }

    # Pull useful information out of the event metadata provided by the dead
    # letter queue, and add it to the new event.
    mutate {
        add_field => {
            "message" => "%{[@metadata][dead_letter_queue][reason]}"
            "plugin_id" => "%{[@metadata][dead_letter_queue][plugin_id]}"
            "plugin_type" => "%{[@metadata][dead_letter_queue][plugin_type]}"
        }
    }
}

How it looks

Dead Letter Queue - Logs
Dead Letter Queue - Logs


Dead Letter Queue - Detail
Dead Letter Queue - Detail


Next steps

I’d love to rewrite this as a codec that could be applied to the input, but that’s currently not possible as the dead_letter_queue input plugin doesn’t make any calls to the codec specified in it’s config.