Overview of Logstash Plugins

2023-10-21

For your reference, below is a list of the articles in this series.

  1. Introduction to Logstash
  2. Overview of Logstash plugins (this article)
  3. Shipping Events to Logstash

Inputs create events, Filters modify the input events, and Outputs ship them to the destination. Inputs and outputs support codecs, enabling you to encode or decode data as it enters or exits the pipeline without the need for a separate filter.

Logstash employs in-memory bounded queues between pipeline stages by default (from Input to Filter and from Filter to Output) to buffer events. However, if Logstash terminates abruptly, any events stored in memory will be lost. To mitigate data loss, you can enable Logstash to persist in-flight events to disk by utilizing persistent queues.

To enable persistent queues, set the queue.type: persisted property in the logstash.yml file, located in the logstash-7.15.0/config folder.

Viewing Installed Logstash Plugins

To see the list of currently installed Logstash plugins, use the following command:

1
bin/logstash-plugin list

Output:

1
2
3
4
5
logstash-codec-avro
logstash-codec-cef
logstash-codec-json
...

Installing Logstash Plugins

To install Logstash plugins, you can use the following command:

1
2
bin/logstash-plugin install logstash-output-email

Input Plugins

Input plugins configure a set of events to be fed into Logstash, allowing you to specify one or more input sources. Here are some of the available input plugins:

1
2
3
4
5
6
7
8
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-jms

Output Plugins

Output plugins are used to send data to a destination. They offer the flexibility to configure single or multiple output sources. Some of the available output plugins are:

1
2
3
4
5
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-pipe
logstash-output-redis

Filter Plugins

Filter plugins are used to transform data. You can combine multiple filter plugins, and the order in which they are applied determines the sequence of transformations. Here are some of the available filter plugins:

1
2
3
4
5
6
7
8
9
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop

Codec Plugins

Codec plugins are used to encode or decode incoming or outgoing events from Logstash. Codecs can be used in input and output configurations. Input codecs decode data before it enters Logstash, and output codecs encode data before it leaves Logstash. Some of the available codec plugins include:

1
2
3
4
5
6
7
8
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk

Data Types for Plugin Properties

  1. Array:

    • An array is a collection of values for a property.
    • Example: path => ["value1", "value2"]

  2. Boolean:

    • A boolean value is either true or false (without quotes).
    • Example: periodic_flush => false

  3. Codec:

    • Codec is not a data type but a way to encode or decode data at input or output.
    • Example: codec => "json"
    • This instance specifies that this codec, at output, will encode all output in JSON format.

  4. Hash:

    • A hash is a key-value pair collection specified as "key" => "value".
    • Multiple values in a collection are separated by a space.
    • Example:
      1
      2
      3
      4
      match => {
      "key1" => "value1"
      "key2" => "value2"
      }

  5. String:

    • String represents a sequence of characters enclosed in quotes.
    • Example: value => "Welcome to ELK"

  6. Comments:

    • Comments begin with the # character.
    • Example: # This represents a comment

Logstash Conditionals

Logstash conditionals are used to filter events or log lines under certain conditions. Conditionals in Logstash are handled like other programming languages and work with if, if else, and else statements. Multiple if else blocks can be nested.

Syntax for conditionals:

1
2
3
4
5
6
7
8
9
if <conditional expression1> {
  # Some statements here.
}
else if <conditional expression2> {
  # Some statements here.
}
else {
  # Some statements here.
}

Conditionals work with comparison operators, boolean operators, and unary operators:
These conditionals allow you to apply specific actions to events based on specified conditions in your Logstash configuration.

  • Comparison operators include:
    • Equality operators: ==, !=, <, >, <=, >=
    • Regular expressions: =~, !~
    • Inclusion: in, not in
  • Boolean operators include and, or, nand, xor
  • Unary operators include !

Example:

1
2
3
4
5
filter {
  if [action] == "login" {
    mutate { remove => "password" }
  }
}

Exploring Plugin Documentation

To learn more about each of these Logstash plugins, you can visit the official documentation:

  1. https://www.elastic.co/guide/en/logstash/8.10/input-plugins.html
  2. https://www.elastic.co/guide/en/logstash/8.10/output-plugins.html
  3. https://www.elastic.co/guide/en/logstash/8.10/filter-plugins.html
  4. https://www.elastic.co/guide/en/logstash/8.10/codec-plugins.html

This concludes our simple introduction to Logstash plugins!