Fluentd¶
Fluentd is an open source data collector for unified logging layer. It allows you to unify data collection and consumption for a better use and understanding of data.
License¶
Fluentd is released under the Apache-2.0 License.
All relevant legal information can be found here
Prerequisites¶
We’re using Ruby in the stable version 2.6:
[isabell@stardust ~]$ uberspace tools version show ruby
Using 'Ruby' version: '2.6'
[isabell@stardust ~]$
Installation¶
Fluentd can be installed as a ruby gem:
[isabell@stardust ~]$ gem install fluentd --no-doc
Fetching http_parser.rb-0.6.0.gem
Fetching concurrent-ruby-1.1.7.gem
Fetching tzinfo-2.0.2.gem
Fetching serverengine-2.2.1.gem
Fetching sigdump-0.2.4.gem
Fetching yajl-ruby-1.4.1.gem
Fetching msgpack-1.3.3.gem
Fetching cool.io-1.6.0.gem
Fetching fluentd-1.11.2.gem
Fetching tzinfo-data-1.2020.1.gem
Fetching strptime-0.2.4.gem
WARNING: You don't have /home/isabell/.gem/ruby/2.6.0/bin in your PATH, gem executables will not run.
Building native extensions. This could take a while...
Successfully installed msgpack-1.3.3
Building native extensions. This could take a while...
Successfully installed yajl-ruby-1.4.1
Building native extensions. This could take a while...
Successfully installed cool.io-1.6.0
Successfully installed sigdump-0.2.4
Successfully installed serverengine-2.2.1
Building native extensions. This could take a while...
Successfully installed http_parser.rb-0.6.0
Successfully installed concurrent-ruby-1.1.7
Successfully installed tzinfo-2.0.2
Successfully installed tzinfo-data-1.2020.1
Building native extensions. This could take a while...
Successfully installed strptime-0.2.4
Successfully installed fluentd-1.11.2
11 gems installed
[isabell@stardust ~]$
Note
You can ignore the warning about the executables not being in the PATH, Uberspace already took care of that.
To verify the installation, check the version of fluentd:
[isabell@stardust ~]$ fluentd --version
fluentd 1.11.2
[isabell@stardust ~]$
Configuration¶
Create configuration¶
Create the default configuration in the ~/fluentd directory.
[isabell@stardust ~]$ fluentd --setup ~/fluentd
Installed /home/isabell/fluentd/fluent.conf.
[isabell@stardust ~]$
Edit the config file ~/fluentd/fluent.conf and comment out the following lines:
## match tag=system.** and forward to another fluent server
#<match system.**>
# @type forward
# @id forward_output
#
# <server>
# host 192.168.0.11
# </server>
# <secondary>
# <server>
# host 192.168.0.12
# </server>
# </secondary>
#</match>
Setup daemon¶
Create ~/etc/services.d/fluentd.ini with the following content:
[program:fluentd]
directory=%(ENV_HOME)s/fluentd
command=fluentd --config fluent.conf --no-supervisor
After creating the configuration, tell supervisord to refresh its configuration and start the service:
[isabell@stardust ~]$ supervisorctl reread
SERVICE: available
[isabell@stardust ~]$ supervisorctl update
SERVICE: added process group
[isabell@stardust ~]$ supervisorctl status
SERVICE RUNNING pid 26020, uptime 0:03:14
[isabell@stardust ~]$
If it’s not in state RUNNING, check your configuration.
Finishing installation¶
Further adjust the configuration in ~/fluentd/fluent.conf according to your needs (see configuration).
Optional: Expose the HTTP Endpoint¶
To expose the fluentd http endpoint (default port 8888), configure a web backend:
[isabell@stardust ~]$ uberspace web backend set /fluentd --http --port 8888 --remove-prefix
Set backend for /fluentd to port 8888; please make sure something is listening!
You can always check the status of your backend using "uberspace web backend list".
[isabell@stardust ~]$
You can then test this endpoint by sending a GET request to https://isabell.uber.space/fluentd/debug.test?json={%22test%22:%22message%22}.
The json payload should then appear in the fluentd logs:
[isabell@stardust ~]$ supervisorctl tail fluentd
...
2020-09-09 12:10:46.630093423 +0200 debug.test: {"test":"message"}
[isabell@stardust ~]$