2014. augusztus 10., vasárnap

Ready with Ansible

In my last post I wrote about finishing my work on  Puppet module and moving to Ansible. You can reason out why I'm writing these sentences - yes, I'm done (as far as a role in Ansible can be done).


This time I choosed a different approach to develop something new. In the case of Salt and Puppet I mapped their configuration syntax to syslog-ng's sysntax. I wanted to give as much freedom as possible to the users. In this particular case with Ansible, I choosed a different approach.


Ansible is a really powerful tool in the hand of DevOps, sysadmins.  It's easy to use and can easily set up your infrastructure from scratch. It's learning curve is definitely flat, after Puppet I almost say it's horizontal :) It uses YAML files for describing your system (like Salt), and SSH to make changes. It doesn't need an agent to be preinstalled just a Python interpreter and some basic Python modules so it can also be used to set up network devices, like switches and routers. You can use it with public key authentication, but it can also ask for passwords.


You can do ad-hoc things with Ansible, or organize the steps into so called playbooks. For example, you can update the cache of apt, install a cool software then configure it. When you do this a lot of times, maybe you want simplify the process and just say: This computer must act as this _role_. This is the key here: organize the steps into roles, that’s what I did, too.


I didn’t want to write a “yet another modul”, which can emit syslog-ng configuration from itself (to be precise, for Ansible and Salt no one existed, by now). I wanted to write some high level stuff, a role, which can be assigned to computers. IMHO most users don’t want to configure syslog-ng itself, they just want it to run and see the logs or would like to create a central log server possibly with TLS and forward all their logs to there.


I have good news for them, the role is ready for use in Ansible-galaxy. Just install it, and use:
- hosts: servers
  vars:
    syslog_ng_mode: server
    syslog_ng_server_sources:
      - "127.0.0.1":
          proto: udp
          port: 1524

    syslog_ng_server_dest_dir: /var/logg
    syslog_ng_server_file_macro: $YEAR.$MONTH.$DAY/$HOST.log

  remote_user: root
  roles:
    - {role: ansible-syslog-ng }


By using this role, you get syslog-ng  installed with the default settings on your machines, this is called local mode. In this mode syslog-ng doesn’t receive logs from network, just local sources, like internal() or system(). You can change this behavior by setting syslog_ng_mode to server, client or manual.


In server mode syslog-ng creates network sources (currently TCP, UDP and TCP-TLS) and stores the logs under a specific directory which can be templated. You can use predefined filters as well in each source:


syslog_ng_server_sources:
 - "candrop.example.com":
     proto: udp
     port: 1234
     filters:
       - f_error
       - f_kern
 - "secure.example.com":
     proto: tls
     port: 10514
     ca_dir: /opt/syslog-ng/etc/syslog-ng/ca.d
     key_file: /opt/syslog-ng/etc/syslog-ng/key.d/client.key
     cert_file: /opt/syslog-ng/etc/syslog-ng/cert.d/client_cert.pem
 - "tcp.example.com":
     proto: udp
     port: 1234

The counterside of server mode is the client mode. Syslog-ng will be configured to send its logs to one or more log servers, via UDP, TCP or TCP-TLS and you can also use filters.

The last mode - manual - is for syslog-ng hackers. If you are not satisfied with local, client or server modes, you can get a “vanilla” syslog-ng.conf. More better, you can use Jinja templates in it and have access to all variables.


There are more configuration options, these are just scratching the surface. You can add includes, set where to place logs, force Ansible to check the generated syntax before reloading syslog-ng, and so on. You can find the code and documentation here [1].


I developed and tested this module on an Ubuntu Trusty, but Travis runs all tests on Precise. So the supported platforms currently are Ubuntu or Debian systems. The 3.3, 3.4 and 3.5 syslog-ng versions are supported, (but unfortunately) their configuration files are not the same.


Finally I hope that these four modes covers the most use cases and will be used by a lot of Ansible users.

2014. augusztus 4., hétfő

Puppet is done and I'm moving to Ansible

Last time, when I wrote about my status, I was a little disappointed with Puppet and Ruby. I just couldn't get them to do what I wanted from them. Two new languages, two new challanges. But time was on my side, and step by step I managed to finish my project :-)


The development of my Puppet module reached its target, and it's ready for use. There will be some modifications (simplifying the configuration syntax), but they will be backward compatible.


You can generate syslog-ng sources, destinations, templates, etc. with this module. After I solved a relative require issue, the Travis builds are in nice green colors. The problem was, I couldn't use my own Ruby library placed next to a Puppet custom function below 1.9.2 Ruby version. The require './mylib' syntax didn't work at all, so I had to use a bit uglier, but working solution [1].


I gave up a separate develop branch and changed to a master branch with pull requests. On my machine it was simple to just commit to the develop branch, then rebase master onto it, but it had some drawbacks. First of all, I cannot run spec tests and build module packages on the same machine. This caused a lot of small commits (the spec tests succeded, but IRL the module didn't work) and an ugly git history. Then I created a local git repo (git clone --bare) and rebased my changes into one commit. That sounds good, but GitHub offers a nicer model. The syslog-ng project uses a master branch, and the contributors can make pull requests against that. The cool thing is, with each pull request Travis runs the tests against the merged code and evaluates the results. No more "build failed" on the master branch and the git history is clean, that's enough for me.


I got used to Ruby, I might even say I’ve grown like it. It's a great language and its block thing is awesome. But there are some weird stuff I can't get used to with my Python history.


I also created a Dockerfile, which prepares a development environment which can be used with my module. I hope it will help people get in touch more easily with this project. It's possible to run the spec tests and build a package, too :)


Last, but not least the documentation is also ready to be read and used. You can find the source code here [2]. The next challenge is Ansible, on which I have already started to work.