cron.weekly issue #20: Containers, PHP, MongoDB, Kernel 4.5, Puppet, perf, tcpdump & moreMarch 20, 2016 - Mattias Geniar
Welcome to cron.weekly issue #20 for Sunday March 20th, 2016.
I consider 20 consecutive issues to a pretty impressive milestone, so if you don’t mind I’ll go ahead and celebrate just a little bit.
You may notice a shift in this issue, as I’m moving more ‘news’ to the tools & project section. New releases for already established projects will now be listed there, no longer in the news part – where they previously got posted.
In addition, the online archives for cron.weekly now have a search function. If you know you’ve read about a particular topic but can’t find it again, try searching for it online. I tweaked the search results page a bit so I hope everyone can quickly find what they’re after.
This post by the Google Cloud Platform team describes the workings of Maglev, the “Fast and Reliable Software Network Load Balancer”. It uses Equal-Cost Multi-Path routing (ECMP) with a consistent hashing technique, making it possible to run your load balancers in active/active setups.
A couple of years ago we had the C10k problem/challenge, where the challenge was to hold 10.000 sockets/connections. Now Nomad, the container scheduler of Hashicorp, has another challenge (which they completed successfully): scheduling 1.000.000 containers in less than 5 minutes.
A vulnerability that got prematurely disclosed, by accident. All versions of git, both client-side and server-side, are vulnerable for a remote code execution vulnerability. Patches are available and if you run a publicly available git server, you need to act ASAP.
We already know PHP 7 is fast. In most cases, up to 2-3x as fast as PHP 5.6. So if you have a lot of servers, upgrading to PHP 7 can save a lot of CPU cycles and memory!
Opinionated, but a clever read nonetheless: the writer complains about MongoDB’s data access layer, the query framework, its MapReduce history, … Lots of food for thought.
Linus released version 4.5 of the Linux Kernel. There’s a human-readable write-up over at KernelNewbies.org. The main features: (file) copy offloading with new copy_file_range(2) system call, btrfs improvements, better epoll multithreading, performance improvements to UDP sockets & lots more.
This topic deserves special highlighting: the 4.6 kernel also introduces Address Space Layout Randomization (ASLR) for 32-bit programs.
A comic that’s worth your read: SIGKILL leaves children without parents. Seriously, it’s bad, m’kay.
Tools & Projects
It was quite the Puppet announce week for new releases: PuppetDB, the backend for Puppet’s exported resources, reached version 4.0. It’s a major release that improves performance and contains many new query features, including a new easier to use query language. It also drops support for Ruby 1.8.7, Puppet 3.x.
Next to PuppetDB, the Puppetlabs crew also released Puppet 4.4. The headline feature here is
Static Catalogs: file resources with ‘puppet:///’ source attributes will now include the checksum of the file inside the catalog, rather than requiring additional http requests to the master as the catalog is being applied. A full changelog is available on the site.
A new version of Nmap, the open source network scanner, was released: 12 new NSE scripts, hundreds of new
OS/version fingerprints, and dozens if smaller improvements and bug fixes.
Simple, single-command web publishing. Publish HTML, CSS, and JS for free, without leaving the command line. This builds upon the popularity of static site generators, massively simplifying the hosting setup required to run the site.
I told you, it’s a heavy Puppet week: this Puppet module allows you to compile simple Puppet manifests into a data structure that is suitable for mgmt (that other config management tool) to consume. It’s basically a convert-puppet-to-mgmt module if you ever want to switch over.
A wrapper for npm so you can play a Space-Invaders-a-like game while installing. This is a terminal-based game, so you can even play it over SSH.
csvkit is a suite of command-line tools for converting to and working with CSV, the king of tabular file formats.
git2consul takes one or many git repositories and mirrors them into Consul (a tool for Service Discovery & configuration). The goal is for organizations of any size to use git as the backing store, audit trail, and access control mechanism for configuration changes and Consul as the delivery mechanism.
If you’re into typography, like me, you’ll find this interesting: a collection of over 20 patched fonts for Powerline, Font Awesome, Octicons, Devicons, and Vim.
Ever tried to run ‘diff’ on 2 binary files? Doesn’t quite work the way you intended. BinDiff is a comparison tool for binary files that helps to quickly find differences and similarities in disassembled code.
There are already a couple of ways to run your PHP code (mod_php, fastcgi, FPM, …), but here’s another – relatively new – one: PPM. PPM is a process manager, supercharger and load balancer for PHP applications. It’s based on ReactPHP and works best with applications that use request-response frameworks like Symfony’s HTTPKernel.
Guides & Tutorials
This is a nice write-up of using OSSEC, an open source intrusion detection system (IDS). Lots of practical content on implementing it on your own.
This is a practical guide on using tcpdump to capture the network packets on a particular interface and investigate them using Wireshark (an easier GUI than plain tcpdump output).
Just like the previous post about tcpdump, this post is written by Julia Evians – a blog I highly recommend following. This post covers the ‘perf‘ tool for analysis and takes a different approach at explaining it: by reading (and explaining, thankfully) the Linux source code.
Monitoring the state of a Kubernetes cluster is not straightforward using traditional monitoring tools. This post covers an alternative called satellite, a simple and extensible monitoring agent / library for Kubernetes.
The short summary of this guide is: import nmap scan results directly into Elasticsearch and easily query them for analysis. Super cool idea.
Straight from the SELinux project lead: Dan Walsh talks about how to go about enabling SELinux to controlling access between standard Docker containers.
Step-by-step instructions on getting Fedora to run on your brand new Raspberry Pi 3.
Slightly older, but still very relevant: this guide covers all the steps involved in running your own mailserver, encrypting all your mails, adding anti-spam services, proper push support, …
This Github repository contains 5 very excellent articles: this is a 5-step description on Linux kernel timers, how the ‘clocksource’ and ‘tick broadcast’ framework works, … Very low-level if you’re into that kind of things.
What if … you wanted a life without tmux. A couple of pointers of reducing the switching between panes and tabs.
LXD is a daemon which provides a REST API to drive LXC containers. LXD containers are ‘heavier’ than Docker containers, they contain a full Linux system, exactly as it would be when run on metal or in a VM. Those containers will typically be long running and based on a clean distribution image. This post explores, in great detail, why you’d want to use it and how.
Very good pointers if you’re just starting out with Elasticsearch: tips on heap sizes, using doc values, storage-backend advice, slow query logging, … I wish I had known these earlier.
This post covers a couple of ‘lookup patterns’, where you tell Puppet to retrieve data for its modules from an external location (be it hiera or something else).
This is a neat trick that doesn’t only apply to GitHub. It describes using 2 separate ~/.ssh/config blocks for “personal” and “work” accounts, each pointing to a different SSH keyfile.
Some good points on a couple of “set” commands in Bash for more sane defaults: fail on exit straight away, better undefined variable handling & when piping commands, let any failed command be the exit code of that set of commands.
The Linux kernel can handle a lot of connections, but not out of the box. This guide explains some of the tweaks needed to be able to handle 12 million concurrent connections. A good reminder on ulimit & TCP port limitations.