cron.weekly issue #11: Ansible, Github, dd, systemd, Agedu, Netboot, SSH and many moreJanuary 17, 2016 - Mattias Geniar
This is edition #11 for cron.weekly on Sunday, January 17th, 2016.
Much if our open source work happens on Github. This open letter to the Github crew addresses some flaws that have plagued us for years. The focus is on custom fields in issues, a better “+1” and better information to be shown when creating new bugreports or feature requests.
The title says it all: better exception handling & reporting, loads of new modules, grouping of related tasks in “blocks” and internal API changes.
Linus announced the release of kernel 4.4, as scheduled. For a more human-friendly write-up of the changes, have a look at this guide. Most notably: better 3D hardware support, improved direct I/O drivers and a much improved TCP stack (lockless).
Not a new commit (it’s from 2014), but it’s gotten a revival of attention the last week: the dd command offers a progress-bar through the status=progress parameter. It may not be available in your package manager yet, though.
An interesting investigation into Venture Capital funding, focussing especially on Open Source projects. Why are Open Source projects so difficult to fund? What are the challenges and pitfalls? Are we all just piggybacking on the hard work from the core contributors of our favourite projects?
A problem in an obscure part of the OpenSSH client (“roaming”) was cause for concern this week. It could, under special curcomstances, allow the server you’re connecting to, to retrieve your SSH private key. Patching is advised!
Some insights into why the Kubernetes project doesn’t use the proven network stack that Docker has released and maintains.
The RPM database with all package information has traditionally always been Berkeley DB. The Fedora team is considering rolling their own solution for the next Fedora releases.
It’s not every day to see the NYTimes write about the CLI. It’s a confirmation on the efficiency and utility of performing actions via the command line, and how it’s creeping in to the lives of ‘normal’ (read: non-IT geeks) people.
Tools & Projects
A single command to set up a privacy- and censorship-aware server for you to browse the internet on: IPSEC, OpenVPN, Stunnel, Tor, … everything ‘s included.
This project can let you run a command as “random user” (hence the name). It picks a random UID and GID (which does not need to be in /etc/passwd) to run specific commands.
FireQOS is a program which sets up traffic shaping from an easy-to-understand and flexible configuration file. Traffic shaping with the native tools can be quite cumbersome and confusing, this seems to clear that up.
Tapir is a distributed transactional storage system used to build consistent transactions with inconsistent replication. Many buzzwords, the academic paper behind the project tries to explain the reasoning and rational behind the project.
A Unix utility for tracking down wasted disk space: it can list big diskspace consumers on your server and order them by “last access” time, showing the most likely candidates for removal. The HTML export option allows for very simple reporting and overviews.
We know awk mostly from some simple CLI commands like printing variables or making sums of values in logs, but awk is a pretty advanced and complete language. This awk-raycaster demo is a Pseudo-3D shooter written completely in awk using raycasting technique.
This project offers real-time notifications for vulnerable, open source, packages. It takes an inventory of your server and reports back which packages are considered vulnerable (much like Red Hat’s satellite service can do for you).
“How much memory is that one process consuming?” – this is actually a pretty hard question in Linux. The ps_mem script attempts to make this easier by calculating the real memory footprint by combining the private and shared RAM and outputting it in a readable way.
This new project is a frontend to tools like iPXE and offers usable PXE booted menus.
A very small (<100 lines) bash script that watches an etcd path and creates IPVS loadbalancing based on the results.
Last week I mentioned ClusterSSH for running SSH commands on multiple machines at the same time. Multiple readers wrote in to inform me about tmux-ssh, a ClusterSSH-like tool but all running from within one real terminal session with tmux.
Guides & Tutorials
This looks like a sales-pitch for AWS, but if offers a great overview of all the AWS services and their main purpose. If you’re thinking of running your Linux machines on AWS, this could be a good starting point.
Quite the list of practical tips and tricks of making the best use of your systemd configurations.
An older post but if you hadn’t read it yet, worth your time. The tl;dr: ports higher than 1024 can be opened by non-priviliged users, so you don’t know which SSH service you’re actually logging in to: the real SSH service or one started by a random user.
Security isn’t a ‘configure once, let it be‘ methodology, unfortunately. This post covers some good basics on hardening your server with practical commands for iptables, SSH configurations, SELinux, TOR exit node blocks, etc.
This one includes some hardware guides too: how to build a 36TB FreeNAS server to run at home (or the office).
A nice reminder that ties in with the NAS guide above, expending ZFS volumes isn’t always as efficient and you can end up losing a couple of extra drives on redundancy you don’t need.
This looks to be a very detailed and practical guide for getting started with Docker, including creating multi-container environments. The link above refers to the Github repo, there’s also a compiled hosted version available if you prefer that layout.
This paper compares NFSv4 to the older NFSv3 using a wide range of benchmarks.
If you’ve ever wanted to write a filesystem that can operate in userspace, you’ll appreciate this guide. Lots of low-level details and practical advice on how to write your own filesystem (other projects that are written in FUSE: SSHFS, GlusterFS, GMailfs).
A very detailed guide on using vim, covering the very basics but quickly advancing to the more complex use cases.