cron.weekly issue #32: httpdiff, minikube, GoAccess, Go, Strace, cron & much more!June 12, 2016 - Mattias Geniar
Welcome to cron.weekly issue #32 for Sunday, June 12th, 2016.
It’s a good mix of topics this week, I’m pretty sure everyone will find at least one article that interests him/her.
A very nice write-up on what it’s like to say “no” to a patch or pull request as an open source maintainer. Some practical tips on how to decline pull requests and contributions too, without being too harsh.
The concept of the Chaos Monkey (as introduced by Netflix) is to randomly kill a server, VM or container to make both developers and sysadmins more aware of failure and anticipate it. This post does something similar, but tied a Docker host to Half Life 2 where if you kill a zombie, it kills a container in your infrastructure.
Not everyone is equally happy with this decision: systemd-resolvd is going to be the new native DNS resolver in Ubuntu 16.10. However, there are some flaws: it allows MITM attacks, has no source port randomization and is easily bypassed by applications and libraries.
An extremely informing and clear presentation recorded at PyCon 2016 where Larry Hastings explains Python’s GIL and the efforts going into removing that bottleneck. Lots of insights in multithreaded programming and Python’s internals.
Spark isn’t the only data analytics tool: this post introduces Grappa, Apache Drill and Apache Kafka.
A very cool feature coming in Kernel 4.7: in-kernel histograms. You can now troubleshoot running applications and get a live histogram (a diagram/graph on screen) with info on which system calls happened, how frequently they happened and how long they took.
Quite a clever attack: this author published a lot of packages on popular package manager sites like PyPi, NPM and Rubygems. But the names were typosquatted from popular packages; instead of the popular package “requests”, the author published “reqeusts”, with a typo. Many sysadmins and developers all over the world accidentally installed his package due to typos.
Just a heads up: James Turnbull’s long awaited book (2 years in the making!) is out, and it looks to be a very complete one: a hands-on introductory book on the art of modern application and infrastructure monitoring and metrics.
A tale from a RHEL/CentOS user on what it’s like working with Ubuntu’s tools and methods for securing your OS.
XKCD is a webcomic that’s probably known to most of you, the latest one (accurately) makes fun of “man pages”.
Tools & Projects
This is a pure-bash ‘cpio’ extraction implementation. Some cool bash-scripting going on here, but it’s nothing more than a proof-of-concept.
Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.
I missed this one last week, but CentOS 6.8 has been released a couple of weeks after RHEL 6.8. Some of the major features: support for TLS 1.2, SSLv3 disabled by default and lots of package upgrades.
Scumblr is a web application that allows performing periodic searches and storing / taking actions on the identified results. It can crawl the web for specific keywords and inform you when they’re found (like compromised credentials, new vulnerabilities, …) or execute other scripts as a result. A mini IFTTT, if you will.
Vitess is the storage platform for scaling MySQL used by YouTube. It is optimized to run as effectively in cloud architectures as it does on dedicated hardware. It combines many important features of MySQL with the scalability of a NoSQL database.
GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser. Version 1.0 has a complete real-time HTML dashboard, lots of visualizations, metrics & more. Looks really great!
Do you ever feel lost in Amazon’s Web Services with terms like EC2, IAM, Lambda, SES, … ? Lots of acronyms, but this site explains each service Amazon offers with a clear example where to use it.
Perform the same request against two HTTP servers and diff the results.
Guides & Tutorials
A very comprehensive style guide by Google on how to write shell scripts. Like many other guides, it covers indentation, testing strings, pipelines, exit codes, function & variable naming, … Very to-the-point and practical code examples.
Apache Hadoop is a suite of open-source components which serve as the building blocks of large distributed systems. Hadoop components are designed to scale horizontally as the system need to handle more load; just add more nodes to the systems under contention.
The site layout is a bit hard to read, but this is a very thorough write-up on using ‘ps’ and alternatives in /proc/ to read the Linux process table.
A nice introduction into the Go language, identifying which problems it solves and where to use it best. While it’s definitely more complex than Bash, its single contained binary make it just as easy to ship as Bash scripts onto your server(s). Go could be the sysadmin language of the future.
If you’ve ever debugged applications via strace, you know the output isn’t always that clear. However, there is a very useful but lesser known parameter that makes things more readable: “-y” and “-yy”. They expand file descripter IDs to full paths, sockets, IPs & ports.
A good introduction to DNS, focussing on ‘dig’ as a CLI tool, interpreting the results, various DNS records, …
Granted, rsync –progress is easier, but this is a really nice hack on top of the ‘cp’ command to get a realtime progress bar!
Here are a couple of clever tricks to find out which environment your cron-jobs will run and how to fake that environment with just a bash script, so you know exactly how your cron scripts will behave.
It doesn’t always take complex bash scripts to automate a task: this post explains how you can make a couple of system Bash aliases to automate some of the more boring and repetitive tasks you do.