Planet Goerzen

John's BlogSound players: Adventures with Ampache, mpd, pulseaudio, Raspberry Pi, and Logitech Media Server

I finally decided it was about time to get my whole-house sound project off the ground. As an added bonus, I’d like to be able to stream music from my house to my Android phone.

Some Background

It was about 2.5 years ago that I last revisited the music-listening picture on Linux. I used Spotify for awhile, but the buggy nature of its support for local music eventually drove me up the wall. I have a large collection of music that will never be on Spotify (local choirs, for instance) and this was an important feature.

When Google Play Music added the feature of uploading your local collection, I used that; it let me stream music from my phone in my car (using the Bluetooh link to the car). I could also listen at home on a PC, or plug my phone into various devices to play. But that was a hassle, and didn’t let me have music throughout the house.

Google Play is reasonable for that, but it has a number of really glaring issues. One is that it often gets album artwork wrong; it doesn’t use the ID3 tags embedded in the files, but rather tries to “guess”. Another is that the “sync” is only “add”. Move files to another place in your collection, or re-rip them to FLAC and replace your old MP3s, and suddenly they’re in Google Play twice. It won’t ever see updated metadata, either — quite a hassle for someone that uses Musicbrainz and carefully curates metadata.

My Hardware

I already have an oldish PC set up as a entertainment box running MythTV. It is a diskless system (boots PAE over the network and has NFS root) and very quiet. It has video output to my TV, and audio output via S/PDIF to my receiver. It is one logical audio frontend.

My workstation in my office is another obvious place.

My kitchen has a radio with a line in jack, and I also have a small portable speaker with a line in jack to make the last two options. I also have a Raspberry Pi model B that I bought awhile ago and was looking for a purpose, so I thought – this should be cheap and easy, right? Well. Cheap, yes. Easy, not so much.

First attempt: Ampache

The Ampache project produces quite a nice piece of software. Ampache has matured significantly in the last 2.5 years, and the usability of its web-interface — with HTML5 and Flash players as options — is quite impressive. It is easily as usable as Google’s, though its learning curve is rather more steep. There are multiple Android apps for Ampache to stream remotely. And, while most are terribly buggy and broken, there is at least one that seems to work well.

Ampache can also output m3u/pls files for a standalone player. It does on-the-fly transcoding. There are some chinks in the armor, however. The set of codecs that are transcoded or passed through is a global setting, not a per-device setting. The bitrates are per-account, so you can’t easily have it transcode FLAC into 320Kbps mp3 for streaming on your LAN and 128Kbps MP3 for streaming to your phone. (There are some hacks involving IP address ranges and multiple accounts, but they are poorly documented and cumbersome.)

Ampache also has a feature called “localplay” in which it drives local players instead of remote ones. I tried to use this in combination with mpd to drive music to the whole house. Ampache’s mpd interface is a bit odd; it actually loads things up into mpd’s queue. Sadly it shares the same global configs as the rest. Even though mpd is perfectly capable of handling FLAC audio, the Ampache web player isn’t, so you have to either make it transcode mp3 for everything or forego the web player (or use a second account that has “transcode everything” set). Frustratingly, not one of the Android clients for Ampache is even remotely compatible with Localplay, and some will fail in surprising ways if you have been using Localplay on the web client.

So let’s see how this mpd thing worked out.

Ampache with mpd for whole-house audio

The primary method here is to use mpd’s pulseaudio driver. I configured it like so:

audio_output {
type "pulse"
name "MPD stream"
#server "remote_server" # optional
sink "rtp" # optional
mixer_type "software"

Then in /etc/pulse/

load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties="device.description='RTP Multicast Sink'"
load-module module-rtp-send source=rtp.monitor

This tells Pulse to use multicast streaming to the LAN for the audio packets. Pulse is supposed to have latency synchronization to achieve perfect audio everywhere. In practice, this works somewhat poorly. Plus I have to install pulse everywhere, which inserts its tentacles way too deeply into the ALSA stack for my taste. (alsamixer suddenly turns useless by default, for instance.)

But I gave it a try. After much fiddling — Pulse is rather poorly documented and the interactions of configuration tools with it even more so — I found a working configuration. On my MythTV box, I added:

load-module module-rtp-recv
load-module module-native-protocol-tcp auth-ip-acl=;x.x.x.x

The real annoyance was getting it to set the output through the S/PDIF digital port. Finally I figured out the magic potion:

set-card-profile 0 output:iec958-stereo+input:analog-stereo

This worked reasonably well. The Raspberry Pi was a much bigger challenge, however.

I put Raspbian on it easily enough, and installed Pulse. But apparently it is well-known in the Pulse community that Pulse’s RTP does not work well with wifi. Multicast itself works poorly with wifi in general, and Pulse won’t do unicast RTP. However, Pulse assumes very low latency and just won’t work well with wifi at all.

Morever, pulseaudio on the rpi is something of a difficult beast to tame. It has crackling audio, etc. I eventually got it working decently with:


realtime-scheduling = yes
realtime-priority = 5
resample-method = src-sinc-fastest
default-sample-rate = 44100
default-fragments = 4

and in, commented out the module-udev-detect and module-detect, and added:

load-module module-alsa-card device_id=0 tsched=0 fragments=10 fragment_size=640 tsched_buffer_size=4194384 tsched_buffer_watermark=262144

plus the rtp-recv and tcp protocol lines as before. This got it working with decent quality, but it was always out of sync by at least a few tenths of a second, if not a whole second, with the other rooms.

Brief diversion: mplayer

Many players besides pulseaudio can play the RTP streams that pulseaudio generates. mplayer, for instance, can. I found this worked on the Raspberry Pi:

mplayer -really-quiet -cache 64 -cache-min 95 -demuxer rawaudio -rawaudio format=0x20776172 rtp://

But this produced even worse sync. It became clear that Ampache was not going to be the right solution.

Logitech Media Server

I have looked at this a few times over the years, but I’ve somehow skipped it. But I looked at it again now. It is an open source (GPL) server, and was originally designed to work with Logitech hardware. There are now all sorts of software clients out there. There is a helpful wiki about it, although Logitech has rebranded the thing so many times, it’s not even consistent internally on what the heck it’s called (is it LMS? or Squeezebox Server? or Slimserver? The answer is: yes.)

LMS does no web-based streaming. At all. But I thought I’d give it a try. Installation is a little… weird (its .deb packages up binary perl modules for things that are already in Debian, for half a dozen architectures and many Perl versions.) I had some odd issues but eventually it worked. It scanned my media collection. I installed squeezelite on my workstation as my first player, and things worked reasonably well out of the box.

I proceeded to install squeezelite on the Raspberry Pi and MythTV server (where I had to compile libsoxr for wheezy, due to an obscure error I couldn’t find the cause of until I used strace). Log in to the LMS web interface, tell it to enable synchronization, and bam! Perfect synchronization! Incredible.

But I ran into two issues: one was that it used a lot of CPU even on my workstation (50% or more, even when idle). Strangely, it used far less CPU on the pi than did pulseaudio. Secondly, I’d get annoying clicks from it from time to time. Some debugging and investigation revealed that they were both somewhat related; it was getting out of sync with the pi and correcting. Initially I assumed this to be an issue with the pi, but tracked it down to something else. I told it to upsample to 48kHz and this made the problem go away. My command on PC hardware is:

squeezelite -a 200:10 -o dmix -u hLX -z

And on the Raspberry Pi:

squeezelite -a 800:10:16:0 -n kitchen -z

It is working perfectly now.

LMS even has an Android remote control app that can control all devices together, queue up playlists, etc. Very slick.

The Raspberry Pi

The Raspberry Pi, even the model B+, apparently has notoriously bad audio output. I noticed this first as very low volume on the output. I found a $11 USB DAC on Amazon that seems to do the trick, so we will see. I am also using the $10 Edimax USB Wifi and it seems to work well.

Other Options

The big-name other option here is Sonos. They have perfect sync audio already working. But they’re pricy, closed, and proprietary; you have to buy Sonos speakers, transmitters, etc. for the entire house. Their receivers start at $200, and it’s at least $350 to integrate an existing stereo into the system. I’d be shelling out over $1000 to use Sonos for this setup. As it is, I’ve bought less than $100 of equipment (a second Pi and its accessories) and getting output that, while no doubt not quite as pristine, it still quite nice and acceptable. Quite nice.

Update: LMS with Apache

It turns out that Logitech Media Server can automatically save created playlists to a spot where Ampache can find them. So an Ampache streaming player can be used to access the same collection LMS uses, with full features. Nice!

Picasa PhotosQuadcopter Snow

Quadcopter Snow

Date: Dec 18, 2014
Number of Photos in Album: 3

View Album

John's BlogAerial Photos: Our Little House on the Prairie


This was my first attempt to send up the quadcopter in winter. It’s challenging to take good photos of a snowy landscape anyway. Add to that the fact that the camera is flying, and it’s cold, which is hard on batteries and motors. I was rather amazed at how well it did!



Flickr PhotosDJI00816

prairiecode posted a photo:



Flickr PhotosDJI00830

prairiecode posted a photo:



Flickr PhotosDJI00812

prairiecode posted a photo:



John's BlogMy boys love 1986 computing

Yesterday, Jacob (age 8) asked to help me put together a 30-year-old computer from parts in my basement. Meanwhile, Oliver (age 5) asked Laura to help him learn cursive. Somehow, this doesn’t seem odd for a Saturday at our place.

2014-11-22 18.58.36

Let me tell you how this came about.

I’ve had a project going on for a while now to load data from old floppies. It’s been fun, and had a surprise twist the other day: my parents gave me an old TRS-80 Color Computer II (aka “CoCo 2″). It was, in fact, my first computer, one they got for me when I was in Kindergarten. It is nearly 30 years old.

I have been musing lately about the great disservice Apple did the world by making computers easy to learn — namely the fact that few people ever bother to learn about them. Who bothers to learn about them when, on the iPhone for instance, the case is sealed shut, the lifespan is 1 or 2 years for many purchasers, and the platform is closed in lots of ways?

I had forgotten how finicky computers used to be. But after some days struggling with IDE incompatibilities, booting issues, etc., when I actually managed to get data off a machine that had last booted in 1999, I had quite the sense of accomplishment, which I rarely have lately. I did something that was hard to do in a world where most of the interfaces don’t work with equipment that old (even if nominally they are supposed to.)

The CoCo is one of those computers normally used with a floppy drive or cassette recorder to store programs. You type DIR, and you feel the clack of the drive heads through the desk. You type CLOAD and you hear the relay click closed to turn on the tape motor. You wiggle cables around until they make contact just right. You power-cycle for the times when the reset button doesn’t quite do the job. The details of how it works aren’t abstracted away by innumerable layers of controllers, interfaces, operating system modules, etc. It’s all right there, literally vibrating your desk.

So I thought this could be a great opportunity for Jacob to learn a few more computing concepts, such as the difference between mass storage and RAM, plus a great way to encourage him to practice critical thinking. So we trekked down to the basement and came up with handfulls of parts. We brought up the computer, some joysticks, all sorts of tangled cables. We needed adapters, an old TV. Jacob helped me hook everything up, and then the moment of truth: success! A green BASIC screen!

I added more parts, but struck out when I tried to connect the floppy drive. The thing just wouldn’t start up right whenever the floppy controller cartridge was installed. I cleaned the cartridge. I took it apart, scrubbed the contacts, even did a re-seat of the chips. No dice.

So I fired up my CoCo emulator (xroar), and virtually “saved” some programs to cassette (a .wav file). I then burned those .wav files to an audio CD, brought up an old CD player from the basement, connected the “cassette in” plug to the CD player’s headphone jack, and presto — instant programs. (Well, almost. It takes a couple of minutes to load a program from audio codes.)

The picture above is Oliver cackling at one of the very simplest BASIC programs there is: “number find.” The computer picks a random number between 1 and 2000, and asks the user to guess it, giving a “too low” or “too high” clue with each incorrect guess. Oliver delighted in giving invalid input (way too high numbers, or things that weren’t numbers at all) and cackled at the sarcastic error messages built into the program. During Jacob’s turn, he got very serious about it, and is probably going to be learning about how to calculate halfway points before too long.

But imagine my pride when this morning, Jacob found the new CD I had made last night (correcting a couple recordings), found my one-line instruction on just part of how to load a program, and correctly figured out by himself all the steps to do in order (type CLOAD on the CoCo, advance the CD to the proper track, press play on the player, wait for it to load on the CoCo, then type RUN).

I ordered a replacement floppy controller off eBay tonight, and paid $5 for a coax adapter that should fix some video quality issues. I rescued some 5.25″ floppies from my trash can from another project, so they should have plenty of tools for exploration.

It is so much easier for them to learn how a disk drive works, and even what the heck a track is, when you can look at a floppy drive with the cover off and see the heads move. There are other things we can do with more modern equipment — Jacob has shown a lot of interest in Arduino projects — but I have so far drawn a blank on ways to really let kids discover how a modern PC (let alone a modern phone or tablet) works.

Update Nov. 24: Every so often, the world surprises me by deciding to, well, read one of my random blog posts. For the benefit of those of you that don’t already know my boys, you might want to know that among their common play activites are turning trees into pretend trains, typing at a manual typewriter, reading, writing their own books, using a cassette recorder, building a PC and learning to use bash or xmonad, making long paper tapes with an adding machine, playing records on a record player, building electric gizmos, and even making mud balls.

I am often asked about the role of the computer in the lives, given that my hobby and profession involves computers. The answer: less than that of most of their peers. I look for opportunities for them to learn by doing, discovering, playing, or imagining. I make no presumption that they will develop the passion for computers that I did. What I want is for them to have the curiosity and drive to learn everything there is to know about whatever they do develop a passion for, so they will be great at it.

Flickr Photos2014-11-22 18.58.36

prairiecode posted a photo:

2014-11-22 18.58.36

John's BlogContemplative Weather

Sometimes I look out the window and can’t help but feel “this weather is deep.” Deep with meaning, with import. Almost as if the weather is confident of itself, and is challenging me to find some meaning within it.

This weekend brought the first blast of winter to the plains of Kansas. Saturday was chilly and windy, and overnight a little snow fell. Just enough to cover up the ground and let the tops of the blades of grass poke through. Just enough to make the landscape look totally different, without completely hiding what lies beneath. Laura and I stood silently at the window for a few minutes this morning, gazing out over the untouched snow, extending out as far as we can see.

Yesterday, I spent some time with my great uncle and aunt. My great uncle isn’t doing so well. He’s been battling cancer and other health issues for some time, and can’t get out of the house very well. We talked for an hour and a half – about news of the family, struggles in life now and in the past, and joys. There were times when all three of us had tears in our eyes, and times when all of us were laughing so loudly. My great uncle managed to stand up twice while I was there — this took quite some effort — once to give me a huge hug when I arrived, and another to give me an even bigger hug when I left. He has always been a person to give the most loving hugs.

He hadn’t been able to taste food for awhile, due to treatment for cancer. When I realized he could taste again, I asked, “When should I bring you some borscht?” He looked surprised, then got a huge grin, glanced at his watch, and said, “Can you be back by 3:00?”

His brother, my grandpa, was known for his beef borscht. I also found out my great uncle’s favorite kind of bread, and thought that maybe I would do some cooking for him sometime soon.

Today on my way home from church, I did some shopping. I picked up the ingredients for borscht and for bread. I came home, said hi to the cats that showed up to greet me, and went inside. I turned on the radio – Prairie Home Companion was on – and started cooking.

It takes a long time to prepare what I was working on – I spent a solid two hours in the kitchen. As I was chopping up a head of cabbage, I remembered coming to what is now my house as a child, when my grandpa lived here. I remembered his borscht, zwiebach, monster cookies; his dusty but warm wood stove; his closet with toys in it. I remembered two years ago, having nearly 20 Goerzens here for Christmas, hosted by the boys and me, and the 3 gallons of borscht I made for the occasion.

I poured in some tomato sauce, added some water. The radio was talking about being kind to people, remembering that others don’t always have the advantages we do. Garrison Keillor’s fictional boy in a small town, when asked what advantages he had, mentioned “belonging.” Yes, that is an advantage. We all deal with death, our own and that of loved ones, but I am so blessed by belonging – to a loving family, two loving churches, a wonderful community.

Out came three pounds of stew beef. Chop, chop, slice, plunk into the cast iron Dutch oven. It’s my borscht pot. It looks as if it would be more at home over a campfire than a stovetop, but it works anywhere.

Outside, the sun came up. The snow melts a little, and the cats start running around even though it’s still below freezing. They look like they’re having fun playing.

I’m chopping up parsley and an onion, then wrapping them up in a cheesecloth to make the spice ball for the borscht. I add the basil and dill, some salt, and plonk them in, too. My 6-quart pot is nearly overflowing as I carefully stir the hearty stew.

On the radio, a woman who plays piano in a hospital and had dreamed of being on that particular radio program for 13 years finally was. She played with passion and delight I could hear through the radio.

Then it’s time to make bread. I pour in some warm water, add some brown sugar, and my thoughts turn to Home On The Range. I am reminded of this verse:

How often at night when the heavens are bright
With the light from the glittering stars
Have I stood here amazed and asked as I gazed
If their glory exceeds that of ours.

There’s something about a beautiful landscape out the window to remind a person of all the blessings in life. This has been a quite busy weekend — actually, a busy month — but despite the fact I have a relative that is sick in the midst of it all, I am so blessed in so many ways.

I finish off the bread, adding some yeast, and I remember my great uncle thanking me so much for visiting him yesterday. He commented that “a lot of younger people have no use for visiting an old geezer like me.” I told him, “I’ve never been like that. I am so glad I could come and visit you today. The best gifts are those that give in both directions, and this surely is that.”

Then I clean up the kitchen. I wipe down the counters from all the bits of cabbage that went flying. I put away all the herbs and spices I used, and finally go to sit down and reflect. From the kitchen, the smells of borscht and bread start to seep out, sweeping up the rest of the house. It takes at least 4 hours for the borscht to cook, and several hours for the bread, so this will be an afternoon of waiting with delicious smells. Soon my family will be home from all their activities of the day, and I will be able to greet them with a warm house and the same smells I stepped into when I was a boy.

I remember this other verse from Home On the Range:

Where the air is so pure, the zephyrs so free,
The breezes so balmy and light,
That I would not exchange my home on the range
For all of the cities so bright.

Today’s breeze is an icy blast from the north – maybe not balmy in the conventional sense. But it is the breeze of home, the breeze of belonging. Even today, as I gaze out at the frozen landscape, I realize how balmy it really is, for I know I wouldn’t exchange my life on the range for anything.

John's BlogComputer Without a Case

My desk today looks like this:

2014-11-12 11.58.45

Yep, that’s a computer. Motherboard to the right, floppy drives and CD drive stacked on top of the power supply, hard drive to the left.

And it’s an OLD computer. (I had forgotten just how loud these old power supplies are; wow.)

The point of this exercise is to read data off the floppies that I have made starting nearly 30 years ago now (wow). Many were made with DOS, some were made on a TRS-80 Color Computer II (aka CoCo 2). There are 5.25″ disks, 3.25″ disks, and all sorts of formats. Most are DOS, but the TRS-80 ones use a different physical format. Some of the data was written by Central Point Backup (from PC Tools), which squeezed more data on the disk by adding an extra sector or something, if my vague memory is working.

Reading these disks requires low-level playing with controller timing, and sometimes the original software to extract the data. It doesn’t necessarily work under Linux, and certainly doesn’t work with USB floppies or under emulation. Hence this system.

It’s a bridge. Old enough to run DOS, new enough to use an IDE drive. I can then hook up the IDE drive to a IDE-to-USB converter and copy the data off it onto my Linux system.

But this was tricky. I started the project a few years ago, but life got in the way. Getting back to it now, with the same motherboard and drive, but I just couldn’t get it to boot. I eventually began to suspect some disk geometry settings, and with some detective work from fdisk in Linux plus some research into old BIOS disk size limitations, discovered the problem was a 2GB limit. Through some educated trial and error, I programmed the BOIS with a number of cylinders that worked, set it to LBA mode, and finally my 3-year-old DOS 6.2 installation booted.

I had also forgotten how finicky things were back then. Pop a floppy from a Debian install set into the drive, type dir b:, and the system hangs. I guess there was a reason the reset button was prominent on the front of the computer back then…

Flickr Photos2014-11-12 11.58.45

prairiecode posted a photo:

2014-11-12 11.58.45

John's BlogI’m hiring a senior Linux sysadmin/architect

I’m never sure whether to post such things here, but I hope that it’s of interest to people: I’m trying to hire a top-notch Linux person for a 100% telecommute position. I’m particularly interested in people with experience managing 500 or more OS instances. It’s a shop with a lot of Debian, by the way. You can apply at that URL and mention you saw it in my blog if you’re interested.

John's BlogDebian – A plea to worry about what matters, and not take ourselves too seriously

I posted this on debian-devel today. I am also posting it here, because I believe it is important to more than just Debian developers.

Good afternoon,

This message comes on the heels of Sam Hartman’s wonderful plea for compassion [1] and the sad news of Joey Hess’s resignation from Debian [2].

I no longer frequently post to this list, but when you’ve been a Debian developer for 18 years, and still care deeply about the community and the project, perhaps you have a bit of perspective to share.

Let me start with this:

Debian is not a Free Software project.

Debian is a making-the-world-better project, a caring for people project, a freedom-spreading project. Free Software is our tool.

As many of you, hopefully all of you, I joined Debian because I enjoyed working on this project. We all did, didn’t we? We joined Debian because it was fun, because we were passionate about it, because we wanted to make the world a better place and have fun doing it.

In short, Debian is life-giving, both to its developers and its users.

As volunteers, it is healthy to step back every so often, and ask ourselves two questions: 1) Is this activity still life-giving for me? 2) Is it life-giving for others?

I have my opinions about init. Strong ones, in fact. [3] They’re not terribly relevant to this post. Because I can see that they are not really all that relevant.

14 years ago, I proposed what was, until now anyhow, one of the most controversial GRs in Debian history. It didn’t go the way I hoped. I cared about it deeply then, and still care about the principles.

I had two choices: I could be angry and let that process ruin my enjoyment of Debian. Or I could let it pass, and continue to have fun working on a project that I love. I am glad I chose the latter.

Remember, for today, one way or another, jessie will still boot.

18 years ago when I joined Debian, our major concerns were helping newbies figure out how to compile their kernels, finding manuals for monitors so we could set the X modelines properly, finding some sort of Free web browser, finding some acceptable Office-type software.

Wow. We WON, didn’t we? Not just Debian, but everyone. Freedom won.

I promise you – 18 years from now, it will not matter what init Debian chose in 2014. It will probably barely matter in 3 years. This is not key to our goals of making the world a better place. Jessie will still boot. I say that even though my system runs out of memory every few days because systemd-logind has a mysterious bug [4]. It will be fixed. I say that even though I don’t know what init system it will use, or how much choice there will be. I say that because it is simply true. We are Debian. We will make it work, one way or another.

I don’t post much on this list anymore because my personal passion isn’t with posting on this list anymore. I make liberal use of my Delete Thread keybinding on -vote these days, because although I care about the GR, I don’t care about it enough to read all the messages about it. I have not yet decided if I will spend the time researching it in order to vote. Instead of debating the init GR, sometimes I sit on the sofa with my wife. Sometimes I go out and fly the remote-control airplane I’m learning to fly. Sometimes I repair my plane after a flight that was shorter than planned. Sometimes I play games with my boys, or help them with homework, or share my 8-year-old’s delight as a text file full of facts about the Titanic that he wrote in Emacs comes spitting out of the printer. Sometimes I write code or play with the latest Linux filesystems or build a new server for my basement.

All these things matter more to me than init. I have been using Debian at home for almost 20 years, at various workplaces for almost that long, and it is not going to stop being a part of my life any time soon. Perhaps I will have to learn how to administer a new init system. Well, so be it; I enjoy learning new things. Or perhaps I will have to learn to live with some desktop limitations with an old init system. Well, so be it; it won’t bother me much anyhow. Either way, I’m still going to be using what is, to me, the best operating system in the world, made by one of the world’s foremost Freedom projects.

My hope is that all of you may also have the sense of peace I do, that you may have your strong convictions, but may put them all in perspective. That we as a project realize that the enemy isn’t the lovers of the other init, but the people that would use law and technology to repress people all over the world. We are but one shining beacon on a hill, but the world will be worse off if our beacon winked out.

My plea is that we each may get angry at what matters, and let go of the smaller frustrations in life; that we may each find something more important than init/systemd to derive enjoyment and meaning from. [5] May you each find that airplane to soar freely in the skies, to lift your soul so that the joy of using Free Software to make the world a better place may still be here, regardless of what /sbin/init is.



[3] A hint might be that in my more grumpy moments, I realize I haven’t ever quite figured out why the heck this dbus thing is on so many of my systems, or why I have to edit XML to configure it… ;-)

[4] #765870

[5] No disrespect meant to the init/systemd maintainers. Keep enjoying what you do, too!

John's BlogBeing Different

This evening, after the boys were in bed, Laura and I sat down to an episode of MASH (a TV series from the 70s) and leftover homemade pumpkin bars. She commented, “Sometimes I wonder what generation we’re in. This doesn’t seem to be something people our age are usually doing.” Probably true. I suppose people my age aren’t usually learning to play the penny whistle or put up antennas in trees either.

We’ve had a fun day today – a different sort of day in a lot of ways. We took the boys for their first Wichita Symphony Orchestra experience — they were doing their first-ever “family concert” (Beethoven Lived Upstairs, which combined Beethoven’s music with a two-person play aimed at kids). And they had an “instrument petting zoo” beforehand. Both boys loved it.

From November 8, 2014

After that, we took them to a sushi place for the first time. We ordered different types of rolls for our table, encouraging them to start with the California roll. They loved it (though Oliver did complain it was a bit hard to eat). Jacob happily devoured everything he could that wasn’t spicy. He would have probably devoured the plate of California roll slices by himself if I hadn’t stopped him and encouraged him to slow down and try some other things too.

It doesn’t seem very common around here to take 5-year-olds to a sushi place and plan on them eating the same sort of food that the adults around them are. It is a lot of fun to be different. Jacob and Oliver both have their unique personalities and interests, and I hope that they continue to find strength and joy in all the ways they are unique.

Picasa PhotosNovember 8, 2014

November 8, 2014

Date: Nov 8, 2014
Number of Photos in Album: 1

View Album

John's BlogHalloween: A Pumpkin and an Insect Matador

You never quite know what to expect with children. For Halloween this year, Laura found some great costumes at a local thrift store. Jacob loved his “matador” costume, with a cape and vest. He had fun swishing the cape around him. But he didn’t want to use the nice hat with a red flower in it that Laura found. Nope. What he wanted was the hat with plastic springy things that she got on a lark – he said it was “insect antennae” and that he was an “insect matador”. This prompted some confused looks and big smiles from the people he saw when we went trick-or-treating!


Oliver, meanwhile, enjoyed his pumpkin outfit, complete with orange hair – his favorite part.


Here’s a typical scene:


And, of course, Jacob running with the cape flowing behind him:


Flickr PhotosIMG_4216

prairiecode posted a photo:


Flickr PhotosIMG_4200

prairiecode posted a photo:


Flickr PhotosMVI_4196

prairiecode posted a video:


Flickr PhotosIMG_4189

prairiecode posted a photo:


Flickr PhotosIMG_4178

prairiecode posted a photo:


Flickr PhotosIMG_4173

prairiecode posted a photo:


Flickr PhotosIMG_4172

prairiecode posted a photo:


Flickr PhotosIMG_4166

prairiecode posted a photo:


John's BlogUpdate on the systemd issue

The other day, I wrote about my poor first impressions of systemd in jessie. Here’s an update.

I’d like to start with the things that are good. I found the systemd community to be one of the most helpful in Debian, and #debian-systemd IRC channel to be especially helpful. I was in there for quite some time yesterday, and appreciated the help from many people, especially Michael. This is a nontechnical factor, but is extremely important; this has significantly allayed my concerns about systemd right there.

There are things about the systemd design that impress. The dependency system and configuration system is a lot more flexible than sysvinit. It is also a lot more complicated, and difficult to figure out what’s happening. I am unconvinced of the utility of parallelization of boot to begin with; I rarely reboot any of my Linux systems, desktops or servers, and it seems to introduce needless complexity.

Anyhow, on to the filesystem problem, and a bit of a background. My laptop runs ZFS, which is somewhat similar to btrfs in that it’s a volume manager (like LVM), RAID manager (like md), and filesystem in one. My system runs LVM, and inside LVM, I have two ZFS “pools” (volume groups): one, called rpool, that is unencrypted and holds mainly the operating system; and the other, called crypt, that is stacked atop LUKS. ZFS on Linux doesn’t yet have built-in crypto, which is why LVM is even in the picture here (to separate out the SSD at a level above ZFS to permit parts of it to be encrypted). This is a bit of an antiquated setup for me; as more systems have AES-NI, I’m going to everything except /boot being encrypted.

Anyhow, inside rpool is the / filesystem, /var, and /usr. Inside /crypt is /tmp and /home.

Initially, I tried to just boot it, knowing that systemd is supposed to work with LSB init scripts, and ZFS has init scripts with carefully-planned dependencies. This was evidently not working, perhaps because /lib/systemd/systemd/ It turns out that systemd has a few assumptions that turn out to be less true with ZFS than otherwise. ZFS filesystems are normally not mounted via /etc/fstab; a ZFS pool has internal properties about which dataset gets mounted where (similar to LVM’s actions after a vgscan and vgchange -ay). Even though there are ordering constraints in the units, systemd is writing files to /var before /var gets mounted, resulting in the mount failing (unlike ext4, ZFS by default will reject an attempt to mount over a non-empty directory). Partly this due to the debian-fixup.service, and partly it is due to systemd reacting to udev items like backlight.

This problem was eventually worked around by doing zfs set mountpoint=legacy rpool/var, and then adding a line to fstab (“rpool/var /var zfs defaults 0 2″) for /var and its descendent filesystems.

This left the problem of /tmp; again, it wasn’t getting mounted soon enough. In this case, it required crypttab to be processed first, and there seem to be a lot of bugs in the crypttab processing in systemd (more on that below). I eventually worked around that by adding to the zfs-import-cache.service file. For /tmp, it did NOT work to put it in /etc/fstab, because then it tried to mount it before starting cryptsetup for some reason. It probably didn’t help that the system’s cryptdisks.service is a symlink to /dev/null, a fact I didn’t realize until after a lot of needless reboots.

Anyhow, one thing I stumbled across was poor console control with systemd. On numerous occasions, I had things like two cryptsetup processes trying to read a password, plus an emergency mode console trying to do so. I had this memorable line of text at one point:

(or type Control-D to continue): Please enter passphrase for disk athena-crypttank (crypt)! [ OK ] Stopped Emergency Shell.

And here we venture into unsatisfying territory with systemd. One answer to this in IRC was to install plymouth, which apparently serializes console I/O. However, plymouth is “an attractive boot animation in place of the text messages that normally get shown.” I don’t want an “attractive boot animation”. Nevertheless, neither systemd-sysv nor cryptsetup depends on plymouth, so by default, the prompt for a password at boot is obscured by various other text.

Worse, plymouth doesn’t support serial consoles, so at the moment booting a system that uses LUKS with systemd over a serial console is a matter of blind luck of typing the right password at the right time.

In the end, though, the system booted and after a few more tweaks, the backlight buttons do their thing again. Whew!

Update 2014-10-13: uau pointed out that Plymouth is more than a bootsplash, and can work with serial consoles, despite the description of the package. I stand corrected on that. (It is still the case, however, that packages don’t depend on it where they should, and the default experience for people using cryptsetup is not very good.)

John's BlogFirst impressions of systemd, and they’re not good

Well, I finally bit the bullet. My laptop, which runs jessie, got dist-upgraded for the first time in a few months. My brightness keys stopped working, and it no longer would suspend to RAM when the lid was closed, and upon chasing things down from XFCE to policykit, eventually it appears that suddenly major parts of the desktop breaks without systemd in jessie. Sigh.

So apt-get install systemd-sysv (and watch sysvinit-core get uninstalled) and reboot.

Only, my system doesn’t come back up. In fact, over several hours of trying to make it boot with systemd, it failed in numerous spectacular and hilarious (or, would be hilarious if my laptop would boot) ways. I had text obliterating the cryptsetup password prompt almost every time. Sometimes there were two processes trying to read a cryptsetup password at once. Sometimes a process was trying to read that while another one was trying to read an emergency shell password. Many times it tried to write to /var and /tmp before they were mounted, meaning they *wouldn’t* mount because there was stuff there.

I noticed it not doing much with ZFS, complaining of a dependency loop between zfs-mount and $local-fs. I fixed that, but it still wouldn’t boot. In fact, it simply hung after writing something about wall passwords.

I’ve dug into systemd, finding a “unit generator for fstab” (whatever the hack that is, it’s not at all made clear by systemd-fstab-generator(8)).

In some cases, there’s info in journalctl, but if I can’t even get to an emergency mode prompt, the practice of hiding all stdout and stderr output is not all that pleasant.

I remember thinking “what’s all the flaming about?” systemd wasn’t my first choice (I always thought “if it ain’t broke, don’t fix it” about sysvinit), but basically ignored the thousands of messages, thinking whatever happens, jessie will still boot.

Now I’m not so sure. Even if the thing boots out of the box, it seems like the boot process with systemd is colossally fragile.

For now, at least zfs rollback can undo upgrades to 800 packages in about 2 seconds. But I can’t stay at some early jessie checkpoint forever.

Have we made a vast mistake that can’t be undone? (If things like even *brightness keys* now require systemd…)

John's BlogThe Thrill and Stress of Too Many Hobbies

Today, 4PM. Jacob and Oliver excitedly peer at the box in our kitchen – a really big box, taller than them. Inside is is the first model airplane I’d ever purchased. The three of us hunkered down on the kitchen floor, opened the box, unpacked the parts, examined the controller, and found the manual with cryptic assembly directions. Oliver turned some screws while Jacob checked out the levers on the controllers. Then they both left for a bit to play with their toy buses.

A little while later, the three of us went outside. It was too windy to fly. I had never flied an RC plane before — only RC quadcopters (much easier to fly), and some practice time on an RC simulator. But the excitement was too much. So out we went, and the plane took off perfectly, climbed, flew over the trees, and circled above our heads at my command. I even managed a good landing in the wind, despite about 5 aborted attempts due to coming in too high, wrong angle, too fast, or last-minute gusts of wind throwing everything off. I am not sure how I pulled that all off on my first flight, but somehow I did! It was thrilling!

I’ve had a lot of hobbies in my life. Computers have run through many of them; I learned Pascal (a programming language) at about the same time I learned cursive handwriting and started with C at around age 10. It was all fun. I’ve been a Debian developer for some 18 years now, and have written a lot of code, and even books about code, over the years.

Photography, music, literature, history, philosophy, and theology have been interests for quite some time as well. In the last few years, I’ve picked up amateur radio, model aircraft, etc. And last month, Laura led me into Ada’s Technical Books during our visit to Seattle, resulting in me getting interested in Arduino. (The boys and I have already built a light-activated crossing gate for their HO-gauge model trains, and Jacob can now say he’s edited a few characters of C!)

Sometimes I find ways to merge hobbies; I’ve set up all sorts of amateur radio systems on Linux, take aerial photographs, and set up systems to stream music in my house.

But I also have a lot less time for hobbies overall than I once did; other things in life, such as my children, are more important. Some of the code I once worked on actively I no longer use or maintain, and I feel guilty about that when people send bug reports that I have no interest in fixing anymore.

Sometimes I feel a need to cut down, and perhaps have; and then, I get an interest in RC aircraft and find an airplane that is great for a beginner and fairly inexpensive.

Perhaps it is the curse of being a curious person living in an interesting world. Do any of the rest of you have a large number of hobbies? How do you feel about that?

Flickr PhotosDJI00719

prairiecode posted a video:


Flickr PhotosDJI00734

prairiecode posted a video:


Flickr PhotosDJI00741

prairiecode posted a photo:



Flickr PhotosDJI00739

prairiecode posted a photo:



Flickr PhotosDJI00736

prairiecode posted a video:


Flickr PhotosDJI00735

prairiecode posted a photo:



Flickr PhotosDJI00733

prairiecode posted a photo:



Flickr PhotosDJI00731

prairiecode posted a photo:



Flickr PhotosDJI00727

prairiecode posted a photo:



Flickr PhotosDJI00717

prairiecode posted a photo:



Flickr PhotosDJI00715

prairiecode posted a photo:



Flickr PhotosDJI00711

prairiecode posted a photo:



Flickr PhotosDJI00710

prairiecode posted a photo:



Flickr PhotosDJI00708

prairiecode posted a photo:



Flickr PhotosDJI00707

prairiecode posted a photo:



Flickr PhotosDJI00705

prairiecode posted a photo:



Flickr PhotosDJI00703

prairiecode posted a photo:



Flickr PhotosDJI00702

prairiecode posted a photo:



Flickr PhotosDJI00700

prairiecode posted a photo:



Flickr PhotosDJI00699

prairiecode posted a photo:



Flickr PhotosDJI00698

prairiecode posted a photo:



Flickr PhotosDJI00695

prairiecode posted a photo:



John's Blog2AM to Seattle

Monday morning, 1:45AM.

Laura and I walk into the boys’ room. We turn on the light. Nothing happens. (They’re sound sleepers.)

“Boys, it’s time to get up to go get on the train!”

Four eyes pop open. “Yay! Oh I’m so excited!”

And then, “Meow!” (They enjoy playing with their stuffed cats that Laura got them for Christmas.)

Before long, it was out the door to the train station. We even had time to stop at a donut shop along the way.

We climbed into our family bedroom (a sleeping car room on Amtrak specifically designed for families of four), and as the train started to move, the excitement of what was going on crept in. Yes, it’s 2:42AM, but these are two happy boys:

2014-08-04 02

Jacob and Oliver love trains, and this was the beginning of a 3-day train trip from Newton to Seattle that would take us through Kansas, Colorado, the Rocky Mountains of New Mexico, Arizona, Los Angeles, up the California coast, through the Cascades, and on to Seattle. Whew!

Here we are later that morning before breakfast:


Here’s our train at a station stop in La Junta, CO:


And at the beautiful small mountain town of Raton, NM:


Some of the passing scenery in New Mexico:


Through it all, we found many things to pass the time. I don’t think anybody was bored. I took the boys “exploring the train” several times — we’d walk from one end to the other and see what all was there. There was always the dining car for our meals, the lounge car for watching the passing scenery, and on the Coast Starlight, the Pacific Parlor Car.

Here we are getting ready for breakfast one morning.


Getting to select meals and order in the “train restaurant” was a big deal for the boys.


Laura brought one of her origami books, which even managed to pull the boys away from the passing scenery in the lounge car for quite some time.


Origami is serious business:


They had some fun wrapping themselves around my feet and challenging me to move. And were delighted when I could move even though they were trying to weight me down!


Several games of Uno were played, but even those sometimes couldn’t compete with the passing scenery:


The Coast Starlight features the Pacific Parlor Car, which was built over 50 years ago for the Santa Fe Hi-Level trains. They’ve been updated; the upper level is a lounge and small restaurant, and the lower level has been turned into a small theater. They show movies in there twice a day, but most of the time, the place is empty. A great place to go with little boys to run around and play games.


The boys and I sort of invented a new game: roadrunner and coyote, loosely based on the old Looney Tunes cartoons. Jacob and Oliver would be roadrunners, running around and yelling “MEEP MEEP!” Meanwhile, I was the coyote, who would try to catch them — even briefly succeeding sometimes — but ultimately fail in some hilarious way. It burned a lot of energy.

And, of course, the parlor car was good for scenery-watching too:


We were right along the Pacific Ocean for several hours – sometimes there would be a highway or a town between us and the beach, but usually there was nothing at all between us and the coast. It was beautiful to watch the jagged coastline go by, to gaze out onto the ocean, watching the birds — apparently so beautiful that I didn’t even think to take some photos.

Laura’s parents live in California, and took a connecting train. I had arranged for them to have a sleeping car room near ours, so for the last day of the trip, we had a group of 6. Here are the boys with their grandparents at lunch Wednesday:

2014-08-06 11

We stepped off the train in Seattle into beautiful King Street Station.


Our first day in Seattle was a quiet day of not too much. Laura’s relatives live near Lake Washington, so we went out there to play. The boys enjoyed gathering black rocks along the shore.


We went blackberry picking after that – filled up buckets for a cobbler.

The next day, we rode the Seattle Monorail. The boys have been talking about this for months — a kind of train they’ve never been on. That was the biggest thing in their minds that they were waiting for. They got to ride in the very front, by the operator.


Nice view from up there.


We walked through the Pike Market — I hadn’t been in such a large and crowded place like that since I was in Guadalajara:


At the Seattle Aquarium, we all had a great time checking out all the exhibits. The “please touch” one was a particular hit.


Walking underneath the salmon tank was fun too.

<object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" data="" height="281" type="application/x-shockwave-flash" width="500"><param name="flashvars" value="intl_lang=en-US&amp;photo_secret=2be96490ba&amp;photo_id=14991010632&amp;hd_default=false"/><param name="movie" value=""/><param name="bgcolor" value="#000000"/><param name="allowFullScreen" value="true"/><embed allowfullscreen="true" bgcolor="#000000" flashvars="intl_lang=en-US&amp;photo_secret=2be96490ba&amp;photo_id=14991010632&amp;hd_default=false" height="281" src="" type="application/x-shockwave-flash" width="500"></embed></object>

We spent a couple of days doing things closer to downtown. Laura’s cousin works at MOHAI, the Museum of History and Industry, so we spent a morning there. The boys particularly enjoyed the old periscope mounted to the top of the building, and the exhibit on chocolate (of course!)


They love any kind of transportation, so of course we had to get a ride on the Seattle Streetcar that comes by MOHAI.


All weekend long, we had been noticing the seaplanes taking off from Lake Washington and Lake Union (near MOHAI). So finally I decided to investigate, and one morning while Laura was doing things with her cousin, the boys and I took a short seaplane ride from one lake to another, and then rode every method of transportation we could except for ferries (we did that the next day). Here is our Kenmore Air plane:


The view of Lake Washington from 1000 feet was beautiful:


I think we got a better view than the Space Needle, and it probably cost about the same anyhow.


After splashdown, we took the streetcar to a place where we could eat lunch right by the monorail tracks. Then we rode the monorail again. Then we caught a train (it went underground a bit so it was a “subway” to them!) and rode it a few blocks.

There is even scenery underground, it seems.


We rode a bus back, and saved one last adventure for the next day: a ferry to Bainbridge Island.

2014-08-11 14

2014-08-11 16

Laura and I even got some time to ourselves to go have lunch at an amazing Greek restaurant to celebrate a year since we got engaged. It’s amazing to think that, by now, it’s only a few months until our wedding anniversary too!

There are many special memories of the weekend I could mention — visiting with Laura’s family, watching the boys play with her uncle’s pipe organ (it’s in his house!), watching the boys play with their grandparents, having all six of us on the train for a day, flying paper airplanes off the balcony, enjoying the cool breeze on the ferry and the beautiful mountains behind the lake. One of my favorites is waking up to high-pitched “Meow? Meow meow meow! Wake up, brother!” sorts of sounds. There was so much cat-play on the trip, and it was cute to hear. I have the feeling we won’t hear things like that much more.

So many times on the trip I heard, “Oh dad, I am so excited!” I never get tired of hearing that. And, of course, I was excited, too.

Picasa Photos2014-08-23


Date: Aug 23, 2014
Number of Photos in Album: 1

View Album

Flickr Photos2014-08-11 16

prairiecode posted a photo:

2014-08-11 16

Flickr Photos2014-08-11 14

prairiecode posted a photo:

2014-08-11 14

Flickr PhotosP8100197

prairiecode posted a photo:


Flickr PhotosP8100151

prairiecode posted a photo:


On the (Briefly) subway

Flickr PhotosP8100146

prairiecode posted a photo:


Flickr PhotosP8100117

prairiecode posted a photo:


Flickr PhotosP8100109

prairiecode posted a photo:


Flickr PhotosP8100100

prairiecode posted a photo:


Getting ready for a ride on the seaplane!

Flickr PhotosP8090094

prairiecode posted a photo: