Planet Goerzen

John's BlogUpdate on the systemd issue

The other day, I wrote about my poor first impressions of systemd in jessie. Here’s an update.

I’d like to start with the things that are good. I found the systemd community to be one of the most helpful in Debian, and #debian-systemd IRC channel to be especially helpful. I was in there for quite some time yesterday, and appreciated the help from many people, especially Michael. This is a nontechnical factor, but is extremely important; this has significantly allayed my concerns about systemd right there.

There are things about the systemd design that impress. The dependency system and configuration system is a lot more flexible than sysvinit. It is also a lot more complicated, and difficult to figure out what’s happening. I am unconvinced of the utility of parallelization of boot to begin with; I rarely reboot any of my Linux systems, desktops or servers, and it seems to introduce needless complexity.

Anyhow, on to the filesystem problem, and a bit of a background. My laptop runs ZFS, which is somewhat similar to btrfs in that it’s a volume manager (like LVM), RAID manager (like md), and filesystem in one. My system runs LVM, and inside LVM, I have two ZFS “pools” (volume groups): one, called rpool, that is unencrypted and holds mainly the operating system; and the other, called crypt, that is stacked atop LUKS. ZFS on Linux doesn’t yet have built-in crypto, which is why LVM is even in the picture here (to separate out the SSD at a level above ZFS to permit parts of it to be encrypted). This is a bit of an antiquated setup for me; as more systems have AES-NI, I’m going to everything except /boot being encrypted.

Anyhow, inside rpool is the / filesystem, /var, and /usr. Inside /crypt is /tmp and /home.

Initially, I tried to just boot it, knowing that systemd is supposed to work with LSB init scripts, and ZFS has init scripts with carefully-planned dependencies. This was evidently not working, perhaps because /lib/systemd/systemd/ It turns out that systemd has a few assumptions that turn out to be less true with ZFS than otherwise. ZFS filesystems are normally not mounted via /etc/fstab; a ZFS pool has internal properties about which dataset gets mounted where (similar to LVM’s actions after a vgscan and vgchange -ay). Even though there are ordering constraints in the units, systemd is writing files to /var before /var gets mounted, resulting in the mount failing (unlike ext4, ZFS by default will reject an attempt to mount over a non-empty directory). Partly this due to the debian-fixup.service, and partly it is due to systemd reacting to udev items like backlight.

This problem was eventually worked around by doing zfs set mountpoint=legacy rpool/var, and then adding a line to fstab (“rpool/var /var zfs defaults 0 2″) for /var and its descendent filesystems.

This left the problem of /tmp; again, it wasn’t getting mounted soon enough. In this case, it required crypttab to be processed first, and there seem to be a lot of bugs in the crypttab processing in systemd (more on that below). I eventually worked around that by adding After=cryptsetup.target to the zfs-import-cache.service file. For /tmp, it did NOT work to put it in /etc/fstab, because then it tried to mount it before starting cryptsetup for some reason. It probably didn’t help that the system’s cryptdisks.service is a symlink to /dev/null, a fact I didn’t realize until after a lot of needless reboots.

Anyhow, one thing I stumbled across was poor console control with systemd. On numerous occasions, I had things like two cryptsetup processes trying to read a password, plus an emergency mode console trying to do so. I had this memorable line of text at one point:

(or type Control-D to continue): Please enter passphrase for disk athena-crypttank (crypt)! [ OK ] Stopped Emergency Shell.

And here we venture into unsatisfying territory with systemd. One answer to this in IRC was to install plymouth, which apparently serializes console I/O. However, plymouth is “an attractive boot animation in place of the text messages that normally get shown.” I don’t want an “attractive boot animation”. Nevertheless, neither systemd-sysv nor cryptsetup depends on plymouth, so by default, the prompt for a password at boot is obscured by various other text.

Worse, plymouth doesn’t support serial consoles, so at the moment booting a system that uses LUKS with systemd over a serial console is a matter of blind luck of typing the right password at the right time.

In the end, though, the system booted and after a few more tweaks, the backlight buttons do their thing again. Whew!

Update 2014-10-13: uau pointed out that Plymouth is more than a bootsplash, and can work with serial consoles, despite the description of the package. I stand corrected on that. (It is still the case, however, that packages don’t depend on it where they should, and the default experience for people using cryptsetup is not very good.)

John's BlogFirst impressions of systemd, and they’re not good

Well, I finally bit the bullet. My laptop, which runs jessie, got dist-upgraded for the first time in a few months. My brightness keys stopped working, and it no longer would suspend to RAM when the lid was closed, and upon chasing things down from XFCE to policykit, eventually it appears that suddenly major parts of the desktop breaks without systemd in jessie. Sigh.

So apt-get install systemd-sysv (and watch sysvinit-core get uninstalled) and reboot.

Only, my system doesn’t come back up. In fact, over several hours of trying to make it boot with systemd, it failed in numerous spectacular and hilarious (or, would be hilarious if my laptop would boot) ways. I had text obliterating the cryptsetup password prompt almost every time. Sometimes there were two processes trying to read a cryptsetup password at once. Sometimes a process was trying to read that while another one was trying to read an emergency shell password. Many times it tried to write to /var and /tmp before they were mounted, meaning they *wouldn’t* mount because there was stuff there.

I noticed it not doing much with ZFS, complaining of a dependency loop between zfs-mount and $local-fs. I fixed that, but it still wouldn’t boot. In fact, it simply hung after writing something about wall passwords.

I’ve dug into systemd, finding a “unit generator for fstab” (whatever the hack that is, it’s not at all made clear by systemd-fstab-generator(8)).

In some cases, there’s info in journalctl, but if I can’t even get to an emergency mode prompt, the practice of hiding all stdout and stderr output is not all that pleasant.

I remember thinking “what’s all the flaming about?” systemd wasn’t my first choice (I always thought “if it ain’t broke, don’t fix it” about sysvinit), but basically ignored the thousands of messages, thinking whatever happens, jessie will still boot.

Now I’m not so sure. Even if the thing boots out of the box, it seems like the boot process with systemd is colossally fragile.

For now, at least zfs rollback can undo upgrades to 800 packages in about 2 seconds. But I can’t stay at some early jessie checkpoint forever.

Have we made a vast mistake that can’t be undone? (If things like even *brightness keys* now require systemd…)

John's BlogThe Thrill and Stress of Too Many Hobbies

Today, 4PM. Jacob and Oliver excitedly peer at the box in our kitchen – a really big box, taller than them. Inside is is the first model airplane I’d ever purchased. The three of us hunkered down on the kitchen floor, opened the box, unpacked the parts, examined the controller, and found the manual with cryptic assembly directions. Oliver turned some screws while Jacob checked out the levers on the controllers. Then they both left for a bit to play with their toy buses.

A little while later, the three of us went outside. It was too windy to fly. I had never flied an RC plane before — only RC quadcopters (much easier to fly), and some practice time on an RC simulator. But the excitement was too much. So out we went, and the plane took off perfectly, climbed, flew over the trees, and circled above our heads at my command. I even managed a good landing in the wind, despite about 5 aborted attempts due to coming in too high, wrong angle, too fast, or last-minute gusts of wind throwing everything off. I am not sure how I pulled that all off on my first flight, but somehow I did! It was thrilling!

I’ve had a lot of hobbies in my life. Computers have run through many of them; I learned Pascal (a programming language) at about the same time I learned cursive handwriting and started with C at around age 10. It was all fun. I’ve been a Debian developer for some 18 years now, and have written a lot of code, and even books about code, over the years.

Photography, music, literature, history, philosophy, and theology have been interests for quite some time as well. In the last few years, I’ve picked up amateur radio, model aircraft, etc. And last month, Laura led me into Ada’s Technical Books during our visit to Seattle, resulting in me getting interested in Arduino. (The boys and I have already built a light-activated crossing gate for their HO-gauge model trains, and Jacob can now say he’s edited a few characters of C!)

Sometimes I find ways to merge hobbies; I’ve set up all sorts of amateur radio systems on Linux, take aerial photographs, and set up systems to stream music in my house.

But I also have a lot less time for hobbies overall than I once did; other things in life, such as my children, are more important. Some of the code I once worked on actively I no longer use or maintain, and I feel guilty about that when people send bug reports that I have no interest in fixing anymore.

Sometimes I feel a need to cut down, and perhaps have; and then, I get an interest in RC aircraft and find an airplane that is great for a beginner and fairly inexpensive.

Perhaps it is the curse of being a curious person living in an interesting world. Do any of the rest of you have a large number of hobbies? How do you feel about that?

Flickr PhotosDJI00719

prairiecode posted a video:

DJI00719

Flickr PhotosDJI00734

prairiecode posted a video:

DJI00734

Flickr PhotosDJI00741

prairiecode posted a photo:

DJI00741

Dcim\100media

Flickr PhotosDJI00739

prairiecode posted a photo:

DJI00739

Dcim\100media

Flickr PhotosDJI00736

prairiecode posted a video:

DJI00736

Flickr PhotosDJI00735

prairiecode posted a photo:

DJI00735

Dcim\100media

Flickr PhotosDJI00733

prairiecode posted a photo:

DJI00733

Dcim\100media

Flickr PhotosDJI00731

prairiecode posted a photo:

DJI00731

Dcim\100media

Flickr PhotosDJI00727

prairiecode posted a photo:

DJI00727

Dcim\100media

Flickr PhotosDJI00717

prairiecode posted a photo:

DJI00717

Dcim\100media

Flickr PhotosDJI00715

prairiecode posted a photo:

DJI00715

Dcim\100media

Flickr PhotosDJI00711

prairiecode posted a photo:

DJI00711

Dcim\100media

Flickr PhotosDJI00710

prairiecode posted a photo:

DJI00710

Dcim\100media

Flickr PhotosDJI00708

prairiecode posted a photo:

DJI00708

Dcim\100media

Flickr PhotosDJI00707

prairiecode posted a photo:

DJI00707

Dcim\100media

Flickr PhotosDJI00705

prairiecode posted a photo:

DJI00705

Dcim\100media

Flickr PhotosDJI00703

prairiecode posted a photo:

DJI00703

Dcim\100media

Flickr PhotosDJI00702

prairiecode posted a photo:

DJI00702

Dcim\100media

Flickr PhotosDJI00700

prairiecode posted a photo:

DJI00700

Dcim\100media

Flickr PhotosDJI00699

prairiecode posted a photo:

DJI00699

Dcim\100media

Flickr PhotosDJI00698

prairiecode posted a photo:

DJI00698

Dcim\100media

Flickr PhotosDJI00695

prairiecode posted a photo:

DJI00695

Dcim\100media

John's Blog2AM to Seattle

Monday morning, 1:45AM.

Laura and I walk into the boys’ room. We turn on the light. Nothing happens. (They’re sound sleepers.)

“Boys, it’s time to get up to go get on the train!”

Four eyes pop open. “Yay! Oh I’m so excited!”

And then, “Meow!” (They enjoy playing with their stuffed cats that Laura got them for Christmas.)

Before long, it was out the door to the train station. We even had time to stop at a donut shop along the way.

We climbed into our family bedroom (a sleeping car room on Amtrak specifically designed for families of four), and as the train started to move, the excitement of what was going on crept in. Yes, it’s 2:42AM, but these are two happy boys:

2014-08-04 02

Jacob and Oliver love trains, and this was the beginning of a 3-day train trip from Newton to Seattle that would take us through Kansas, Colorado, the Rocky Mountains of New Mexico, Arizona, Los Angeles, up the California coast, through the Cascades, and on to Seattle. Whew!

Here we are later that morning before breakfast:

IMG_3776

Here’s our train at a station stop in La Junta, CO:

IMG_3791

And at the beautiful small mountain town of Raton, NM:

IMG_3805

Some of the passing scenery in New Mexico:

IMG_3828

Through it all, we found many things to pass the time. I don’t think anybody was bored. I took the boys “exploring the train” several times — we’d walk from one end to the other and see what all was there. There was always the dining car for our meals, the lounge car for watching the passing scenery, and on the Coast Starlight, the Pacific Parlor Car.

Here we are getting ready for breakfast one morning.

IMG_3830

Getting to select meals and order in the “train restaurant” was a big deal for the boys.

IMG_3832

Laura brought one of her origami books, which even managed to pull the boys away from the passing scenery in the lounge car for quite some time.

IMG_3848

Origami is serious business:

IMG_3869

They had some fun wrapping themselves around my feet and challenging me to move. And were delighted when I could move even though they were trying to weight me down!

IMG_3880

Several games of Uno were played, but even those sometimes couldn’t compete with the passing scenery:

IMG_3898

The Coast Starlight features the Pacific Parlor Car, which was built over 50 years ago for the Santa Fe Hi-Level trains. They’ve been updated; the upper level is a lounge and small restaurant, and the lower level has been turned into a small theater. They show movies in there twice a day, but most of the time, the place is empty. A great place to go with little boys to run around and play games.

IMG_3896

The boys and I sort of invented a new game: roadrunner and coyote, loosely based on the old Looney Tunes cartoons. Jacob and Oliver would be roadrunners, running around and yelling “MEEP MEEP!” Meanwhile, I was the coyote, who would try to catch them — even briefly succeeding sometimes — but ultimately fail in some hilarious way. It burned a lot of energy.

And, of course, the parlor car was good for scenery-watching too:

IMG_3908

We were right along the Pacific Ocean for several hours – sometimes there would be a highway or a town between us and the beach, but usually there was nothing at all between us and the coast. It was beautiful to watch the jagged coastline go by, to gaze out onto the ocean, watching the birds — apparently so beautiful that I didn’t even think to take some photos.

Laura’s parents live in California, and took a connecting train. I had arranged for them to have a sleeping car room near ours, so for the last day of the trip, we had a group of 6. Here are the boys with their grandparents at lunch Wednesday:

2014-08-06 11

We stepped off the train in Seattle into beautiful King Street Station.

P8100197

Our first day in Seattle was a quiet day of not too much. Laura’s relatives live near Lake Washington, so we went out there to play. The boys enjoyed gathering black rocks along the shore.

IMG_3956

We went blackberry picking after that – filled up buckets for a cobbler.

The next day, we rode the Seattle Monorail. The boys have been talking about this for months — a kind of train they’ve never been on. That was the biggest thing in their minds that they were waiting for. They got to ride in the very front, by the operator.

P8080073

Nice view from up there.

P8080078

We walked through the Pike Market — I hadn’t been in such a large and crowded place like that since I was in Guadalajara:

P8080019

At the Seattle Aquarium, we all had a great time checking out all the exhibits. The “please touch” one was a particular hit.

P8080038

Walking underneath the salmon tank was fun too.

<object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" data="https://www.flickr.com/apps/video/stewart.swf" height="281" type="application/x-shockwave-flash" width="500"><param name="flashvars" value="intl_lang=en-US&amp;photo_secret=2be96490ba&amp;photo_id=14991010632&amp;hd_default=false"/><param name="movie" value="https://www.flickr.com/apps/video/stewart.swf"/><param name="bgcolor" value="#000000"/><param name="allowFullScreen" value="true"/><embed allowfullscreen="true" bgcolor="#000000" flashvars="intl_lang=en-US&amp;photo_secret=2be96490ba&amp;photo_id=14991010632&amp;hd_default=false" height="281" src="https://www.flickr.com/apps/video/stewart.swf" type="application/x-shockwave-flash" width="500"></embed></object>

We spent a couple of days doing things closer to downtown. Laura’s cousin works at MOHAI, the Museum of History and Industry, so we spent a morning there. The boys particularly enjoyed the old periscope mounted to the top of the building, and the exhibit on chocolate (of course!)

P8100146

They love any kind of transportation, so of course we had to get a ride on the Seattle Streetcar that comes by MOHAI.

P8090094

All weekend long, we had been noticing the seaplanes taking off from Lake Washington and Lake Union (near MOHAI). So finally I decided to investigate, and one morning while Laura was doing things with her cousin, the boys and I took a short seaplane ride from one lake to another, and then rode every method of transportation we could except for ferries (we did that the next day). Here is our Kenmore Air plane:

P8100100

The view of Lake Washington from 1000 feet was beautiful:

P8100109

I think we got a better view than the Space Needle, and it probably cost about the same anyhow.

P8100117

After splashdown, we took the streetcar to a place where we could eat lunch right by the monorail tracks. Then we rode the monorail again. Then we caught a train (it went underground a bit so it was a “subway” to them!) and rode it a few blocks.

There is even scenery underground, it seems.

P8100151

We rode a bus back, and saved one last adventure for the next day: a ferry to Bainbridge Island.

2014-08-11 14

2014-08-11 16

Laura and I even got some time to ourselves to go have lunch at an amazing Greek restaurant to celebrate a year since we got engaged. It’s amazing to think that, by now, it’s only a few months until our wedding anniversary too!

There are many special memories of the weekend I could mention — visiting with Laura’s family, watching the boys play with her uncle’s pipe organ (it’s in his house!), watching the boys play with their grandparents, having all six of us on the train for a day, flying paper airplanes off the balcony, enjoying the cool breeze on the ferry and the beautiful mountains behind the lake. One of my favorites is waking up to high-pitched “Meow? Meow meow meow! Wake up, brother!” sorts of sounds. There was so much cat-play on the trip, and it was cute to hear. I have the feeling we won’t hear things like that much more.

So many times on the trip I heard, “Oh dad, I am so excited!” I never get tired of hearing that. And, of course, I was excited, too.

Picasa Photos2014-08-23

2014-08-23

Date: Aug 23, 2014
Number of Photos in Album: 1

View Album

Flickr Photos2014-08-11 16

prairiecode posted a photo:

2014-08-11 16

Flickr Photos2014-08-11 14

prairiecode posted a photo:

2014-08-11 14

Flickr PhotosP8100197

prairiecode posted a photo:

P8100197

Flickr PhotosP8100151

prairiecode posted a photo:

P8100151

On the (Briefly) subway

Flickr PhotosP8100146

prairiecode posted a photo:

P8100146

Flickr PhotosP8100117

prairiecode posted a photo:

P8100117

Flickr PhotosP8100109

prairiecode posted a photo:

P8100109

Flickr PhotosP8100100

prairiecode posted a photo:

P8100100

Getting ready for a ride on the seaplane!

Flickr PhotosP8090094

prairiecode posted a photo:

P8090094

Flickr PhotosP8080078

prairiecode posted a photo:

P8080078

Flickr PhotosP8080073

prairiecode posted a photo:

P8080073

Flickr PhotosP8080061

prairiecode posted a video:

P8080061

Salmon

Flickr PhotosP8080038

prairiecode posted a photo:

P8080038

Flickr PhotosP8080019

prairiecode posted a photo:

P8080019

Flickr PhotosIMG_3956

prairiecode posted a photo:

IMG_3956

Flickr Photos2014-08-06 11

prairiecode posted a photo:

2014-08-06 11

Lunch in the parlor car, with the boys' grandparents

Flickr PhotosIMG_3908

prairiecode posted a photo:

IMG_3908

Oliver and his stuffed cat Roo watching the scenery

Flickr PhotosIMG_3898

prairiecode posted a photo:

IMG_3898

Hats!

Flickr PhotosIMG_3896

prairiecode posted a photo:

IMG_3896

The Pacific Parlor car on the Coast Startlight, lower level (theater).

Flickr PhotosIMG_3880

prairiecode posted a photo:

IMG_3880

The boys tried to see if they were too heavy to prevent me from moving if they sat on my feet. Nope, not yet.

Flickr PhotosDJI00561

prairiecode posted a photo:

DJI00561

Dcim\100media

Flickr PhotosDJI00559

prairiecode posted a photo:

DJI00559

Dcim\100media

Flickr PhotosDJI00558

prairiecode posted a photo:

DJI00558

Dcim\100media

Flickr PhotosDJI00557

prairiecode posted a photo:

DJI00557

Dcim\100media

Flickr PhotosDJI00556

prairiecode posted a photo:

DJI00556

Dcim\100media

Flickr PhotosDJI00551

prairiecode posted a photo:

DJI00551

Dcim\100media

Flickr PhotosDJI00548

prairiecode posted a photo:

DJI00548

Dcim\100media

Flickr PhotosDJI00546

prairiecode posted a photo:

DJI00546

Dcim\100media

Flickr PhotosDJI00545

prairiecode posted a photo:

DJI00545

Dcim\100media

Flickr PhotosDJI00543

prairiecode posted a photo:

DJI00543

Dcim\100media

Flickr PhotosDJI00538

prairiecode posted a photo:

DJI00538

Dcim\100media

Flickr PhotosDJI00535

prairiecode posted a photo:

DJI00535

Dcim\100media

Flickr PhotosDJI00532

prairiecode posted a video:

DJI00532

slow tractor flyover and landing

Footnotes