Planet Goerzen

John's BlogA Dry Spring

Spring in the prairie is a bit of an odd thing this year. Here and there, near ditches and creeks, a short, soft blanket of lush green grass covers the ground. A few feet away, patches of green are visible between the brown shoots of last year’s grass. Some trees are already turning green, purple, red, and white, while others stand still and brown, stubbornly insisting that spring is not here yet. To look at the thermometer may not be much guide either; two days after the temperature was nearly 90, we woke to see a dusting of snow on the ground.

It’s been dry, terribly try in Kansas. Grass next to a gravel driveway or road often has a chalkish layer of dust on it, kicked up by passing cars or even a stiff wind. The earth thirsts.

It is somehow fitting to celebrate Easter, that spring holiday, in the midst of the dry ground, to remember that water is not the only thing that can quench thirst.

Easter morning began sleepily, as we got up early to head to a sunrise service. It was in a pasture just outside a small Kansas town, and we gathered there at about 6:15, wearing only light jackets against the breeze. A fire was burning, and there was water on hand to quickly douse any grass that caught first that wasn’t supposed to — and it was occasionally used.

I was doing the prelude for the service, playing on my penny whistle. I enjoyed being able to do that, and was glad that the wind was calm enough that it didn’t interfere too much with the music.

We sang some hymns, listened to some Bible readings, and just stood in silence, listening to the crackle of the fire, some country dogs playing, and watching the sky to the east transform as the sun came up.

Then it was on to church for breakfast, and a break before the Easter service — the pipe organ ringing, piano playing with it, and deep trombone and full sanctuary of people singing our 4-part Easter hymns celebrating the day. Laura had the idea of pinning carnations onto the cross, and we got to watch everyone come up and add theirs.

Jacob and Oliver enjoyed the sunrise service. They decided they would keep a watchful eye on the first and the dogs, they enjoyed muffins at breakfast and playing in the church after that. But if you are 4 or 7, what is Easter without an Easter egg hunt? And they got in several.

Laura and I hid some eggs around the yard. Jacob asked me to use a radio to tell them when the eggs were ready. Here they are, bounding out the door to begin the hunt!

And, of course, if you are 4 or 7 and have a geek for a dad, you will naturally think to bring radios with you to the next hunt. To tell your brother what you’re finding, of course.

It was a good weekend, and in fact, Jacob even volunteered to put up a “wet floor” sign after he spilled some water:

On the last car ride of the day, Jacob decided he would write a story about his Easter. He decided he would publish a big book, and be a famous author and make other children happy. Oliver, of course, decided he needed an Easter story also. We couldn’t very well publish a book in the car, but I did manage to use my phone to capture their stories.

It’s been a long and busy week, but there is much to be joyful about, even when tired.

Picasa Photos2014-04 easter

2014-04 easter

Date: Apr 21, 2014
Number of Photos in Album: 20

View Album

John's BlogSpringtime in the Mountains

The scene: early one morning as the sun has just started to rise. Jacob and Oliver, ages 7 and 4, are the first people to wake up in the house — their grandparents’ in California, where the four of us are visiting for the first time as a family.

They have a conversation and decide that would be a good to go “find a mystery.”

They decide to take their flashlights — pink and blue, matching each boy’s favorite color — and slowly, but not very quietly, open their bedroom door and creep out.

“Brother, you forgot your flashlight!” says Oliver.

“Oh, thanks brother! I’ll get it!” says Jacob.

Meanwhile, Laura’s mom wakes up, and notices two boys with flashlights creeping through the living room. Pretty soon they reach the kitchen, open the dishwasher, spy a suspicious-looking bowl, and decide that they have found the mystery — a clean bowl!

Or, at least that’s the story that I pieced together based on what a 4-year-old and the grandma he awakened told me.

We were on our first family trip to the Fresno, California area, to visit Laura’s parents — Jacob and Oliver’s new grandparents. They’ve played together before, but as this was our first visit to their place, there was quite the excitement. The boys had flown before, but it was several years ago and neither of them remember it well, so they were excited about that, too.

The night before, Jacob woke up to tell me “Dad, I am too excited to sleep. I think I will go downstairs and watch some TV.” He didn’t get too far with that plan. But he was excited. We went through security at the laid-back Wichita airport (where the TSA agents smile and there are often no security lines at all). We found our gate with enough time to grab lunch, which we did. The boys and I then did what we often do to kill time: explore. We explored the terminal, watching carpet-layers cut out carpet for the jetway, watching the construction of the new Terminal 3 out the window. And, of course, watching airplanes take off and land from the terminal’s large windows.

Finally it was our turn to board, and we all got on the plane: Jacob and Oliver with their backpacks of on-board activites, Laura and me with the rest of our carryon luggage, for the short trip to Denver.

Jacob and Oliver’s noses were pressed against the windows. Or, well, Jacob’s was. Oliver’s window was a little too high for him, but he was thrilled anyhow. They delighted in the airplane snacks, and the fact that they were allowed to drink pop on the plane. We packed books and some new art supplies for them (colored post-its, pages from a train-themed page-a-day calendar, a notebook, and a set of colored pens really seemed to do the trick.)

We had a choice of 35 minutes or 4 hours between flights in Denver, and I had chosen 4 hours, thinking that would be a lot less stressful with boys. And it was. We found a nice corner of the mezzanine to sit for awhile — they did art projects and played a game with Laura. Then I took them exploring Denver. We rode the moving sidewalks up and down the terminal, took a train ride to another terminal and back, ate supper all together, and flew to Fresno.

We had stopped in the Wichita airport to buy them each a souvenir airplane, and these came out often during the rest of the trip.

They enjoyed the mockups of the sequoias in Fresno Yosemite International Airport, enjoyed their beds and their room at the house, and did actually manage to fall asleep eventually.

We had a few days there, where they played in a park, with bubbles on the patio, or croquet in the yard (I even discovered Jacob happily using the cast his broken arm is in as a hammer to pound the hoops into the ground!)

There are a lot of miniatures in the house, and the boys enjoyed exploring the dollhouses — and especially the N-gauge model train. Jacob enjoyed it so much he asked me to record a video of him playing with the trains.

Evenings often brought book-reading, from the many children’s books in the house. At home, Laura and I and both boys often scrunch onto an oversized chair and read a book and sing a song (one I make up on whatever topic they choose). Over there, we often had Laura, Jacob, Oliver, Laura’s mom, and me scrunched up somewhere while the boys heard a story read to them by their grandma. That happened plenty of times other than bedtime, too. (Or Jacob would take his favorite books and read them to himself.)

Laura’s parents organized a reception Sunday for us, for the people from that area that couldn’t make it to our wedding. Jacob and Oliver, predictably, had fun playing and even talked to some of the adults. The adults that didn’t ask Jacob about his cast, anyhow (he dislikes talking about it).

The boys discovered a live mic at the church where the reception was, and do I detect two future pastors in our midst?

We had a great time at Laura’s uncle and aunt’s place. The boys were happy to discover an orange tree in their backyard, a tetherball post not far away, and an uncle ready to give them a demonstration of a “swimming pool vacuum cleaner” or sit at the piano with them. Jacob’s favorite part, though, was when the hamburger buns his great uncle were toasting were left on the grill during the prayer before the meal, got a bit scorched, and the uncle remarked with a chuckle that “I guess the Lord was tired of listening to me drone on!” Jacob loved his meal, and cackled at the thought of a prayer causing buns to get scorched.

But their highlight was the visit to the sequoias at Kings Canyon National Park the next day. The excitement had been building for that day all weekend. On the way out, we stopped at a fruit stand and bought some delicious strawberries — the fresh, juicy, sweet and tasty kind that are red all the way through. We continued up through the foothills, stopping periodically to get out and stretch, look at the sights, take some photos, or borrow grandpa’s binoculars.

I knew we’d be traveling in two cars, so I had the thought to pack some 2-way radios before we left. I gave one to the boys and one to the grandparents. All weekend long, whenever the six of us went somewhere, the boys (and especially Jacob) would give directions to the car that was following. “Turn right! … The light is green! … Catch up, you’re going too slow!” So all the way into the mountains, Jacob would send back instructions on what to do.

We saw Grant Grove, home to the worlds third-largest tree (267ft/81m tall and 3000 years old). It’s quite the impressive tree — the trunk’s diameter near the ground is 29ft or almost 9m. As we walked the trails, their speed kept increasing as they were hunting for the “tree tunnel” I had told them about — a tree that fell centuries ago and had been hollowed out to make a home. That trunk was easily 8ft or more in diameter, and I could stand up completely in places. We found it, to much delight from the boys — “So this is what it’s like to be inside a tree!”

Our trip home brought a delay in Denver and a missed flight, which excited the boys when I told them “now we get to eat supper in the airport!” I wonder how long that tactic will work… But Jacob was also excited because the plane we were put on in the end was bigger than the one we were scheduled on, so that was another piece of excitement.

We got home, and I carried two sleeping boys in from the car, upstairs, tucked them in, pulled off their shoes, and put their favorite stuffed animals in their arms. They were happy to be home, and with memories to treasure for a long time.

Picasa Photos2014 Fresno Web

2014 Fresno Web

Date: Mar 31, 2014
Number of Photos in Album: 8

View Album

John's BlogAgile Is Dead (Long Live Agility)

In an intriguing post, PragDave laments how empty the word “agile” has become. To paraphrase, I might say he’s put words to a nagging feeling I’ve had: that there are entire books about agile, conferences about agile, hallway conversations I’ve heard about whether somebody is doing this-or-that agile practice correctly.

Which, when it comes down to it, means that they’re not being agile. If process and tools, even if they’re labeled as “agile” processes and tools, are king, then we’ve simply replaced one productivity-impairing dictator with another.

And he makes this bold statement:

Here is how to do something in an agile fashion:

What to do:

  • Find out where you are
  • Take a small step towards your goal
  • Adjust your understanding based on what you learned
  • Repeat

How to do it:

When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier.

Those four lines and one practice encompass everything there is to know about effective software development.

He goes on to dive into that a bit, of course, but I think this man has a rare gift of expressing something complicated so succinctly. I am inclined to believe he is right.

Picasa Photos2014-03 laura's pie

2014-03 laura's pie

Date: Mar 11, 2014
Number of Photos in Album: 9

View Album

John's BlogGood all-day light laptop?

I am trying to find a laptop with all-day battery life that’s fairly light. Perhaps I am dreaming too big here, but I thought I’d toss out my hopes and see if there are any recommendations.

I am hoping for:

  • Battery life around 9 hours powered up
  • Fairly light. 2 or 3 pounds would be good.
  • Small. 10″ to 12″ screen is fine.
  • CPU – needs to be something “real”. No ARM or Atom.
  • Storage – size is less important than performance.
  • Screen – resolution has to be better than 768 vertical lines.
  • Ideally, capable of running Debian well.

The point is that I want to be able to do general web browsing with a real browser (Firefox), run Thunderbird for mail, hopefully even be able to run a VM or two under KVM or VirtualBox (with the understanding that this will kill battery life).

I have been using a Thinkpad T420s for a couple of years now. Its battery life wasn’t great even when it was new, and is worse now, of course. An Asus TF700t tablet has great battery life, but the storage system in it is so slow that I rather suspect that the browser cache is hurting, not helping, performance.

I am also rather disappointed with the Android system. I can’t really develop for it with my usual tools. Enough components are closed that, like Windows or MacOS X, I can’t feel like I can truly trust it with sensitive data. Although it has SSH clients available, the SSH server there wants you to pay to use public key auth, and the SSH clients don’t work all that well. Git can work, sort of. Battery life is great, and the keyboard is fine, but even flashing a different OS won’t fix that terrible performance.

I’ve had people recommend certain Asus laptops, a Microsoft Surface Pro, or a Macbook Air. Any thoughts?

John's BlogBacking up every few minutes with simplesnap

I’ve written a lot lately about ZFS, and one of its very nice features is the ability to make snapshots that are lightweight, space-efficient, and don’t hurt performance (unlike, say, LVM snapshots).

ZFS also has “zfs send” and “zfs receive” commands that can send the content of the snapshot, or a delta between two snapshots, as a data stream – similar in concept to an amped-up tar file. These can be used to, for instance, very efficiently send backups to another machine. Rather than having to stat() every single file on a filesystem as rsync has to, it sends effectively an intelligent binary delta — which is also intelligent about operations such as renames.

Since my last search for backup tools, I’d been using BackupPC for my personal systems. But since I switched them to ZFS on Linux, I’ve been wanting to try something better.

There are a lot of tools out there to take ZFS snapshots and send them to another machine, and I summarized them on my wiki. I found zfSnap to work well for taking and rotating snapshots, but I didn’t find anything that matched my criteria for sending them across the network. It seemed par for the course for these tools to think nothing of opening up full root access to a machine from others, whereas I would much rather lock it down with command= in authorized_keys.

So I wrote my own, called simplesnap. As usual, I wrote extensive documentation for it as well, even though it is very simple to set up and use.

So, with BackupPC, a backup of my workstation took almost 8 hours. (Its “incremental” might take as few as 3 hours) With ZFS snapshots and simplesnap, it takes 25 seconds. 25 seconds!

So right now, instead of backing up once a day, I back up once an hour. There’s no reason I couldn’t back up every 5 minutes, in fact. The data consumes less space, is far faster to manage, and doesn’t require a nightly hours-long cleanup process like BackupPC does — zfs destroy on a snapshot just takes a few seconds.

I use a pair of USB disks for backups, and rotate them to offsite storage periodically. They simply run ZFS atop dm-crypt (for security) and it works quite well even on those slow devices.

Although ZFS doesn’t do file-level dedup like BackupPC does, and the lz4 compression I’ve set ZFS to use is less efficient than the gzip-like compression BackupPC uses, still the backups are more space-efficient. I am not quite sure why, but I suspect it’s because there is a lot less metadata to keep track of, and perhaps also because BackupPC has to store a new copy of a file if even a byte changes, whereas ZFS can store just the changed blocks.

Incidentally, I’ve packaged both zfSnap and simplesnap for Debian and both are waiting in NEW.

John's BlogMigrated from Hetzner to OVH hosting

Since August 2011, my sites such as complete.org have been running on a Xen-backed virtual private server (VPS) at Hetzner Online, based in Germany. I had what they called their VQ19 package, which included 2GB RAM, 80GB HDD, 100Mb NIC and 4TB transfer.

Unlike many other VPS hosts, I never had performance problems. However, I did sometimes have hardware problems with the host, and it could take hours to resolve. Their tech support only works business hours German time, which was also a problem.

Meanwhile, OVH, a large European hosting company, recently opened a datacenter in Canada. Although they no longer offer their value-line Kimsufi dedicated servers there — starting at $11.50/mo — they do offer their midrange SoYouStart servers there. $50/mo gets a person a 4-core 3.2GHz Xeon server with 32GB RAM, 2x2TB SATA HDD, 200Mbps bandwidth. Not bad at all! The Kimsufi options are still good for lower-end needs as well.

I signed up for one of the SoYouStart servers. I’ve been pleased with my choice to migrate, and at the possibilities that having hardware like that at my disposal open up, but it is not without its downside.

The primary downside is lack of any kind of KVM console. If the server doesn’t boot, I can’t see the Grub error message (or whatever) behind it. They do provide hardware support and automatic technician dispatching when the server isn’t pingable, but… they state they have no KVM access at all. They support many OS flavors, and have a premade image for them, but there is no using a custom ISO to install; if you want ZFS on Linux, for instance, you can’t very easily build it into root.

My server was promised within 72 hours, but delivered much quicker: within about 1. I had two times they said they had to replace a motherboard within the first day; once they did it in 30 minutes, and the other took them 2.5 hours for some reason. They do have phone support, which answers almost immediately, but the people there are not the people actually in the datacenter. It was frustrating with a server down for hours and nobody really commenting on what was going on.

The server performs quite well, and after the initial issues, I’ve been happy.

I was initially planning an all-ZFS installation. SoYouStart does offer a rescue environment, but it doesn’t support ZFS, so I figured I better stick with an ext4 root at least. The default Debian install uses RAID1 on md-raid, with a 20GB root partition and the rest of the 2TB drive in /home, and then a swap partition on each drive (mysteriously NOT in the RAID!) So I broke the mirror on /home and converted those into the two legs of a mirrored vdev for a zpool.

I run all of the real work inside KVM VMs, so that should minimize the number of times I have to do anything to the root filesystem that could cause trouble.

SoYouStart includes 100GB of space on a separate FTP server for backup purposes. I have scripts that upload nightly tarballs of the root filesystem, plus full “zfs send” streams of everything else. Every hour, it uploads an incremental “zfs send” stream as well. This all works quite nicely; even if the machine is a complete loss, I’d never lose more than an hour’s work, and could restore it completely from a rescue environment. Very nice!

I’ll write more in a few days about the ZFS setup I’m using, and some KVM discoveries as well.

John's BlogVirtFS isn’t quite ready

Despite claims to the contrary [PDF], VirtFS — the 9P-based virtio KVM/QEMU layer designed to pass through a host’s filesystem to the guest — is quite slow. I have yet to get it to perform at even 1/10 the speed of the virtual block device (VBD). That’s unfortunate, because in theory it should be significantly faster. At this rate, I suspect even NFS will be significantly faster.

Beyond that, it seems impossible to use VirtFS as the root filesystem in a VM, at least with Debian; initramfs-tools doesn’t know how to build an initrd in that situation, and the support is just not there.

It would make a great combination with btrfs or zfs, but unfortunately looks to be just not ready yet.

John's BlogHow to fix “fstrim: Operation not supported” under KVM?

Maybe someone out there will have some ideas.

I have a KVM host running wheezy, with wheezy-backports versions of libvirt and qemu. I have defined a guest, properly set discard=unmap in the domain XML file for it, verified that’s being passed to the guest, but TRIM/DISCARD is just not working.

Mounting the ext4 filesystem with discard has no effect, and fstrim / always reports:

fstrim: /: FITRIM ioctl failed: Operation not supported

Every single time.

I’ve tried with the virtio, IDE, and SCSI (both default and virtio-scsi) backend drivers. The guest is also running wheezy (i386 version; the host is amd64) and I’ve tried the latest 3.12 backported kernel for it. No dice.

If I shut down the VM and mount the filesystem on the host, fstrim works fine.

Everything says this should work. But it doesn’t.

Any ideas?

Picasa Photos2014-02-05 snow

2014-02-05 snow

Date: Feb 5, 2014
Number of Photos in Album: 4

View Album

John's BlogWhy and how to run ZFS on Linux

I’m writing a bit about ZFS these days, and I thought I’d write a bit about why I am using it, why it might or might not be interesting for you, and what you might do about it.

ZFS Features and Background

ZFS is not just a filesystem in the traditional sense, though you can use it that way. It is an integrated storage stack, which can completely replace the need for LVM, md-raid, and even hardware RAID controllers. This permits quite a bit of flexibility and optimization not present when building a stack involving those components. For instance, if a drive in a RAID fails, it needs only rebuild the parts that have actual data stored on them.

Let’s look at some of the features of ZFS:

  • Full checksumming of all data and metadata, providing protection against silent data corruption. The only other Linux filesystem to offer this is btrfs.
  • ZFS is a transactional filesystem that ensures consistent data and metadata.
  • ZFS is copy-on-write, with snapshots that are cheap to create and impose virtually undetectable performance hits. Compare to LVM snapshots, which make writes notoriously slow and require an fsck and mount to get to a readable point.
  • ZFS supports easy rollback to previous snapshots.
  • ZFS send/receive can perform incremental backups much faster than rsync, particularly on systems with many unmodified files. Since it works from snapshots, it guarantees a consistent point-in-time image as well.
  • Snapshots can be turned into writeable “clones”, which simply use copy-on-write semantics. It’s like a cp -r that completes almost instantly and takes no space until you change it.
  • The datasets (“filesystems” or “logical volumes” in LVM terms) in a zpool (“volume group”, to use LVM terms) can shrink or grow dynamically. They can have individual maximum and minimum sizes set, but unlike LVM, where if, say, /usr gets bigger than you thought, you have to manually allocate more space to it, ZFS datasets can use any space available in the pool.
  • ZFS is designed to run well in big iron, and scales to massive amounts of storage. It supports SSDs as L2 cache and ZIL (intent log) devices.
  • ZFS has some built-in compression methods that are quite CPU-efficient and can yield not just space but performance benefits in almost all cases involving compressible data.
  • ZFS pools can host zvols, a block device under /dev that stores its data in the zpool. zvols support TRIM/DISCARD, so are ideal for storing VM images, as they can instantly release space released by the guest OS. They can also be snapshotted and backed up like the rest of ZFS.

Although it is often considered a server filesystem, ZFS has been used in plenty of other situations for some time now, with ports to FreeBSD, Linux, and MacOS. I find it particularly useful:

  • To have faith that my photos, backups, and paperwork archives are intact. zpool scrub at any time will read the entire dataset and verify the integrity of every bit.
  • I can create snapshots of my system before running apt-get dist-upgrade, making it easy to track down issues or roll back to a known-good configuration. Ideal for people tracking sid or testing. One can also easily simply boot from a previous snapshot.
  • Many scripts exist that make frequent snapshots, and retain the for a period of time as a way of protecting work in progress against an accidental rm. There is no reason not to snapshot /home every 5 minutes, for instance. It’s almost as good as storing / in git.

The added level of security in having cheap snapshots available is almost worth it by itself.

ZFS drawbacks

Compared to other Linux filesystems, there are a few drawbacks of ZFS:

  • CDDL will prevent it from ever being part of the Linus kernel tree
  • It is more RAM-hungry than most, although with tuning it can even run on the Raspberry Pi.
  • A 64-bit kernel is strongly preferred, even in low-memory situations.
  • Performance on many small files may be less than ext4
  • The ZFS cache does not shrink and expand in response to changing RAM usage conditions on the system as well as the normal Linux cache does.
  • Compared to btrfs, ZFS lacks some features of btrfs, such as being able to shrink an existing pool or easily change storage allocation on the fly. On the other hand, the features in ZFS have never caused me a kernel panic, and half the things I liked about btrfs seem to have.
  • ZFS is already quite stable on Linux. However, the GRUB, init, and initramfs code supporting booting from a ZFS root and /boot is less stable. If you want to go 100% ZFS, be prepared to tweak your system to get it to boot properly. Once done, however, it is quite stable.

Converting to ZFS

I have written up an extensive HOWTO on converting an existing system to use ZFS. It covers workarounds for all the boot-time bugs I have encountered as well as documenting all steps needed to make it happen. It works quite well.

Additional Hints

If setting up zvols to be used by VirtualBox or some such system, you might be interested in managing zvol ownership and permissions with udev.

John's BlogDebian-Live Rescue image with ZFS On Linux; Ditched btrfs

I’m a geek. I enjoy playing with different filesystems, version control systems, and, well, for that matter, radios.

I have lately started to worry about the risks of silent data corruption, and as such, looked to switch my personal systems to either ZFS or btrfs, both of which offer built-in checksumming of all data and metadata. I initially opted for btrfs, because of its tighter integration into the Linux kernel and ability to shrink an existing btrfs filesystem.

However, as I wrote last month, that experiment was not a success. I had too many serious performance regressions and one too many kernel panics and decided it wasn’t worth it. And that the SuSE people got it wrong, deeply wrong, when they declared btrfs ready for production. I never lost any data, to its credit. But it simply reduces uptime too much.

That left ZFS. Before I build a system, I always want to make sure I can repair it. So I started with the Debian Live rescue image, and added the zfsonlinux.org repository to it, along with some key packages to enable the ZFS kernel modules, GRUB support, and initramfs support. The resulting image is described, and can be downloaded from, my ZFS Rescue Disc wiki page, which also has a link to my source tree on github.

In future blog posts in the series, I will describe the process of converting existing Debian installations to use ZFS, of getting them to boot from ZFS, some bugs I encountered along the way, and some surprising performance regressions in ZFS compared to ext4 and btrfs.

John's BlogMarried!

One week before the wedding, to Laura: “Mono won’t just clear up right away.”

One week before the wedding, to me: “That’s going to need stitches.”

Yes, not long before the wedding, Laura had come down with mononucleosis and I had cut into my finger with a very sharp knife while cutting bread requiring a trip to the emergency room to get stitches. Two days before we got married, instead of moving furniture, I was getting stitches out of my finger.

It wasn’t the kind of week we had planned.

But it was the happiest, most amazing occasion I could have ever imagined.

As I wrote last month, I am richly blessed indeed.

Our wedding was three days after Christmas. The church was still decorated for Christmas, with the tree in on corner, glittering stars suspended in mid-air on cables from the walls, wreaths and candles in the windows, and it was a joy-filled day.

Before the ceremony, we took pictures — the only part of the day Jacob and Oliver weren’t thrilled with. Nevertheless, we got some fun ones.

Laura and I seem to know quite a few pastors between us – and not just because Laura is a pastor. My brother officiated with the wedding vows, his wife with scripture and a prayer of blessing, and the church’s pastor gave the message.

Laura and I wrote in our wedding program, “Music has long been a thread running through both our lives. We have enjoyed singing together, playing piano and pennywhistle duets, attending concerts, and even exploring old hymnals. Music is also one of the best ways to have a conversation – even a conversation with God.” We wrote a page in the program about each of the hymns that were a part of the wedding, and why we picked them. The combined church choirs of my home church and Laura’s church sang John Rutter’s beautiful arrangement of For the Beauty of the Earth (click here to listen to a different choir). Hearing “For the beauty of each hour”, “For the joy of human love”, and “Lord of all, to thee we raise this our joyful hymn of praise” was perfect for the day.

It was with such great happiness that we walked out of the sanctuary, a married couple, to the sound of the congregation singing Joy to the World!

Jacob and Oliver were so very excited on our wedding day. They happily explored the church while waiting for things to happen. We had them help us light our unity candle, and they were pleased with that. Jacob loved his suit, which made him look just like me. And they were, of course, delighted with the cake and in the middle of it all.

For our honeymoon, we managed to get two weeks of vacation, and spent about half of it at home. We had looked at various options for retreats in the country, but eventually concluded that our house is a retreat in the country, so might as well enjoy it at home.

We also went to the Palo Duro Canyon area near Amarillo in the Texas panhandle, staying in a small B&B in Canyon, TX. Palo Duro is the second-largest canyon in North America, and quite colorful year-round. What a beautiful place to go for our honeymoon! By the time we got there, Laura was getting past mono, and we went for hikes in the canyon on two different days — hiking a total of 10 miles, including a hike up the side of the canyon.

After we got back home, on the last weekday of our honeymoon, we went back to the Flint Hills of Kansas, to some of the same places we had spent our third date. We climbed the windy staircase at the Chase County Courthouse, the oldest courthouse still in use in Kansas.

And peered out its famous oval window.

We found the last remnant of the old ghost town of Elk, ate at the same restaurant we had that day. It brought back wonderful memories, and it was a good day in itself. Because even though it was a gold, drizzly, overcast day in January, this time, we were married.

And this year, Thanksgiving is all year.

Picasa Photos2014-01-21 honeymoon public

2014-01-21 honeymoon public

Date: Jan 21, 2014
Number of Photos in Album: 8

View Album

Picasa Photos2013-12 wedding highlights

2013-12 wedding highlights

Date: Jan 21, 2014
Number of Photos in Album: 11

View Album

John's BlogRichly Blessed

“It’s wedding week! Wedding week! Wedding week! Wedding week! Oh, also Christmas. Oh dad, it’s wedding week! I can’t believe it! It’s finally here! Wedding week!” – Jacob, age 7, Sunday

“Oh dad, this is the best Christmas EVER!” – Jacob, Wednesday

“Dad, is the wedding TODAY?” – Oliver, age 4, every morning this week

This has certainly been a Christmas like no other. I have never known something to upstage Christmas for Jacob, but apparently a wedding can!

Laura and I got to celebrate our first Christmas together this year — together, of course, with the boys. We enjoyed a wonderful day in the middle of a busy week, filled with play, family togetherness, warmth, and happiness. At one point, while I was helping the boys with their new model train components, Laura was enjoying playing Christmas tunes on the piano. Every time she’d reach the end, Jacob paused, and said, “That was awesome!”, beating me to it.

That’s a few days before Christmas — Jacob and Oliver demanding snow ice cream, and of course who am I to refuse?

Cousins opening presents

After his school Christmas program, Jacob has enjoyed singing. Here he is after the Christmas Eve program, where he excitedly ran up into the choir loft, picked up a hymnal, and pretended to sing.

And, of course, opening of presents at home.

Sometimes I think about how I didn’t know life could get this good. Soon Laura and I will be married, and it will be even better. Truly we have been richly blessed.

Picasa Photos2013-12

2013-12

Date: Dec 26, 2013
Number of Photos in Album: 7

View Album

John's BlogDelicious Holiday Recipes

I’ve come up with some new favorites this season. The boys and Laura were around for all three, and I am happy to report there were many kitchen smiles over these!

From-Scratch Hot Chocolate

There’s something about hot chocolate made from scratch, with chocolate melted into milk, instead of a powder stirred in. It takes quite a bit more time, and probably has more calories, but it is quite delicious.

The key to a delicious result where milk is concerned is to take things slow and keep stirring. You don’t want the chocolate to scorch at the bottom of the pan. Heating up the milk before the chocolate should help things mix in more easily as well.

  • Begin with 3 cups milk and 1 cup heavy whipping cream. Heat slowly over moderate to low heat, stirring periodically. Once you see bubbles start to form around the edges, it is plenty hot (or even a bit more hot than it needs to be).
  • Add one cup of semisweet chocolate chips, 1 teaspoon sugar, and 1/2 tsp vanilla extract.
  • Stir constantly until all the chocolate is melted and well mixed. There will still be some small bits of chocolate within, but if it is all done slowly like this, the chocolate should be pretty well melted.

The basis for this recipe was here, and it called for 2 cups milk and 2 cups half-and-half. I trust my heavy whipping cream was fine! <grin> There are also some other variations on that site.

This nearly made my little cast iron kettle overflow, so next time I made a 3/4 recipe.

Hot Spiced Cider

We put up a Christmas tree yesterday, so I thought hot spiced cider would be perfect for the occasion. I went searching for recipes, and many of them called for cloves (which have to be sifted out later or put in a spice bag). I wasn’t going to have time to delay two boys from setting up a Christmas tree long enough for that, so I found this basic recipe to work well. However, I, as usual, made some modifications ;-)

  • Warm 4 cups apple cider (not juice, as the recipe suggests) in a pot.
  • Add 1/2 tsp cinnamon
  • Add 1/4 tsp nutmeg or allspice (I used allspice because I was mysteriously out of nutmeg, but will probably use nutmeg next time)
  • Add 1 tbsp brown sugar
  • Stir constantly until sufficiently dissolved. Pour immediately before drinking, as the contents will tend to separate.

Mmmmm…. yum….

Turkey or Chicken Noodle Soup

The annual “what to do with all that leftover turkey” quest strikes again. I like chicken noodle soup, so why not a turkey noodle soup done the same way?

Here’s what I used, roughly, in my large 6-quart cast iron cooking pot (aka “Dutch oven”):

  • 9 cups chicken/turkey broth. Your own if you have it, or the canned variety works. Or make your own with boullion if you have it.
  • 2 chopped yellow onions. (I added half a chopped red onion as well because I had it sitting around. Nobody complained, but 2.5 onions was a little much.)
  • 4 tsp fresh basil or 1 tsp dried basil
  • 4 tsp fresh oregano or 1 tsp dried oregano
  • 1/2 tsp pepper
  • 1/2 tsp salt
  • 1 tsp beef bouillon
  • 2 bay leaves
  • 20 oz frozen mixed vegetables (I’d probably add more than that next year; this wasn’t quite enough)
  • Plenty of wide egg noodles. The recipe I used called for 1 cup, which was laughably inadequate. I just dumped until it looked right, and then the package was almost empty so I dumped the rest in too.
  • 4 cups cooked turkey or chicken, cubed (a kitchen scissors makes quick work of that)
  • Two 14.5-oz cans diced tomatoes (do not drain)

Start with the broth, onion, basil, oregano, pepper, and bay leaf. Heat up the mixture and add the vegetables. Bring it to boiling, then add the uncooked noodles. Return to boiling, then reduce heat, cover, and simmer for 8 minutes. Add the turkey or chicken and diced tomatoes, and simmer until hot enough to serve.

The nice thing about soups is that they freeze well and make great winter leftovers. This recipe makes quite a lot of soup; you may wish to halve it.

This recipe was adapted from one in a Better Homes & Gardens cookbook.

John's BlogResults with btrfs and zfs

The recent news that openSUSE considers btrfs safe for users prompted me to consider using it. And indeed I did. I was already familiar with zfs, so considered this a good opportunity to experiment with btrfs.

btrfs makes an intriguing filesystem for all sorts of workloads. The benefits of btrfs and zfs are well-documented elsewhere. There are a number of features btrfs has that zfs lacks. For instance:

  • The ability to shrink a device that’s a member of a filesystem/pool
  • The ability to remove a device from a filesystem/pool entirely, assuming enough free space exists elsewhere for its data to be moved over.
  • Asynchronous deduplication that imposes neither a synchronous performance hit nor a heavy RAM burden
  • Copy-on-write copies down to the individual file level with cp --reflink
  • Live conversion of data between different profiles (single, dup, RAID0, RAID1, etc)
  • Live conversion between on-the-fly compression methods, including none at all
  • Numerous SSD optimizations, including alignment and both synchronous and asynchronous TRIM options
  • Proper integration with the VM subsystem
  • Proper support across the many Linux architectures, including 32-bit ones (zfs is currently only flagged stable on amd64)
  • Does not require excessive amounts of RAM

The feature set of ZFS that btrfs lacks is well-documented elsewhere, but there are a few odd btrfs missteps:

  • There is no way to see how much space subvolume/filesystem is using without turning on quotas. Even then, it is cumbersome and not reported with df like it should be.
  • When a maxmium size for a subvolume is set via a quota, it is not reported via df; applications have no idea when they are about to hit the maximum size of a filesystem.

btrfs would be fine if it worked reliably. I should say at the outset that I have never lost any data due to it, but it has caused enough kernel panics that I’ve lost count. I several times had a file that produced a panic when I tried to delete it, several times when it took more than 12 hours to unmount a btrfs filesystem, behaviors where hardlink-heavy workloads take days longer to complete than on zfs or ext4, and that’s just the ones I wrote about. I tried to use btrfs balance to change the metadata allocation on the filesystem, and never did get it to complete; it seemed to go into an endless I/O pattern after the first 1GB of metadata and never got past that. I didn’t bother trying the live migration of data from one disk to another on this filesystem.

I wanted btrfs to work. I really, really did. But I just can’t see it working. I tried it on my laptop, but had to turn of CoW on my virtual machine’s disk because of the rm bug. I tried it on my backup devices, but it was unusable there due to being so slow. (Also, the hardlink behavior is broken by default and requires btrfstune -r. Yipe.)

At this point, I don’t think it is really all that worth bothering with. I think the SuSE decision is misguided and ill-informed. btrfs will be an awesome filesystem. I am quite sure it will, and will in time probably displace zfs as the most advanced filesystem out there. But that time is not yet here.

In the meantime, I’m going to build a Debian Live Rescue CD with zfsonlinux on it. Because I don’t ever set up a system I can’t repair.

John's BlogWhy are we still backing up to hardlink farms?

A person can find all sorts of implementations of backups using hardlink trees to save space for incrementals. Some of them are fairly rudimentary, using rsync --link-dest. Others, like BackupPC, are more sophisticated, doing file-level dedup to a storage pool indexed by a hash.

While these are fairly space-efficient, they are really inefficient in other ways, because they create tons of directory entries. It would not be surprising to find millions of directory entries consumed very quickly. And while any given backup set can be deleted without impact on the others, the act of doing so can be very time-intensive, since often a full directory tree is populated with every day’s backup.

Much better is possible on modern filesystems. ZFS has been around for quite awhile now, and is stable on Solaris, FreeBSD and derivatives, and Linux. btrfs is also being used for real workloads and is considered stable on Linux.

Both have cheap copy-on-write snapshot operations that would work well with a simple rsync --inplace to achieve the same effect ad hardlink farms, but without all the performance penalties. When creating and destroying snapshots is a virtually instantaneous operation, and the snapshots work at a file block level instead of an entire file level, and preserve changing permissions and such as well (which rsync --link-dest can have issues with), why are we not using it more?

BackupPC has a very nice scheduler, a helpful web interface, and a backend that doesn’t have a mode to take advantage of these more modern filesystems. The only tool I see like this is dirvish, which someone made patches for btrfs snapshots three years ago that never, as far as I can tell, got integrated.

A lot of folks are rolling a homegrown solution involving rsync and snapshots. Some are using zfs send / btrfs send, but those mechanisms require the same kind of FS on the machine being backed up as on the destination, and do not permit excluding files from the backup set.

Is this an area that needs work, or am I overlooking something?

Incidentally, hats off to liw’s obnam. It doesn’t exactly do this, but sort of implements its own filesystem with CoW semantics.

John's BlogTwo Kittens

Almost every time he got off the bus for the past month and a half, Jacob started his afternoon in the same way. Before toys, before his trains and his toy bus, before anything indoors, he went for our cats. Here he is, cradling his favorite, Tigger:

Laura and I both grew up around cats. We had been talking about kittens, and shortly after we got engaged, one of my relatives offered us some free kittens. We went to his place one evening and selected two of them – one calico and one tiger-colored. Since what is now my place will soon be our place, they came to live with me. Our cats were one of the first things we did to prepare for our lives together.

Oliver wanted to name them some rather impractical sentence-long names (“The Cat Who Always Likes To Run”), so Laura and I suggested some names from one of their favorite books: Tigger and Roo. They both liked the names, but Oliver thought they should be called “Tigger the Digger” and “Roo the Runner”. Never mind that they were just 6 weeks old at the time, and not really old enough to either dig or run. Here’s Oliver with Roo, the day after the kittens arrived here.

I have always had outside cats, both because I’m allergic to cats so I need them to be outside, and because they sometimes literally quiver with joy of being outdoors. Tigger and Roo often chased insects, wrestled with each other, ran up (and slowly came back down) trees, and just loved the outside. Sometimes, I have taken my laptop and wireless headset and work from the back porch. The kittens climb up my jeans, inspect the laptop, and once Roo even fell asleep on my lap at one of those times.

Jacob has been particularly attached to Tigger, calling him “my very best friend.” When Jacob picks him up after school, Tigger often purrs while cradled in Jacob’s arms, and Jacob comments that “Tigger loves me. Oh dad, he knows I am his friend!”

The kittens have been growing, and becoming more and more comfortable with their home in the country. Whenever I go outside, it isn’t long before there are two energetic kittens near my feet, running back and forth, sometimes being very difficult to avoid stepping on. I call and I see little heads looking at me, from up in a tree, or peeking out from the grain elevator door, or from under the grill. They stare for just a second, and then start running, sometimes comically crashing into something in their haste.

Yesterday when I went to give them food, I called and no cats came. I was concerned, and walked around the yard, but at some point either they come or they don’t.

Yesterday afternoon, just after the bus dropped off Jacob, I discovered Tigger on the ground, motionless. Once Jacob was in the house, I went to investigate, and found Tigger was dead. As I was moving his body, I saw Roo was dead, too. Both apparently from some sort of sudden physical injury — a bit mysterious, because neither of them were at a place where they had ever gone before. While all this was happening, I had to also think about how I was going to tell the boys about this.

I tried to minimize what he could see, Jacob had caught an unavoidable glimpse of Tigger as we were walking back from the bus, but didn’t know exactly what had happened. He waited in the house, and when I came back, asked me if Tigger was dead. I said he was. Jacob started crying, saying, “Oh Dad, I am so sad”, and reached up for a hug. I picked him up and held him, then sat down on the couch and let him curl up on my lap. I could quite honestly let him know he wasn’t alone, telling him I am sad, too.

Oliver arrived not long after, and he too was sad, though not as much as Jacob. Both boys pretty soon wanted to see them. I decided this was important for them for closure, and to understand, so while they waited in the house, I went back out to arrange the kittens to hide their faces, the part that looks most unnatural after they die. The boys and I walked out to where I put them, then I carried both of them the last few feet. We stood a little ways back — close enough to see who was there, far enough to not get too much detail — and they were both sniffling. I tried to put voice to the occasion, saying, “Goodbye, Tigger and Roo. We love you.” Oliver asked if they could hear us. I said “No, but I told them what I felt like anyway.” Jacob, through tears, said, “Dad, maybe they are in heaven now.”

We went back inside. Jacob said, “Oh dad, I am so sad. This is the saddest day of my life. My heart is breaking.” Hearing a 7-year-old say that isn’t exactly easy for a dad. Pretty soon he was thinking of sort of comfort activities to do, saying “I think I would feel better if we did…” So they decided to watch a favorite TV program. Jacob asked if Laura knew yet, and when I said no, he got his take-charge voice and said, “Dad, you will start the TV show for us. While we are watching, you will send Laura an email to tell her about Tigger and Roo. OK?” What could I say, it wasn’t a bad idea.

Pretty soon both boys were talking and laughing. It was Big Truck Night last night, at a town about half an hour away. It’s an annual event we were already planning to attend, where all sorts of Big Trucks – firetrucks, school bus, combine, bucket truck, cement truck, etc – show up and are open for kids to climb in and explore. It’s always a highlight for them. They played and sang happily as we drove, excitedly opened and closed the big door on the school bus and yelled “All Aboard!” from the top of the combine. We ate dinner, and drove back home. When we got home, Jacob mentioned the cats again, in a sort of matter-of-fact way, and also wanted to make sure he knew Laura had got the message.

A person never wakes up expecting to have to dump a bowl of un-eaten cat food, or to give an impromptu cat funeral for little boys. As it was happening, I wished they hadn’t been around right then. But in retrospect, I am glad they were. They had been part of life for those kittens, and it is only right that they could be included in being part of death. They got visual closure this way, and will never wonder if the cats are coming back someday. They had a chance to say goodbye.

Here is how I remember the kittens.

Picasa PhotosTigger & Roo

Tigger & Roo

Date: Oct 8, 2013
Number of Photos in Album: 26

View Album

Picasa Photos2013-09-28 jacob pumpkin patch wichita selections

2013-09-28 jacob pumpkin patch wichita selections

Date: Oct 1, 2013
Number of Photos in Album: 30

View Album

John's BlogEarthrise

Today I link you to a video narrated by the legendary Carl Sagan – The Frontier is Everywhere.

Partial quotes:

We were hunters and foragers. The frontier was everywhere. We were bounded only by the earth, and the ocean, and the sky. The open road still softly calls. Our little terraqueous globe as the madhouse of those hundred thousand millions of worlds. We, who cannot even put our own planetary home in order, riven with rivalries and hatreds; are we to venture out into space?

By the time we are ready to settle even the nearest other planetary systems, we will have changed… For all our failings, despite our limitations and fallibilities, we humans are capable of greatness… How far will our nomadic species have wandered, by the end of the next century, and the next millennium?

Our remote descendants, safely arrayed on many worlds through the solar system, and beyond, will be unified, by their common heritage, by their regard for their home planet, and by the knowledge that, whatever other life may be, the only humans in all the universe, come from Earth.

They will gaze up and strain to find the blue dot in their skies. They will marvel at how vulnerable the repository of all our potential once was, how perilous our infancy, how humble our beginnings, how many rivers we had to cross, before we found our way.

Can you imagine seeing Earthrise?

Earthrise, 1968 December 24

Incredible.

John's BlogEngaged!

Today I have the delightful chance to write about a deep and wonderful joy.

Yes, Laura and I are engaged!

There are no words adequate for this kind of occasion, but there is a picture that gets close:

I never imagined a person could find a friend so wonderful, a person that enjoys so much in common, someone that understands me and that I understand so well. And yet, here I am, engaged to that friend. “Amazing” only begins to describe the feeling.

One of the first things Laura and I talked about was a hymn I was tinkering with, typesetting with GNU LilyPond. That hymn ends like this:

“Since Love is Lord of heav’n and earth, how can I keep from singing?”

It is a wonderful thought, and very true. Even literally true; I often find myself singing, humming, or playing the penny whistle during my day.

One of my good friends once told me, “I am completely sure that your happiest days lie ahead of you.” And he was right. I have already experienced them. This is a wonderful time, and every day brings plentiful reasons to be thankful. And as Laura and I prepare for a life together, I know that it is true not only that my friend was right, but that he is right — my happiest days are yet to come, and our happiest days are yet to come, too.

I have been blessed in many ways, and feel like the luckiest man alive.

To be loved, and to love, a person so wonderful is truly a remarkable gift.

Picasa Photos2013 with Laura

2013 with Laura

Date: Aug 25, 2013
Number of Photos in Album: 2

View Album

Picasa Photos2013-08 kittens

2013-08 kittens

Date: Aug 24, 2013
Number of Photos in Album: 5

View Album

John's BlogA 4-year-old’s sudden interest in German monorail

Today, Oliver and Jacob heard about monorails. I showed them a wikipedia article about monorails, and it had a picture of a monorail in Germany.

Oliver, age 4: I have been on that monorail!

Me: Sorry, Oliver, but that monorail is in Germany. You have never been to Germany.

Oliver: But I HAVE been to Germany!

Me: No, you have never been to Germany.

Oliver: Dad, I HAVE been to Germany. I love Germany!

Jacob, taking an interest: Dad, can we go to Germany sometime?

Me: Yes, we probably could sometime.

Jacob: Great! Then we could ride a Deutsche Bahn train!

At this point, we had a brief discussion about the fact that we can’t take a train from the United States to Germany, but we can fly there and then take trains.

Oliver: Dad, we should go the the airport and tell them to take us to Germany right now!

John's BlogDevelopment Freedom in New Platforms

I started writing code when I was somewhere around 1st grade, hacking some rudimentary BASIC on a TRS-80. Since then, I’ve used more programming languages than I can count; everything from C to Prolog, Python to Haskell, and probably a dozen others. I enjoy diversity in programming languages and tools.

I’ve had an interest in both web and mobile development for some time, but have done little with either. Reflecting on that lately, I realize that both platforms severely restrict my freedom to use the tools I choose. Even on an old x86 with 640K of RAM, I had choices; BASIC, Pascal, and C would work under DOS. (Plus assembler, but I had yet to learn that at the time.) On Android, my choices are Java or… Java. (OK, or something that compiles to Java, but those selections are still limited.) Plus it seems like I’d have to use Eclipse, which seems to have taken the kitchen sync crown from Emacs long ago, and not in a good way. On iOS, the choices are Objective C. And on the web, it’s JavaScript.

The problem with all this is that there’s little room for language innovation on those platforms. If predictions about the decline of development on the PC pan out, how will the next advance in programming languages gain traction if the hot spots for development are locked down to three main languages? Will we find ourselves only willing to consider languages that can compile to Java/Dalvik bytecode? And won’t that put limits on the possible space for innovation to occur within?

Yes, I know about things like Mono and the Android Scripting Environment, which offer some potential for hope, but I think it’s still safe to say that these platforms are pretty closed to development in “unofficial” languages. I don’t know of major apps in the Android market written in Python, for instance.

I know of efforts such as Ubuntu’s, but even they simply lock down development to a different toolkit. There is no choice there of GTK vs. Qt, or such things.

Sadly, I don’t think it has to be this way. My phone has way more power than my computer did just a few years ago, and that computer was capable of not just executing, but even compiling, code written in all sorts of different languages.

But I don’t know if there’s much we can do to change things. Even with native code on Android, plenty still has to go through Dalvik (at least that’s my understanding). If you wanted to write a Python app with a full graphical touch interface on Android, I don’t think this would be all that easy of a task.

John's BlogSpring Always Comes

“There will come a time when you believe everything is finished. That will be the beginning.”

– Louis L’Amour

Last year, I wrote about the difficult times in my life and hope for future, but never really explained why. I have written little since, because there is little I can write without a bit of explanation. It is partly because of the complexity of the task of telling my story without telling too much of others’ stories.

But it is important I tell this story. So often on the Internet, we hear only the brave face, the positive things that happen. This story involves tears and difficulty. And also we often see only anger and bitterness. This story involves joy and celebration.

One afternoon last year, I was working as usual (I work from home) when two sheriff deputies arrived on my doorstep. They gave me paperwork showing that my now ex-wife had filed for divorce, had asked them to serve me the paperwork, and that she had been given temporary possession of the house. I had 2 hours to gather up clothes and a little computer equipment (there was a list) and leave the house.

Thus began the most difficult time in my life. I went from reading a bedtime story and singing a bedtime song to my young boys every night to seeing them only a little, from living in the house my grandparents and dad had lived in to having no particular plan for where I’d sleep that night, from thinking I had a good idea of what the future held to not knowing when, if ever, I’d ever be back home. I worried about how the boys would fare (they have done well so far). And it was incomprehensible; I couldn’t find answers to “why?”.

In the time since, the divorce became final, I did return home, the boys spend more time with me, and a new normal emerged.

At the time, it seemed like a sudden, deep winter blizzard. I couldn’t see very far down the road, spring seemed far off, and I couldn’t see very well either forward or backward.

But I was determined to find positives in the situation. It started almost immediately; I had never been a person to talk about pain, but just a few hours after the divorce was filed, I knew I needed to talk to someone about it, and did. A week later, I shared about it in church. Amazing friends, locally and all over the world, provided support and encouragement. I had less total time with the boys than before the divorce, but more time with just the three of us, and we used it to play together at home, spend days in town, and even take a train trip to Santa Fe, where none of us had ever been before.

I realized how much I could forgive, and that my ex-wife probably did the best she could with the persistent legacy of difficult life events that happened to her long before she met me. I understood this, and was never angry, just sad, for everyone. I have always known nobody is perfect, myself included, but can be hard on myself when I’m less than perfect. I forgave myself, too, realizing that I did my best to help in the most unfamiliar of waters, and although I sometimes didn’t get it right, my conscience is clear because my heart was in the right place and I tried, very hard.

Most incredibly, I became a person with a deep sense of inner peace. I always tried to work hard to set life on a good path; I got good grades in school, am a good employee, and have a strong set of values. But where courts are involved, there’s a strong sense of powerlessness. At times, there was nothing I could do to make life better for my boys or for me. I finally had to let go of taking on responsibility for all that on my own shoulders. I simply knew that things would be OK, and in fact were OK, and that there is nothing in life that really deserves worries. That doesn’t mean worries are never present, but that mostly they are subdued, like a radio quietly playing in another room. When they aren’t, I can sit down at the piano, play my penny whistle, sing, walk to my creek, talk to friends, or any of so many things that let them melt away. I stopped searching for happiness and peace, and let those things find me.

In religious terms, my faith became not just an intellectual one, but also a spiritual one. An atheist friend asked me, “Just what does religion mean to you anyway?” My answer: “The certainty that spring always comes, for everyone that understands this.”

“People speak of misfortunes and sufferings,” remarked Pierre, “but if at this moment I were asked: ‘Would you rather be what you were before you were taken prisoner, or go through all this again?’ then for heaven’s sake let me again have captivity and horseflesh! We imagine that when we are thrown out of our usual ruts all is lost, but it is only then that what is new and good begins. While there is life there is happiness. There is much, much before us.”

– Tolstoy, War and Peace

By Thanksgiving, I had much to be thankful for. Some of it under my nose waiting to be rediscovered after years of distraction, such as the glorious Kansas sunrises. Some things were new, such as roasting a turkey all by myself (or, rather, with Jacob and Oliver) for the first time, and having it come out absolutely perfectly. And some were just the things of everyday life: that I lived in my own house again, that I could walk out to the creek at the edge of my property whenever I wanted, that I could play piano, that the sounds of laughter and little running feet again could often be heard on my wood floors.

That’s not to say everything was easy; the courts sometimes made decisions sometimes I didn’t think were in the boys’ best interests, legal things dragged on and on, but in the end, peace endured. Happiness endured. I found myself thinking at Thanksgiving that it was the best Thanksgiving ever. Not much later, I considered myself happier than I’ve ever been. I was focusing on the daily gifts of life, marveling at the sunrises, looking forward to life’s next adventure, confident that it would be far better than the last.

And then, to my complete surprise, I found myself in a relationship again. On top of all the wonderful things happening in my life, I met Laura. I never could have imagined a friend so wonderful, a relationship so loving and joyful, something to treasure so deeply. I can look back at events and shake my head in amazement and wonder, that I found myself happier than I’d ever been, and then this wonderful relationship on top of that. I have been blessed to have the life I do, and feel almost embarrassingly lucky.

I share this story because friends that had been through divorce years ago shared their stories. They gave me hope. And if I didn’t share this story in this public way, I would be squandering an opportunity to find more positives from what happened. I hope that this helps, somehow, someone that is in pain know that there is beauty in the valley, and spring always comes, every single year.

And I share it because happiness like this can’t be repressed for long. Tolstoy was right. While there is life, there is happiness.

I’ll end with a story from last Thanksgiving. It was 6:10AM that weekend. I was still asleep, and heard this:

Jacob, yelling from his room: “Dad? Dad!”

Me, groggy: “Yes, Jacob?”

Jacob: “Can I go down and look at the Christmas tree?”

Me: “Sure, and you can turn it on too.”

Jacob: “Great!”

At that point, I could have gone back to sleep. I was really short on sleep that morning, and Jacob would have been fine. But I gave him a couple of minutes, then I went downstairs too. He was curled up on the piano bench, looking at the tree. I quietly turned up the downstairs thermostat, got a chair, put it next to the piano bench, and sat down by the tree too. Jacob crawled over onto my lap and snuggled up for awhile. Neither of us said anything. Then:

“Dad?”

“Yes?”

“This is the best Thanksgiving ever.” And he gave me a big hug.

And he was right. Yes, he was RIGHT!

“That will be the beginning.” Spring comes!

John's BlogSometimes, it’s best not to know

Jacob, Oliver, and I were driving back home, about an 8-hour drive. At one point I heard Jacob say from the back seat, “Oliver, you can’t have my book. It has a bookmark in it!”

Aside from the apparent anti-sharing properties of bookmarks, I sort of smiled at Jacob inventing a bookmark.

An hour later, I heard Oliver saying, “Jacob, you can’t have MY book. It has a bookmark too!”

And again I thought that sounded rather cute. The thought of traveling boys — and they did travel very well — bookmarking the books I brought for them was a nice one.

And then I realized: I hadn’t brought bookmarks. Or, as far as I could recall, bookmark-making material.

I really wanted to say, “Uhm, Jacob, just what did you make that bookmark from?” But I didn’t. I figured everyone would be happier not having to deal with that question.

Picasa Photos2013-06-27 orange sunset

2013-06-27 orange sunset

Date: Jun 27, 2013
Number of Photos in Album: 3

View Album

Picasa Photos2013-06-27

2013-06-27

Date: Jun 27, 2013
Number of Photos in Album: 1

View Album

John Twitter

jgoerzen: @llaisdy I seem to detect a certain common thread ;-)

John Twitterjgoerzen: Took the boys to town today to buy paper &amp; ribbon for their adding machines. They were super-excited to be able to add in red AND black!

jgoerzen: Took the boys to town today to buy paper & ribbon for their adding machines. They were super-excited to be able to add in red AND black!

John Twitter

jgoerzen: I wrote about two boys, shrimp, and stars today. http://t.co/PSelssfiNJ

John's BlogTwo boys, shrimp, and stars

I recently made a routine analysis of my kitchen. (Of course I make a routine analysis of my kitchen; don’t you?) In it, I discovered these items, still usable, but approaching that magic “throw it out” date:

  • Baby carrots
  • Potatoes
  • Lemons
  • Green beans
  • Corn

So, I thought, what can I make that would use all of these? And I realized I had some shrimp in the freezer, so: a shrimp boil! I tossed it all, plus some various seasonings and a few other veggies, into the Dutch oven, and boiled.

Jacob and Oliver watched the activity with interest. Well, except for the potato-peeling part. For that, they went and played with their toy school buses. But the rest was good. They carefully observed me adding some spices, some vegetables, the shrimp, and watched it all simmer. Then it was time to eat. Excitement!

Of course, it did take a few minutes to boil, so Jacob got down his whiteboard while Oliver looked on.

They enjoyed learning how to peel the shell from the shrimp and devoured their food.

And another night recently, Jacob unexpectedly showed up in the kitchen at 10PM. He said he was thirsty, so I got him some water. He asked, “Dad, did you make ice cream?”

Earlier that day, I had prepared ice cream with Oliver, but it was a kind that had to be cooked (lemon with pureed strawberries and peaches) and it wasn’t cool enough to finish before their bedtime. I did let them help add the ice and salt to the ice cream freezer just before they went to bed.

So I told Jacob that yes, the ice cream was done.

He stood there, tiredly, considering, with this “oh he’ll never say yes to ice cream at 10PM” look on his face.

So I said, “would you like one bite right now?”

The look of delight on his face was amazing; a broad smile, a twinkle in his eyes, and a clap. So I got out the big bowl of ice cream and scooped up one big spoonful. He loved it. Then I said, “should we go look at the stars?”

I carried Jacob outside to the porch. We stood there, looking up. I used to do this with him periodically, but it had been about a year. So he was thrilled. It was a partially overcast night, but there were still some stars visible. He had no idea there were some stars missing. To him, it was amazing and wonderful and infinite. “Oh dad, there are way too many stars to count!”

He stayed there, arms around my neck, for a minute or two, then was ready to go back inside. I set him down, gave him a hug, said “Goodnight, Jacob.” And off he trotted, back upstairs, wearing a contented smile, and he fell asleep almost immediately.

All it takes to delight children is a bit a shrimp or some stars. And those things delight me, too.

Picasa Photos2013-05-30 blog

2013-05-30 blog

Date: May 30, 2013
Number of Photos in Album: 16

View Album

John Twitter

jgoerzen: @kayray Not bad at all! https://t.co/97OM4Gzjba

John Twitter

jgoerzen: Listening to Beethoven's 9th today: Freude, schöner Götterfunken Tochter aus Elysium, Wir betreten feuertrunken, Himmlische, dein Heiligtum!

John Twitter

jgoerzen: Clicked to unsubscribe from "all Sonicwall newsletters". Result: "You have succesfully activated your newsletter subscriptions." Sigh.

John Twitter

jgoerzen: 5 spams from Dell SonicWall this morning, each in a different language. "because... you were interested in info from Dell." Not true!

John Twitter

jgoerzen: Preparing a shrimp boil with the boys. They are interested, but more excited about the whipped cream we just made for pumpkin cake.

John Twitter

jgoerzen: @Wilboard I have been pretty busy lately, but I will try!

John Twitter

jgoerzen: Today's spam featured "home contents insurance for women" from a South African company, that is proud they insure handbags.

John Twitter

jgoerzen: "Rain, snow, and sleet before 2pm, then rain and snow likely." - National Weather Svc. 2 days ago we were above 80F/27C. #kansasweather

John Twitter

jgoerzen: "Dad, I am all full of books!" - Oliver

John Twitter

jgoerzen: Dr. Seuss had some remarkable advice sometimes. 30 life lessons from Dr. Seuss: http://t.co/BHefxjFmxb

John Twitter

jgoerzen: Amazement and home on the range: http://t.co/atKH6JzGhR

John's BlogAmazement

One of the benefits of working from home is that I have a great view of Kansas from my desk. While I work, I have seen sunrises, snowfall, birds, rain, ice, and all sorts of wildlife. I heard this verse of Home on the Range the other day, which reminded me of this:

“How often at night when the heavens are bright
With the light from the glittering stars
Have I stood here amazed and asked as I gazed
If their glory exceeds that of ours.”

One time, I asked Jacob if I should wake him up at night to see the stars. He said an excited yes! So when it was dark outside, I woke him up, carried him outside, and held him while he looked up. He said a long, breathy “Wooooow!” Then he went back to bed, curled up with his butterfly, and fell asleep smiling. Every so often, we repeat this little routine.

There are many opportunities in life to just stand somewhere and be amazed. You don’t have to be in Kansas. Children know this. The rest of us just have to notice.

Picasa Photos2013-04-17 amazement

2013-04-17 amazement

Date: Apr 17, 2013
Number of Photos in Album: 1

View Album

Picasa PhotosApril 17, 2013

April 17, 2013

Date: Apr 17, 2013
Number of Photos in Album: 1

View Album

John Twitter

jgoerzen: RT @mcbazon: "The purpose of life: (1) discover the chroot, (2) escape the chroot. (3) realise you're still in a chroot, see (2)" — @chr ...

John Twitter

jgoerzen: Well hello, gust front. Nice to meet you.

John Twitterjgoerzen: Temps today: 2:10 PM 70F. 2:20PM 57F. 4:00PM 47F. Yes, dropped 13 degrees in less than 10 minutes, and 23deg in &lt; 2 hrs #kansasweather

jgoerzen: Temps today: 2:10 PM 70F. 2:20PM 57F. 4:00PM 47F. Yes, dropped 13 degrees in less than 10 minutes, and 23deg in < 2 hrs #kansasweather

John Twitter

jgoerzen: I said just now : "I suppose it's not entirely out of character for me to get a song from a movie that came out in 1930 stuck in my head."

John Twitter

jgoerzen: Made Italian herb bread this morning. House has been smelling good all day. It's finally baking. It better taste as good as it smells.

John Twitter

jgoerzen: Just got an order update email from @mpix saying "Rejoice! The following order has shipped." Well, OK then. I will!

Footnotes