17 August 2014

Developer toolchain, 2014 Edition

I try to pause every so often and record what my toolchain looks like.   Sort of like people posting on Everyday Carry, but for what I use every day in development.

  • Development machine: 13" MacBook Pro Retina, 2.8GHz Core i7, 8GB RAM, 250GB SSD.  I love this machine.  My wife calls it my woobie.  She's not far from right.
  • OS: Mac OS 10.9.4.  Unix when I want it to be, polished Consumer OS when I just don't care.   It's been 3 years since I ran a windows box as a development machine and with virtualization I can't see running windows as a primary OS ever again.
  • Physical Setup: Thunderbolt Gigabit ethernet, Thunderbolt-to-DVI single 23" monitor, Microsoft Natural Ergonomic Keyboard 4000.   I've dabbled in multi-monitors and buckling spring keyboards, but this setup keeps my attention focused and my repetitive strain to a minimum.  I'll eventually wear out this keyboard and probably buy another one just like it.
  • Note taking tool: Evernote, but honestly I'm dissatisfied everywhere I turn.  Evernote gets closest to what I want amid everything else I've tried:  Google Keep, OneNote (which is an abomination on the Mac), paper notebooks, PDAs, Google Drive Documents, emails.  I basically pendulum between over-noting and under-noting everything.  Google Drive wants to be the master, but I resist.  I'm not sure why.
  • Blogging Platform: Blogger, through simple inertia.  Wordpress seems like exchanging one master for another.  I'm taking a hard look at Github:Pages and Jekyll because I like finer-grained creative control and Markdown, but I can't commit to it.
  • Terminal Program: iTerm.   Tabs are nice.  I spend lots of time here these days in irb or grails shell.
  • Command line shell: zsh + oh-my-zsh.   Tabbing command-line completion and aliases for most things.  (Ex: gc == "Git commit"), with a vibrant community to go with it.
  • Package Manager: Homebrew.   I have no idea how I survived on windows without a decent package manager like apt, yum, or brew.  I'd switch away from Windows to *nix or Mac for this alone.
  • Editor: Sublime Text 3.  I'm an old vi guy.  Sublime has let me forget the envy I always had for not learning emacs.  Like homebrew or oh-my-zsh, has a rabid, vibrant community.
  • IDE:  Honestly, none.  I like command line tools and Sublime, but I acknowledge IntelliJ 13 is peerless in the Java/JVM IDE world.  (Sorry, Eclipse...I've been a user since 2.11, but you're just a hot mess.)
  • Virtualization: VirtualBox.  A colleague turned me on to this a few years ago, and paired with Vagrant for setup/teardown, it's great.
  •  Scripting Language: Ruby.    This one feels like the old Simpsons plot "So you finally decided to steal cable tv."  Ruby is NICE.  It's like PERL grown-up, without the awk acne:    The language just falls away, and you're dealing with the problem at hand.
  • Backend framework (tie):  Rails 4.x or Grails 2.x.  Yeah, I'm not taking a position here.  If you're playing with the JVM ecosystem, Grails is the obvious pick.    Rails is (and remains) a breath of fresh air, especially with the tools that go with it:  Bundler, Berkshelf, rbenv.  Moreover, there's a frenetic vibrancy in the Rails community that's infectious.   Maddening--sure--but infectious.
  • Browser: Chrome.  Frau Perry croons, "We fight, we breakup, we kiss, we makeup."  Chrome makes me uneasy the same way IE 5 did back in the day--it's becoming a monoculture, and Google's starting to get evil with it.  Still, it's fast and its fundamental architecture still has Mozilla's Firefox playing catchup.
  • Source Control:  Git, specifically Github and Github:Enterprise.  Pull Requests and social coding have changed how developers work in the last 5 years.
  • Mobile: None.  I was a phandroid, but I'm still on the smartphone wagon for 1 year and 6 days now.  I'm not first in line for a mobile development position, but there's enough left in server and web development that I think I'll be okay.
  • Social Media:  Twitter for short-form and Facebook for long-form.   Facebook's just inescapable if I want to interact with family, friends, and church.  They seem intent on running everyone away, but the simple network effect keeps everyone there.  I much prefer twitter, but understand not everyone can stand it, particularly since they went public and started screwing with everyone's   Google+ is dead, and has been since 3 months after it launched.  Google should merge it with Youtube (the perfectly good social network they already had) and be done with it.
  • Miscellaneous: Vagrant, Chef, gvm, Github Pages, Jekyll, Markdown, npm, 
/Stream-of-consciousness

An Afternoon with the Fleet

There aren't really pics to go with this.  Sorry about that.

It began simply enough:  A dog days Saturday that promised mild temps and good weather.  Car parts in two separate boxes in my garage.  "I'll probably have this done before you guys get back from the hair place, " I said.

I set myself two tasks:  Kill the Check Engine light in the Camry that'd been on for two years and 5 days, and give my 1995 truck a tuneup--plugs, wires, rotor, distributor cap.  I had all the parts, and plenty of hand tools.

Simple enough....

In Which Your Author Hugs a Camry

I knew the problem with my Camry was the Vacuum Switching Valve.  It was throwing trouble code P0401, which meant a problem with the EGR system.  Cars are so polite these days, telling you these things.  Back in the day, one had to diagnose from symptoms, now the sensors throughout the system tell you your car that's otherwise running fine needs help.  Okay, so it didn't need help, but I was tired of staring at a CEL, and I'd already paid about $400 to have the EGR valve and Modulator replaced. The internets told me it was the Vacuum Switching Valve.  What's a vacuum switching valve?  I've no idea, but given its powered and has an input and output vacuum line, I assume it's something the ECM uses to turn the EGR system on and off.

Here we run into trouble, because like most modern day DIY mechanics the first thing I did to prep for the task was to search Youtube for a suitable video.  No suitable video exists.  The closest I could get was a guy doing the swap on his 5SFE (2.2L Toyota 4 Cylinder) RAV 4, and that had blurry shots of some dark recess of the engine and lots of grunting.

...which, turns-out is exactly what's involved.

To wit:


(The little bugger in question is #8, above.)

So I chocked the wheels, jacked-up the car as far as I could, put the front on jackstands, then slid underneath.  Then I began the search for the VSV.  Uhh...as near as I could tell, the thing lived somewhere behind the intake manifold, above the motor mount, which is as close to the Bermuda Triangle you get in the otherwise-roomy 4th XV20 Camry nose.  Seriously--I've never found anything on this car hard to work-on.  Well, at 200,014 miles, today was the day.

I couldn't visualize it, and I couldn't get my hands on it.  I was stuck, so I slid out and tried to look from above.  That didn't help.   I finally figured I needed to take the passenger-side front wheel off.  

(This was when my wife and daughters showed-up and my "before your haircut's through" estimate evaporated like Jeff George's QB credibility in the middle 1990's.)

With the wheel off I could finally visualize the VSV.  Well, okay, I could visualize the bolt that held on the VSV and a sure pathway to get at it:  Lie directly beneath the passenger subframe, and fish my left hand between the exhaust and the subframe, then send my right arm in behind the wheel hub and between the engine mount and the engine accessory pulleys.

Once I did that, I could use a 12mm 6-point 3/8" drive socket, a 6" extension, and a U-joint attached to a ratchet to get it off.  

It struck me how similar this must be to laparoscopic surgery, but without the benefit of a laparoscope.

Basically, from there, I got the VSV off, put the new one on, and got things buttoned up after only 2.5 hours.  The car ran, and after I disconnected and reconnected the battery terminal, the CEL cleared.  VICTORY!

In which Our Author Renders American Iron Immobile

Alright.  A tuneup.  This should be easy--I've done plugs and wires on the Camry (twice), the bmw (once).  All didn't require a trip on a rollback to be fixed.  I was even handy with a feeler gauge.

Problem the first:  My feeler gauge had gone missing.  Trip to Advance Auto Parts #1 for the day.

Problem the second:  All the aforementioned cars had had at least one tuneup after leaving the factory.  That's right, as near as I could tell, my 1995 Chevy TBI 5.7L was running roughly 20 model years later on the original ignition system.  Bravo, AC Delco.  Bra-frickin-vo.

Some things I found out doing a tuneup on a small block chevy:
  • Sitting in the engine bay is a requirement.  This was a novelty for me, having come up long after the death of big American cars.  Thanks to Chevy sticking the distributor behind the intake beside the firewall, you'll either be laying on the engine or sitting beside it.  By the end of the day, I'd done both.
  • You've got to remove the Air cleaner and it's metal housing, exposing the Throttle body injectors (awww, cute...this is how to do fuel injection on an old carbureted motor!  Make a fuel injector that fits on a carb manifold!)
  • Disconnect the battery.  No, nothing happened, but an energized ignition coil has like 22k volts. 
  • Tools you'll need:
    • SAE wrenches and sockets.  This thing is American as they come, right down to the bizarro sockets in the 32nds of an inch.  Aside from my GTO, I'd always worked metric, but I had plenty of SAE.  Specifically:
      • A 5/8" deep socket or spark plug socket of the same diameter, 3/8" drive
      • A little screwdriver-pick thingy.  Something like the third tool from the left here: 
    • WD40.  You're dealing with steel and cast iron.  Things will be rusty.
    • A 7/32nds 1/4"-drive socket and a 6" 1/4 inch drive extension.  You'll be sorry if you don't have this when it comes time to take the distributor off.  I had to make another trip to Advance Auto to get a (very overpriced) 1/4" extension to make this work.
    • A 3/8ths drive micro-torque wrench that reads up to 250 inch-lbs  (correct, inch-pounds.  This isn't the 2' long jobber you use to torque your lugnuts.  This is a delicate instrument.
  • Do one sparkplug and wire at a time.  Don't touch the distributor until you've done all the wires.  I must've spent a total of 1.5 hours just taking the darn retaining clips off the wire hangers on both sides of the engine.  These will be old, brittle plastic, and that pick is invaluable to get them loose.  Once I got going, I usually replaced the wire, then replaced the plug.
  • Gap the plugs to .035" ("Thirty-five Thousandths") That's the spec for a SBC of this vintage.  
    • The plugs you get from AC Delco will be pre-gapped at .045", so you'll need to re-gap each one.
  • Start the plugs by putting them in the socket, grab the socket with your hands, and rotate the plug counter-clockwise until you feel it drop into place, then begin to tighten (clockwise).  That helps you avoid cross-threading the plug.
    • Seems like most manufacturers (NGK, etc.) tell you to avoid anti-seize, so I didn't use any
  • Torque to 20 ft-lbs or 240 in-lbs.  It's tight in there, so use the u-joint.
  • Squirt the distributor screw/bolts with WD40.
  • Use the 7/32" socket and the extension to remove the distributor cap.  Sure, you can try to use the phillips-head screws, but once you strip those hopelessly like I did, use the socket.
  • Transfer the wires one-at-a-time from the old to the new distributor cap.  This way you avoid screwing-up the firing order of your engine.
  • While the caps's off, yank the rotor off the top of the distributor.  It requires quite a bit of even upward force to come off.
  • Put the new rotor on.  Again, it requires quite a bit of force to seat it.
  • Put the new cap on, wires attached.  
  • Spend an hour trying to get the screws to go in.
  • Give up and go eat dinner.  Obsess about what went wrong.
So yeah, after all that, I didn't get it done yesterday.  I couldn't get the distributor cap to seat properly so the two screws that hold it on would start.  In desperation, I tried putting the old cap back on just so I wouldn't have an immobile 5000lb truck in my back drive, but I couldn't get that cap on either.

My going theory is the rotor's not seated fully, so the cap won't go down all the way.  Or something else obvious I'm missing.  In any case, an afternoon wrenching on my old cars, outside, with at least one success.

11 August 2014

Why are there no Software Development simulators?

Saturday, I took my daughters to the kids day/open house down at our local PBS station.  Getting past the claustrophobia of jamming hundreds of kids, strollers, and overwrought parents into the narrow hallways and anechoic studios, it was great kidly fun--lots of booths, sing-alongs, and face painting.  On our way out, we noticed a smart-looking medium duty truck painted like a firetruck, pulling a trailer packed to the gills with equipment.  Inside were two "simulator" booths where two robust gentlemen were showing how they train police officers and firemen to drive their vehicles in all sorts of conditions and situations.

Watching a ten-year-old attempt to pilot a full-sized firetruck in a city (protip: make wide turns), it struck me:
There is effectively no simulation training for Software Developers.
That's right:  For the pilot of your 777, there's a simulator.  She's required to log so many hours in life-threatening situations so that she can respond with confidence when the big crisis occurs.   You're trusting her with your life, and the only thing telling you she's worthy is her logbook of hours and her simulation record.  Apparently, there's something similar for firemen and police officers.

However, there's nothing like that for software developers, and I'd argue it's sorely missed.

First, professional software development gives you tunnel vision.  Software development is problem solving, and human beings have two methods to problem solve: Autonomous pattern matching and deep cognition.  Autonomous pattern matching is when you reach back into your cache of experience and retrieve THE WAY TO DO IT.  Otherwise, you really have to think.

Let me show you an example.  Ever see anyone solve one of those horseshoes with ring puzzles?


If you've never solved it before, your cache is empty.  You scan the thing, handle it, feel around it looking for a way out.   Your pupils are dilated, your heart rate increases, and your cortex is consuming extra oxygen--you're really thinking.  This is deep cognition.

Now, if you *have* seen one before, none of that physiology happens.  You grab the shoes, twist them just so, and out pops the ring.  Easy!  Your body barely reacts at all, and studies confirm your brain uses no extra oxygen.  Moreover, if you've done it enough times, you'll have a hard time even teaching someone how to do it.

All that to say: Once you've been developing software for awhile, many, many things move into that autonomous pattern matching system.  You become expert, but you also become blind.   You crutch on your autonomous system and your deep cognition atrophies.  (This is why my wife derides my omnipresent internet perusal as fluff.  Rightly, she concludes that I'm just absorbing more information to pattern match, without thinking critically.)

Why does this matter?   Because it costs money, and it can also cost lives.   Psychologists conclude that our deep cognition system, while immensely powerful, is fundamentally lazy.  It costs energy, and we're wired to avoid using it if we can.  That's just our biology.  When presented with problems almost like one in our cache, we'll rationalize that it "must be that" and go with our gut.

Our gut is sometimes wrong.  Some examples:

  1. That customer issue that looks like the 15 other customer issues with the same log statements.  Did you test the fundamentals and reproduce the issue, or just send them the patch and pray it fixed it?
  2. That sneaky concurrency bug.  Hey, all the code you've shipped worked that way, so why try and fix it?  (This is also the Gambler's fallacy, another product of our autonomous pattern matching system)
  3. How many times have you just stared at a stack trace, hex trace, or other telemetry and had a flash of insight as to what it "must be."  That's a combination of your autonomous system jumping up and your cognition crying uncle.
That's why simulators or trial scenarios are so useful:
  • They draw on common experience.  Many firemen have driven over curbs, failed to stop because of poor weather conditions, and been unable to negotiate tight quarters in firetrucks.  The simulator lets those play-back for every newbie.
  • They're non-destructive.  Screw-up a crosswind approach in a 777, and 400+ people will die.  Do so in a simulator, and you learn without killing anyone. 
  • They provide consistency.  Everyone who's been through the simulator has been through the 'Kobiashi Maru' or whatever that equivalent is for your org.  That builds peer confidence in the capability of the organization.
  • They provide "impossible" situations.  It's unlikely an aircraft will have 3 major emergencies simultaneously without a SAM involved.  However, one can simulate those trivially in a simulator, and help a pilot understand how to balance the workload to a good result.  

The last is the big win for me.   Critical customer issues happen, and they're high-stress, and it's difficult to balance your own physiology and the matter at hand.  The insidious autonomous system looms, waiting to give you the "almost right" answer without you thinking.


Here's the thing:  I think we have the technology to build simulators right now, without extraordinary effort.  Here's what I see:
  • Use Vagrant and Chef to codify the machine configurations in question.  Maybe it's a downlevel CentOS box with a wonky configuration.  Let's say the http_proxy isn't configured and you want to show the trainee how the system behaves.  That's trivial to codify and check-in to your 'simulator'.  Destroying the box is no consequence; one can delete and rebuild with two commands.
  • Your 'development box' is a set of tools similarly checked-in to SCM, with vagrant and chef creating a vm (either locally, on a VM server in-house, or on IaaS).   Maybe you want to teach someone how to attach remote debug to a running JVM that's dying because of MaxPermGen config issues.  Easy enough.
  • You can use ssh or the equivalent to monitor and remote control the development box, much like a simulator instructor can turn on rain, wind, or other environmental factors.  Failing hard drive?  Flaky Wifi?  Throw the student some curve-balls, and make that cognitive system work.
Katas can help.  Learning new languages and paradigms can help.  

However, nothing would be better than a real scenario simulator.

06 August 2014

It's just that easy: Configuration as Code 2014

Ramping on a new project is fun!

Actually, it resembles lighting yourself on fire, running and jumping onto a moving train with jugs of water on it, only to discover the first car has flammable liquids, then the next car, then the next.

Restated: It's never boring.

So, I'm collaborating with a team writing in Rails 4, and that team's adopted the 12 Factor App religion. Great!  I've dabbled in heroku before, and it's neat to see apps written that way from the first git init.

Anyway, thought I'd jot down just how "easy" it is to get an app running.

  1. Install git.  Pull the code for the project.  Everything's self-contained from there, right? Er...
  2. Install VirtualBox.  This will let you create run virtual machines on your development box.  (You'll need about 8GB of RAM installed...you have that, right?)
  3. Install Vagrant.  This is like a build system for your virtual machines.
  4. Install Chef.  This is a configuration system for your virtual machines.  Once you boot the machine, you run chef to install software, set permissions, etc.  
  5. Install the vagrant plugins listed in the README.md file.
  6. Use the 'berksfile' utility that comes with the 'Chef-DK' (har!) to 'vendor' your dependencies.
    1. Vendoring is deep-copying any external dependent applications under your app.  That way, you're insulated from change should someone go and upgrade one of those dependencies to an incompatible version.  It's one way of handling 'DLL Hell' situations.  Another (more draconian) method is #golang's method of just compiling everything into one static executable.
  7. Do 'vagrant up' and wait a few minutes for it to create your configured running machine from scratch.
When it works, it's fricking beautiful.  You've eliminated whole classes of bugs:  System configuration vagaries, OS package differences.   All developers are using the same system to develop, test, and deploy.

When you're a n00b, it's a bit daunting, but I'm liking it.  If nothing else, it lets me develop on my OS of choice, then run on a different target--usually the same one as production.  

05 August 2014

"Ideal team size 5 to 10." (Still no cure for cancer)

You've gotta love social scientists.

In the ongoing quest to squeeze every ounce of productivity from the burnout-destined drone age 20-to-40, they're studying ways to measure collective intelligence.

Quotable quote:

Right now, the optimal size is probably somewhere between five and 10, but with the right collaboration tools, you could imagine having a group that kept getting more intelligent, up to 50, 100, or even 500 or 5,000 people. 

::sigh::

Okay, be proud: You've got your name on the company, and you're the centerpiece of this spiffy article.  What you're trying to do, though, it surmount human biology:  We can keep 7 +/- 2 ( that is, anywhere from 5 to 9) things in our active memory at any one time.  Whenever you go above that number, we forget.  Managing over that many relationships day-to-day simply creates overhead.