Saturday, June 25, 2016

i'm not a print designer but i play one on the web

It's Porchfest season! Besides helping Somerville's Porchfest website survive the onslaught, I've been doing a ton of design work for JP Porchfest.

Last year we started making 11x17 Porchfest Poster Maps to have at various locations (one part of me thinks everyone should "save a tree" and have people just use our web/mobile site, but there's still something that attracts people to the physical artifact.) With 70-80 porches and 100-200 bands, getting all that information on a sheet of paper is a design challenge.

I'm not a print guy, and the information was already online, so I created a web page to use as raw material for the page layout. One trick is how I used the (somewhat obscure) CSS property "zoom" to work at 3x size, to account for how print is 300dpi and the web tends to be more like 72. I then used that phantom.js trick to make a screenshot, even though the page was much bigger than my screen.

Here was last year's result:

I like the bold title on block schedule site. That was really a way of me coping with a layout issue, the block schedule "wanted" to be skinnier than the page, but the end result was striking.

I did a lot of work with the iconography, both highlighting sponsors and figuring out how to show performance sites that had toilet facilities.  (Near the end of the work process, I got told I needed to put in a lot of acknowledgements, hence the odd jumble at the bottom, but I think it kind of worked.)

Observing people using the posters in 2015, I realized there was a usability issue; people had to keep flipping back and forth to see what bands were playing vs where they were located. I thought about trying to somehow integrate the block schedule over the map, like in bits and pieces, but that seemed pretty labor intensive (and there are always plenty of last minute changes that would have to be incorporated). Instead, for 2016 I decided to arbitrarily divide JP's map into North and South sections:

With this layout I almost had "too much" space, so I filled it with extra titles and logos and stuff.

I miss the bold sideways title, but I think it came out pretty well. I'm worried people won't "get" the North/South split at first, but if they look for a second I think it will make sense, and in fact seeing the numbering broken up might make even more clear that the number is by north/south order. (Which isn't perfect - I wasn't smart enough to come up with an algorithm to sort by north south but by "visual cluster" instead of strict latitude.)

Friday, June 17, 2016

doing the thing right vs doing the right thing

Getting back to the concepts of unit testing etc - there was this intro article on TDD. (Not that we’re fully embracing that path at my current employer - though obviously some principles apply) I always hate how cherry picked TDD examples are - always functional programming-ish-- some subroutine function that does a simple data transformation, so the input is (often extremely) straightforward, and the output is similarly self contained, and easy to check, and there are no side-effects or external dependencies to think about.

Pondering on the coding I do (and preparing to drink the kool-aid of TDD and heavy unit testing-- or at least more deeply understand what its fans dig so much about it, and why they find it worth the self-evident cost of writing everything twice, and maintaining tons of mocks as systems grow) - I think most of the UI code I’ve done here falls into 3 rough categories:
  1. ‘pure functional’ transformations (rare-ish)
  2. functions fiddling with the UI, with fairly tightly coupled side-effects (angular does better than some at reducing the coupling)
  3. general plumbing functions, really just infrastructure getting data from one place to another, maybe translating along the way. 
Only the first category is really friendly with unit tests. I know some of the drive of TDD is to learn to factor so that category 2 and 3 look more like 1. For example, refactoring a form validation so everything is happening at one place, a decoupling which helps both in testing and understanding. I see the value of that. (2020 Update: some frameworks like React are more amenable to slotting in with tests than old Angular was.)

But I feel like the majority of bugs I actually run into take place “out-of-band” of this kind of testing - like with that form validation, it’ll turn out the view doesn’t update the DOM like I expected with the scope validation variables. Or out-of-band in a “falls between the cracks” sense- say my new component made assumptions about the format of its input, and its tests are all green, but the old component its getting the data from had different assumptions about its output, but of course ITS tests are coded around THOSE assumptions.... so both pass their tests fine, but those 2 trees aren’t making a good forest.  Yeah, that’s what the functional or integration tests mean to catch, but then I can’t get away from the thought that most good “units” for unit testing are so trivial that when they fail, it’s some environmental glitch - and the testing environment is often a bit different than the real running environment anyway.

(I kind of scoff when I read, for the umpty-umpth time, 'see, you'll sleep better, because you KNOW YOUR CODE WORKS because it's unit tested'. People get paged at 2am because disks get filled up, or data in the format from 3 sprints ago got pulled up and mis-displayed (but all the tests were updated to assume the new data format), or connectivity to the backend server temporarily went down. It's almost always big stuff outside of the scope of tests, test that tend to be run in their own microcosm runtimes, and it's hardly-ever 'oh that error would have been caught as an edge case in your oh-so-clever unit test' - dang it, if it was an edge case I could have thought of when I was writing the unit, I would have made the code work right, so that then the unit test would just be bragging! (Admittedly, unit tests are pretty good at documenting the edge case to make sure a refactored version of the code supported the same weirdness. But I also hate, hate, hate the idea that unit tests can substitute proper, human readable documentation. You still want a map of the holistic forest, not just a listing of the reductionist trees.)

I've found SOME parallels in the way I write code relative to the Proper Unit Tester (PUTter). I'm a ridiculously incremental coder; I hate being "wrong", and kind of scared of making a block of code so big that when I'm finally ready to run it I don't what part might be going wrong, and so I've always relied on tight code/run/eval loops... lots of manual inspection and testing of assumptions at each little baby step. Often I'd even write little homebrew test functions for the various edge cases. The difference between old me and a true PUTter, then, is I would then throw away those little functions and logging statements in the same way a building site takes down the scaffolding once the building is standing. But PUTing says - look, that's how you know this code still works in the future, nurture that scaffolding, keep it around... make it survive any refactoring, fix all the mocks when the endpont input or output gets added to, etc etc.  That's a huge cost, and I'm still trying to convince my self it's worth it.

In short, unit tests are decent at A. saying you did what you said you'd do, but terrible at B. saying what you're doing is the right thing. And that's a problem for me, because they seem about as expensive as the core code to do up, and being benefit A alone make the expense harder to justify. (I mean really, shouldn't most proper "units" be too small to fail anyway?)

(See also this slide show I made, more gently and shallow-ly griping about this kind of status quo.)

Wednesday, June 15, 2016

php recursive directory iterator and funky character cleanup

Lately I've been switching to PHP from Perl, not just for server processing for basic webpages but for command line maintenance tasks as well. The syntax is so much less wonky, the core documentation solid, and the baked-in utility functions are so well thought out, that I haven't been missing Perl much at all. (On the other hand, I seem to be memorizing very few php functions names - but the process to get stuff done is still so fast, even with all the googling involved.)

(And like I said last year, PHP is much better than Javascript at dealing with global variables in functions.)

I've been running the user-submitted poetry site for decades, literally (if not so literarily). It's had a few overhauls, but it's got is still that bad old Perl, and I guess most of it isn't super-UTF8/16 aware. Usually that's not a problem, but sometimes (I think when someone copies and pastes from Word or what not) funky characters show up - smart quotes, ellipses, etc. I wanted to do a search and replace across my file store. (Guilty confession: for medium-size sites that don't need a lot cross-content grepping, I tend to turn first to flat files in folders- often many small ones, so something like atomicity is preserved even if things go wrong. It has some drawbacks, but I never have to worry about database conversions, and I obviously have a number of tools for hacking the content directly)

Anyway, here is the hack I came up with... and it had to do a lot of work, the Blender has had over 27,000 submissions over the years.

the core idea was to come up with a set of "replacers", and then do a simple search and replace with fixFile()... but to get there I need a few iterations of diagnostic work to figure out what the replacements should be, you can see lookForFunkyCharactersInFile() for oldschool "8-bit ASCII characters 128-255", mostly, and lookForFunkyEntityStringsInFile() for this longer bit of escaping that has been showing up lately. (For the former was a huge help at figuring out what bad charaters were meant to be, I made a quick macro to import a big swath of the the foriegn characters in a single gulp.)


$directory = '/home/kirkjerk/domains/';

$breaker = 0; # either only show one "funky character" type or 0 for all

$replacers = array(
    "&#226;&#128;&#153;" => "'",
    "&#226;&#128;&#156;" => "\"",
    "&#226;&#128;&#157;" => "\"",
    "&#226;&#128;&#166;" => "...",
    "&#195;&#162;&#226;&#130;&#172;&#194;&#166;" => "...",
    "&#195;&#162;&#226;&#130;&#172;&#226;&#132;&#162;" => "'",
    "&#195;&#162;&#226;&#130;&#172;&#197;&#147;" => "\"",
    "&#195;&#162;&#226;&#130;&#172;&#194;&#157;" => "\"",
    chr(128) => "",
    chr(131) => "&fnof;",
    chr(133) => "...",
    chr(144) => "",
    chr(145) => "'",
    chr(146) => "'",
    chr(147) => "\"",
    chr(148) => "\"",
    chr(150) => "-",
    chr(151) => "--",
    chr(152) => "", #probably some kind of soft hyphen
    chr(153) => "", #probably some kind of soft hyphen
    chr(156) => "", #probably some kind of soft hyphen
    chr(157) => "", #probably some kind of soft hyphen
    chr(160) => " ",
    chr(161) => "&iexcl;",
    chr(163) => "&pound;",
    chr(165) => "&yen;",
    chr(166) => "",#probably some kind of soft hyphen
    chr(167) => "&sect;",
    chr(169) => "(c)",
    chr(170) => "",
    chr(173) => "",
    chr(174) => "(r)",
    chr(180) => "'",
    chr(183) => "*",
    chr(189) => "1/2",
    chr(226) => "'", #this one shows up even tho it's also &acirc;
chr(191) => "&iquest;",chr(192) => "&Agrave;",chr(193) => "&Aacute;",chr(194) => "&Acirc;",chr(195) => "&Atilde;",
chr(196) => "&Auml;",chr(197) => "&Aring;",chr(198) => "&AElig;",chr(199) => "&Ccedil;",chr(200) => "&Egrave;",
chr(201) => "&Eacute;",chr(202) => "&Ecirc;",chr(203) => "&Euml;",chr(204) => "&Igrave;",chr(205) => "&Iacute;",
chr(206) => "&Icirc;",chr(207) => "&Iuml;",chr(208) => "&ETH;",chr(209) => "&Ntilde;",chr(210) => "&Ograve;",
chr(211) => "&Oacute;",chr(212) => "&Ocirc;",chr(213) => "&Otilde;",chr(214) => "&Ouml;",chr(215) => "&times;",
chr(216) => "&Oslash;",chr(217) => "&Ugrave;",chr(218) => "&Uacute;",chr(219) => "&Ucirc;",chr(220) => "&Uuml;",
chr(221) => "&Yacute;",chr(222) => "&THORN;",chr(223) => "&szlig;",chr(224) => "&agrave;",chr(225) => "&aacute;",

chr(227) => "&atilde;",chr(228) => "&auml;",chr(229) => "&aring;",chr(230) => "&aelig;",
chr(231) => "&ccedil;",chr(232) => "&egrave;",chr(233) => "&eacute;",chr(234) => "&ecirc;",chr(235) => "&euml;",
chr(236) => "&igrave;",chr(237) => "&iacute;",chr(238) => "&icirc;",chr(239) => "&iuml;",chr(240) => "&eth;",
chr(241) => "&ntilde;",chr(242) => "&ograve;",chr(243) => "&oacute;",chr(244) => "&ocirc;",chr(245) => "&otilde;",
chr(246) => "&ouml;",chr(247) => "&divide;",chr(248) => "&oslash;",chr(249) => "&ugrave;",chr(250) => "&uacute;",
chr(251) => "&ucirc;",chr(252) => "&uuml;",chr(253) => "&yacute;",chr(254) => "&thorn;",chr(255) => "&yuml;"


$it = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($directory));

while($it->valid()) {
    $pathname = $it->getSubPathName();
    if (endsWith($pathname, ".work")) {

function fixFile($pathname){
    $guts = file_get_contents($pathname);
    $guts = replaceBads($guts);

function lookForFunkyEntityStringsInFile($pathname){
    $lines = file($pathname);

    foreach($lines as $line){
        $line = replaceBads($line);
            $url = pathnameToLink($pathname);
            print $pathname."  ".$url."\n".$line."\n";

function lookForFunkyCharactersInFile($pathname){
    global $breaker;
    $lines = file($pathname);
    foreach($lines as $line){
        $line = replaceBads($line);
        $lineIfFunky = isFunky($line,$breaker);
       if($lineIfFunky != "") {
            $url = pathnameToLink($pathname);
            print $pathname."  ".$url."\n".$line.$lineIfFunky."\n";

function isFunky($line,$ugh){
    for($i = 0; $i < strlen($line); $i++){ 
        $c = substr($line,$i,1);
          ($ugh == 0 && ord($c) >= 128) || 
          ($ugh != 0 && ord($c) == $ugh)
          ) {
            return str_repeat(" ",$i)."^         ------ at pos $i got ".ord($c);
    return "";
function pathnameToLink($path){
    return "".str_replace(".work","",str_replace("/",".",$path));
function replaceBads($line){
        global $replacers;
      foreach ($replacers as $badFrom => $badTo){
        $pos = strpos($line,$badFrom);
        if ($pos !== false) {
           # print "  FROM ".$line;
            $line = str_replace($badFrom,$badTo,$line);
           # print "   TO  ".$line;
    return $line;
function endsWith($haystack, $needle) {
    return $needle === "" || (($temp = strlen($haystack) - strlen($needle)) >= 0 && strpos($haystack, $needle, $temp) !== false);


Friday, June 10, 2016

the little things

Sometimes it's the little details, seemingly utterly dropped on the floor, that bum me out the most.

OSX's calendar app's week view shows you the current time, via a horizontal line, which works well enough, but then uses a circle to the right of the day column to indicate which day it is.

Would it have killed them to make it, say, a triangle/arrow pointing to which event/day it's referring too? Or somehow otherwise bracket the current moment - two circles, or maybe ( ) or < > characters to their side?

I guess the "you are here" circle is meant to mirror the circle used to highlight the current day number at the top, but to be honest this post is the first time I noticed the circle at the top, because my attention (and "the user") in general is below that, in the day view itself.

Friday, June 3, 2016

the amazing palm pilot

Before the iPhone, before Microsoft's "Pocket PC", but after the Newton... there was Palm.

For nostalgic purposes, I recently ebay'd up an original Pilot 1000 - back before they even had backlights, with 128K of memory. (Admittedly, that was a bit cramped even for the era.)

Every once in a while I fantasize about disconnecting and going back to this PDA - and while forcing a social media disconnect and getting over my need to always be available is tempting, going back to a time when I had to have separate gadgets for photos, GPS, and music seems a bit too far.

Still, for juggling notes, my datebook, and TODOs, I could totally be content with a gadget like this. There's something so comforting about scratching out letters with the stylus. The "Graffiti" writing system was pretty great, and sometimes I still draw my 'E's using its backwards-3 approach:

(It's a real shame Xerox won its bogus lawsuit, saying its "one stroke, easy for a computer to recognize" system was ripped-off by Palm... I mean look at this junk:
who could ever remember that? And so Palm replaced Graffiti with Graffiti 2, a generally inferior system that bedeviled people who had internalized the original characters.)

The best resource I dug up this round of exploring the device's history was the PalmPilot page at the Computer History Museum. I had heard of, but never seen a picture of, Jeff Hawkins' wooden prototype:
Chopstick Stylus Included!
Also intriguing, a prototype for a keyboard-based Pilot:
Given the later (albeit impermanent) popularity Blackberry by offering keyboard devices this model provokes some "what-if" thinking. Like the museum's Jeff Hawkins video explains, the pen wasn't the point, but simplicity, portability, and connectivity were. It succeeded where the more capable Newton failed by getting more of those factors right.

Palm: The Rise and Fall of a Legend was another good history I found, and How Did We Get to the iPhone is well worth the 99-cent admission fee, covering an even wider swath of gadgets such as the Psion.

Heh, maybe next time I should talk about the Game Boy or the Etch-A-Sketch Animator, two other gadgets that made brilliant use of that chrome-ish, big-pixel technology.

Thursday, June 2, 2016

trivial PHP survey/email collection with redirect PHP

Brookline Porchfest folks wanted a simple email collection script for a one-page site I made and host for them. Putting this here for future reference, save my future self 20 minutes of reading PHP docs in the future: (I called it "save.php" and made sure there wasn't any whitespace before the first tag, which might mess up the redirect)

    #$email = str_replace(array("\n", "\r"), '', $_POST["email"]);
    $email = $_POST["email"];
    if(filter_var($email, FILTER_VALIDATE_EMAIL)){
        header( 'Location: thanks.html' ) ;
    } else {
        header( 'Location: error.html' ) ;

So, simple... get the email address submitted, if it looks legit append it to a text file and redirect to a thanks page, otherwise redirect to an error page. (It seems a good practice to dump to a new page, both to avoid reload issues, and so that this functional script is more reusable, separation of concerns.)

The first line is commented out, before I realized PHP had a decent email validator I wanted to at least make sure that any garbage entered was at least limited to one line, even if I was mostly relying on post-facto data cleaning.

Oh, and the form looks like

<form method="POST" action="save.php">
Join our mailing: 
<input name="email" placeholder="Your E-mail Address"> 

I really appreciate how easy PHP makes stuff like this, and the docs are really clear and straightforward as well.

the same thing in different ways

Fronted HyperPolyglot: the same little snippets as expressed in different frontend frameworks.