Monday, August 19, 2019

the social network graveyard

Why These Social Networks Failed So Badly - myspace and vine are the ones I think are the biggest losses. Along with the blogosphere in general.

Tuesday, August 13, 2019

meanwhile, at a place of business...

Coding for Fun and the Culture of Learning I made an entry for my company's engineering blog, about the fun of old 8-bit computers, the comapany's Peer-led classes, and the fun of programming stuff in Processing and p5.js.

I haven't written post there before this, in part because I'm less comfortable speaking in the authoritative voice I think an enterprise blog would need - my style is much friendlier and "here's what I found out, let me know if it's useful".

Monday, August 12, 2019

the email gun

I just finished the Emily St. John Mandel's post-apocalyptic novel  "Station Eleven". One recurring trope is how non-dark screens for computers or phones are just a memory-- I enjoyed this passage:
"Why did we always say we were going to shoot emails?"
"I don't know. I've wondered that too."
"Why couldn't we just say we were going to send them? We were just pressing a button, were we not?"
"Not even a real button. A picture of a button on a screen."
"Yes, that's exactly what I'm talking about."
"There was not, in fact, an email gun. Although that would've been nice. I would've preferred that."
...I recorded a few more quotes from the book in my commonplace blog.

Email guns are the best gun idea I've heard since the videogame Saints Row IV's Dubstep Gun:

Friday, August 9, 2019

on privacy and adblockers

At work someone posted a piece from the EFF, Adblocking: How About Nah? Talking about the arms race between users who want to see stuff online and resent the privacy violations ad providers utilize and the content providers and the advertisers. "How About Nah?" is the message of a consumer using an Adblocker to the site that wants them to accept the ads along with the content.

My responses were as follows:
Two thoughts: 
First is, one example of an “adversarial interoperability” are botnets that pretend to suck down content and dutifuly view the ads  - or sometimes just watching the ads themselves. If you’ve ever wondered how that Captcha of “I am not a robot” works when it’s just a checkbox - it is tons and tons of data scraping, from if your mouse movement is following human-ish patterns to IP locations and the like. One thing I learned working for adserving platforms is how there is rampant number abuse, and a completely unfair landscape for anyone trying to set up an “honest” ad service - it has to look competitive with completely inflated numbers from the other folks services 
The other is, “How About Nah” has a lot of swagger, but I think a more relevant concept - one the article doesn’t touch on - is “information wants to be free, but rent wants to be paid”. So much of this springs from the 90s establishing how people will embrace crappy and free before clean and cheap. 
I can’t blame people - they are rightfully nervous about how a penny here, a 1/10 penny there might add up. I think the first iPhone succeeded hinged on its plan from AT+T featuring all you can eat data - (which given networks speeds wasn’t all that much :-D)  - but if people made the move from the hell of WAP (infamous for nickel and dime-ing folks) to the real Internet on the go, they needed some reassurance they weren’t going to get hosed on data fees. 
Personally, I have a moderate stance on privacy - instinctively I don’t care that much as long as I know my value is only as a demographic’d consumer, and I don’t like that the conversation doesn’t distinguish between truly invasive privacy stuff (like humans seeing my content I thought was private) and more run of the mill stuff — or from people who take the “it’s my computer and my bandwidth, how dare those hucksters steal it” while ignoring the costs and salaries that might have gone into getting the content up that brought the person there in the first place

Wednesday, August 7, 2019

js: on the conditional deleting/removal of things

Making up my p5 "good parts" guide I noticed that a version of the same problem that haunted me in the Java days (safely removing an object from a collection) is still sort of around-
if I have, say, a collection (either an array or an object I'm treating as a hashmap) of key/value objects and I want to remove particular ones, what does that look like?

Probably the cleanest way in modern JS is to set the array equal to a filtered clone of itself:
let arraythings = [
 arraythings = arraythings.filter((thing)=> (thing.msg !== 'killme') ); 

and if that collection were an object:

let things = {
  'a':{msg:'ok1'}, 'b':{msg:'killme'},'c':{msg:'ok2'},
things = Object.entries(things).reduce((acc, [key, value]) => {
    if (value.msg !== 'killme') {
        acc[key] = value;
    return acc;
}, {});

A coworker helped me with that, better approach that my idea of making an array of keys to kill and doing a foreach. But the reduce approach looks kind of ugly to me. I'm glad this situation doesn't come up very often, but I should probably work on being more fluent with reduce().

It is interesting that neither approach is super Funtional Programming-ish, and in both cases I am reassigning the collection variable (though it always seems weird to me that a const object is just guaranteeing that you're pointing to the same collection, not that the contents of said collection are in any way frozen... that's what Object.freeze() is for.

p5 reference: the "good parts" edition

UPDATE: Ben Moren made a great cheatsheet for beginners [My Mirror Here] that is a good complement to what I have here...

I recently taught a 2 hour class in programming for fun - "After Hours: Making Virtual Toys, Games and Art".  It was a fun chance to show people (both experienced developers and programming novices) how to use Processing and P5.js - and the P5 editor is the best online thing I've seen for learning and sharing.

Both P5 and Processing have great websites with lots of learning tools, and it is awesome how all of their reference pages come with a tiny, ready-to-run example. But, for example, the p5 reference page has dozens and dozens of functions and special variables - I think that is intimidating to a new user who might only need to know 10-20 commands or so to make a lot of cool stuff. I made this page to be just those functions and special variables I use again and again in my decade and a half of Processing and P5.  (Stuff that's marked "BONUS" isn't as crucial but can add in the fun.)

Start up:
  • setup() - make this function and it runs once, at he beginning
  • createCanvas() - call this in setup() to set the width and height of your canvas
  • BONUS:  preload() - if you need to load images etc and don't want to start until its loaded, you should know about making this function...
Your main loop:
  • draw() - make this function and it gets called every clock tick
  • background() - many times your routine will start with this to erase the background (see the coloring things section below, and remember you can even set the transparency value for a cool ghost effect.)
  • width and height - often as you draw it's good to check these special variables and scale things properly
  • BONUS: call noLoop() if you don't actually need to loop - that can save device battery! And then call loop() if you change your mind
  • EXTRA BONUS: frameCount is a variable telling you the current frame number... sometimes good to setup a score time or somesuch if the goal is to stay alive as long as possible...
  • EXTRA EXTRA BONUS: saveCanvas() makes it super easy to save a snapshot of what's on your canvas... just be careful if you're doing it again and again and again inside your draw() loop!
Things to draw:

It's probably easy for non-programmers to forget how weird it is that for x,y coordinates, 0,0 is the top left corner, and 100,100 (or whatever the Canvas size is) is the bottom right... that applies to all of these commands:

Coloring things and pen widths:

The p5.js color system is a little odd but simple and fun once you get it... functions like background() and fill() and stroke() can take 1 argument (a number from 0 (black) to 255 (white)), 3 arguments (3 numbers from 0-255 for Red, Green, Blue), and then for either of those you can add an EXTRA argument to say how transparent it is (0 = it's invisible! 255 = it's fully opaque)
Interact-y stuff (mouse and keyboard):
BONUS Content:
function draw() {
  if(mouseIsPressed) line(pmouseX,pmouseY,mouseX,mouseY);
is like the world's simplest paint program!

Mathy Mathy Math
  • random() - often for generative art, it's nice to have a random number...
  • dist() - Pythagorean theorem - how far apart are two points? (useful to know if two circles overlap, say)
  • map() - if you want to take a number that might be in a range from 0 to 1 and change it to a range from 0 to 100, map() is your friend
  • cos() / sin() - sometimes you need to remember your old Geometry class...
  • BONUS: constrain() - limit a value to a range
External graphics

 Because I'm lazy, I often prefer to make games and toys with the shape drawing stuff, but sometimes it's good to know how to import an image...
  • loadImage() - load the image
  • image() - draw the image you loaded - you can mess with its size and scale and stuff too

BONUS: Changing the frame of reference for where we draw

It's often useful to change where "0,0" is, and/or rotate stuff for some cool effects...
  • push() - remember where we were drawing now
  • translate() - move to where we want our new center of drawing to be
  • rotate() - spin stuff around (PROTIP: you usually want to call translate() then rotate() unless you mean to spin the world
  • pop() - pop back to where we push()d before


These are only the commands I've found myself using again, but P5 does a ton more.. I've barely touched the sound APIs (so many of my games are quieter than they should be) - there's also stuff with microphones, and some mobile-specific stuff as well.  The p5 examples page can be an inspiration in many ways, and teach you basic programming concepts too.

Nerdy Javascript basics...
This page isn't meant to be a programming class, and the p5 examples page can teach you many things, and there are some great youtube tutorials - The Coding Train seems pretty cool.

Obviously, you should know about variables - historically you declare a variable with var (like var x = 10; )though more modern javascript favors saying let x = 10; (for things that change) or const x = 10; for things that are "constant" and don't change.

Also, it's usually pretty easy to set up an array (using [] symbols) of objects (key-value pairs that are wrapped in {}). So you might have an array of, say, circle objects that you "iterate" over (i.e. go over each one) and moving them (maybe drawing them to the mouse?) and drawing them, so they can be seen

One thing that's trickier, oddly, is safely removing stuff from an array. In some cases, you might just want to mark something as "removed" and ignore it... but I'm going to include an example of using the (modern javascript, not p5 in particular) filter command to remove it.

Here's a simple program that just puts 10 dots on the canvas:
var dots = [];
function setup() {
  createCanvas(400, 400);
  for(var i = 0; i < 10; i++){

function draw() {
//fancier way of the old for(var i = 0; i < dots.length; i++) 
// { var dot = dots[i];
  for(dot of dots) { 

So each dot is an object with an x and a y propery: {x:_, y:_ }. Lets make a version of this program where if you click on a dot, it gets an extra "imdead" property, and then skip drawing it. Here's all the code with the new parts in bold...

var dots = [];

function setup() {
  createCanvas(400, 400);
  for (var i = 0; i < 10; i++) {
    dots.push({x: random(width),y: random(height)});

function draw() {
  for (dot of dots) {
    if (!dot.imdead) {
      ellipse(dot.x, dot.y, 20, 20);

function mousePressed() {
  for (dot of dots) {
    if (dist(dot.x, dot.y, mouseX, mouseY) < 10) {
      dot.imdead = true;

So, if we didn't want to check imdead all the time (in this example it's no problem to ignore the dead ones, but either for performance reasons or just for simpifying code we might not want to have to keep checking) we can change that draw() loop to what follows:

function draw() {
  for (dot of dots) {
      ellipse(dot.x, dot.y, 20, 20);
  dots = dots.filter(dot => (!dot.imdead));

we got rid of the imdead check wrapping the ellipse() command, and then used a fancy new js "filter()" that kicked out all the imdead dots. Obviously there are many ways we could have used that filter (like right in the mousePressed() for instance) but in my experience, setting a property like that and then removing the thing later is something that comes up a lot in games I make. (Like maybe the alien plummets to its death before being removed from the screen...)

Tuesday, August 6, 2019

wacky joins of flat json data and dumping to excel

Just keeping this around for my own reference, it might be too esoteric for anyone else.

I use a filesystem based database for my Porchfest work: one folder roughly equals a table, and each json file corresponds to a row, and the filename acts as the key. (and each file is a simple map, where the keys correspond to column names and the values are the column values)

So I have one folder for porches, one for bands, and one for "gigs", which is one to many join table for porches going to bands, along with what time.

My porchfest runner asked if I could dump out the porch / band / what time info to a flatfile (or rather, to a dirty "tab delimited file pretending to be a spreadsheet" that excel lets me get away with)

The format she wanted was a little idiosyncratic: one row per porch, and then each band and gig kind of flattened on the same row-- like if I was doing it on my own I might have done it

so that columns were "clean" and nothing was duplicated (but different rows mean different things) or possibly
where columns are still clean, but information is duplicated...

She asked for something more like

I personally find it less intuitive, since different porches have different numbers of bands, and columns have to be "sort of" duplicated etc, but I kind of get it... I'm thinking too much like a computer and not an excel user :-D

Anyway, here's the code that does it.  I read in all the metadata as GET parameters (though the commented out variables give a feel for the values I'm passing in ), I build a top row as header values, and then if I'm passed a file name I use that dirty old php to excel hack otherwise I just dump the info to the browser <pre> tags

    prints out select contents of three tables assuming a one to many relationship,

   print out will be 
      leftdata rightdata1 joindata1 rightdata2 joindata2
        in practice:
        porch1 band1 gig1 band2 gig2
        porch2 band3 gig3

        $lefttable = "porch";
        $righttable = "band";
        $jointable = "gig";
        $leftjoinfield = "porchid";
        $rightjoinfield = "bandid";
        $leftfields = "address|spacetype|zone|capacity|raindateable|notes|maincontact|email";
        $rightfields = "bandname|email|howloud|needelectricity|bandsize|candoraindate|preferredstarttime|performancelength";
        $joinfields = "starthour|startminute";
        #$leftfields = "address";
        #$rightfields = "bandname";
        #$joinfields = "starthour|startminute";
        $lefttable = $_GET["lefttable"];
        $righttable = $_GET["righttable"]; 
        $jointable = $_GET["jointable"];
        $leftjoinfield = $_GET["leftjoinfield"];
        $rightjoinfield = $_GET["rightjoinfield"];
        $leftfields = $_GET["leftfields"];
        $rightfields = $_GET["rightfields"];
        $joinfields = $_GET["joinfields"];
        $filename = isset($_GET["filename"]) ? $_GET["filename"] : "";    
        $leftguts = getGuts($lefttable);
        $rightguts = getGuts($righttable);
        $joinguts = getGuts($jointable);
        $buf = "";
        $maxjoins = 0;
        foreach($leftguts as $leftkey=>$left) {
            getValuesForKeysInString($left,$leftfields)  ;
            $joinmatchkeys = array();
            foreach($joinguts as $joinkey => $join){ //go over all keys in join table
                if($join[$leftjoinfield] == $leftkey) { //if this is a join
            if(count($joinmatchkeys) > $maxjoins) {
                   $maxjoins = count($joinmatchkeys);
            foreach($joinmatchkeys as $joinmatchkey) {
                $rightkey = $joinguts[$joinmatchkey][$rightjoinfield];
                #$buf .= "$rightkey\t";
                getValuesForKeysInString($rightguts[$rightkey],$rightfields)  ;
                getValuesForKeysInString($joinguts[$joinmatchkey],$joinfields)  ;
            $buf .= "\n";

        $headerbuf = "";
        foreach(explode("|",$leftfields) as $leftfield) {
            $headerbuf .=  "$leftfield\t";   
        for($i = 0; $i < $maxjoins; $i++){
            foreach(explode("|",$rightfields) as $rightfield) {
                $headerbuf .=   "$rightfield\t";    
            foreach(explode("|",$joinfields) as $joinfield) {
                $headerbuf .=   "$joinfield\t";    
        $headerbuf .= "\n";
        $buf = $headerbuf.$buf;
        if(! $filename) {
           print "<pre>$buf</pre>";        
        } else {
            header("Content-Disposition: attachment; filename=\"$filename.xls\""); 
            header("Content-Type: application/");
            print "$buf";
        function getValuesForKeysInString($guts,$keystring){
            global $buf;
            foreach(explode("|",$keystring) as $key) {
                $val = $guts[$key];
                $val = str_replace("\t"," ",$val);
                $val = str_replace("\n"," ",$val);
                $val = str_replace("\r"," ",$val);
                $buf .= $val . "\t";

        function getGuts($tablename) {
            global $dbroot;
            $guts = array();
            $path = $dbroot."/db/".$tablename."/";
            $files = scandir($path);
            foreach($files as $file){
                if(substr($file, 0, 1) != "."){ //ignore hidden files
                    $rowguts = json_decode(file_get_contents("$path/$file"),true);
                    $guts[$file] = $rowguts;
            return $guts;