Well. That was the idea, anyway. SPOILERS: It didn’t work.

Vladimir Costescu has upped the ante and bought a day of my time this month, requesting:

It would be cool to read about you tinkering with a Raspberry Pi or similar cheap device and trying to get it to do cool stuff (where “cool stuff” is left up to your discretion).

Well it just so happens that I already have a Raspberry Pi. I got it at PyCon US, I think three years ago, when they gave every single attendee a Pi for free. I thought it was super duper cool and I spent a whole afternoon tinkering in their Raspberry Pi lab and then I came home and put it in a drawer forever because I had no idea what to use it for.

At first I thought it would be cool to rig something that would download a random wad from idgames (like vectorpoem‘s WADINFO.TXT) and just launch it and let you play it. A teeny tiny portable Doom box.

Then I realized you’d still need a mouse and keyboard (well, at least a keyboard) to actually play, which is a little bit more cumbersome and detracts from the portability a bit.

But I remembered hearing about a Linux-only project that had managed to interface with the Wii U GamePad. Run ZDoom on a light wireless controller with gyros and everything? That sounds awesome.

So off I went.

Twitter recently lost its only black engineer in a management role.

The top-voted comment about this story on Hacker News begins thusly (emphasis mine):

I’ve been working for over 20 years in tech at 10+ different companies around the Valley, and I can count on 1 hand the number of direct coworkers that were black, and on 2 hands the number of coworkers that I indirectly worked with that were black.

I don’t believe this is due to any sort of racism, but rather due to the education system in general. Trying to solve the diversity issue at the hiring end, when the number of qualified candidates is so small, is not the right way to solve the problem. The only way you will hit higher-than-normal diversity numbers is to reduce hiring standards, which is wrong.

This is an interesting thing to say (and upvote) when the article itself said the exact opposite:

In the course of the meeting, [the VP of Engineering] suggests that we begin tracking the ethnicity of potential candidates in the pipeline to understand better where candidates are falling out. I agreed that this is an important metric to track and conveyed that the current data we had indicated that the problem is not just the pipeline. While ethnic and gender data early in the pipeline is incomplete, we do know that in 2013, 4.5% of CS graduates from the top 25 schools were African-American, and 6.5% were Hispanic/Latino.

A chart in the article indicates that in 2014, Twitter’s tech employee population was 1% black and 3% Hispanic.

Sketch bought a day of my time this month (and, er, last month) to work on adding “pull requests” to a git-backed wiki. I picked spline, because, uh, I wrote it.

Don’t use pickle. Don’t use pickle. Don’t use pickle.

The problems with Python’s pickle module are extensively documented (and repeated). It’s unsafe by default: untrusted pickles can execute arbitrary Python code. Its automatic, magical behavior shackles you to the internals of your classes in non-obvious ways. You can’t even easily tell which classes are baked forever into your pickles. Once a pickle breaks, figuring out why and where and how to fix it is an utter nightmare.

Don’t use pickle.

So we keep saying. But people keep using pickle. Because we don’t offer any real alternatives. Oops.

You can fix pickle, of course, by writing a bunch of __setstate__ and __reduce_ex__ methods, and maybe using the copyreg module that you didn’t know existed, and oops that didn’t work, and it’s trial and error figuring out which types you actually need to write this code for, and all you have to do is overlook one type and all your rigor was for nothing.

What about PyYAML? Oops, same problems: it’s dangerous by default, it shackles you to your class internals, it’s possible to be rigorous but hard to enforce it.

Okay, how about that thing Alex Gaynor told me to do at PyCon, where I write custom load and dump methods on my classes that just spit out JSON? Sure, you can do that. But if you want to serialize a nested object, then you have to manually call dump on it, and it has to not do the JSON dumping itself. There’s also the slight disadvantage that all the knowledge about what the data means is locked in your application, in code — if all you have to look at is the JSON itself, there’s no metadata besides “version”. You can’t even tell if your codebase can still load a document without, well, just trying to load it. We’re really talking about rolling ad-hoc data formats here, so I think that’s a shame.

But I have good news: I have solved all of your problems.

Last month @zandrmartin asked me on Twitter:

i would love to read a post about things you would recommend newish programmers learn, esp those coming from a php web background ⮹

not even tutorials as such, just like ‘you should learn about x and how to implement it effectively’ would be great ⮹

like when you were talking about bit masks a while back, i have no idea what those are or why you’d use them, but i want to ⮹

I’ve opined on this sort of thing briefly in various impermanent places, but somehow never tried to consolidate it at all. So here you go, some stream of consciousness on being more better at computers.

I spent a good chunk of the last four days installing an Internet web forum, which claims it can be up and running in 30 minutes.

I like to think I’m pretty alright at computers. So what went wrong here? Well let me tell you.

I’m assuming, if you are on the Internet and reading kind of a nerdy blog, that you know what Unicode is. At the very least, you have a very general understanding of it — maybe “it’s what gives us emoji”.

That’s about as far as most people’s understanding extends, in my experience, even among programmers. And that’s a tragedy, because Unicode has a lot of… ah, depth to it. Not to say that Unicode is a terrible disaster — more that human language is a terrible disaster, and anything with the lofty goals of representing all of it is going to have some wrinkles.

So here is a collection of curiosities I’ve encountered in dealing with Unicode that you generally only find out about through experience. Enjoy.

Also, I strongly recommend you install the Symbola font, which contains basic glyphs for a vast number of characters. They may not be pretty, but they’re better than seeing the infamous Unicode lego.

I love programming. It’s like playing with Lego — here are some blocks, see what you can build with them.

That sounds a bit less impressive now, but when I was a kid walking uphill both ways, I only had a very generic Lego set where all the pieces were cuboids. If I wanted to build a house with a sloped roof, well, that was too bad. I could cheat a little, though, by making several layers in a terrace pattern. It wasn’t actually sloped, but it did the job well enough by making creative use of the tools I had within the constraints I was given. You might call it a hack.

Self-identified hackers will often lament how “hack” now has two meanings and everyone assumes the wrong one. I think there’s really only one meaning, and the “break into computers” sense is a special case. It’s not like breaking into a system is magic, or done by running hack.exe; it’s just a creative use of the tools you have within the constraints you’re given. Like when the constraint is “your username is placed in a string of SQL” and you decide to place a couple quotation marks in your username.

So I’m always a little surprised when programmers don’t get security issues or how to defend against them, because to me, it requires exactly the same mindset as programming. And I suspect the problem is a quiet assumption most people tend to make: no one is that much of an asshole.

That’s not entirely unreasonable. Every stranger you pass on the street could be a hired assassin, but that’s fairly unlikely, and we have punishments to discourage that sort of thing. Ultimately we have to have some level of trust in other people in order to be around them at all.

And yet.

This Friday, June 12, will be my last day at Yelp.

I don’t intend to look for another tech job.

Or another job at all.

Ever.

As part of my experiment to monetize my personal brand, or however we’re describing this now, I have a milestone that lets a patron impose a blog topic of their choosing on me. What could possibly go wrong?

And so, this month, Russ brings us:

You should totally write about text editors.

I totally should. I mean, wait, no I shouldn’t. I haven’t seriously used a text editor other than Vim for years.

Thankfully this was a moderately vague request, so here’s what I’ve done: I’ve subjected myself to all these hip shiny text editors that I haven’t been bothering with and taken notes of my initial impressions. I only had a few hours to devote to each, so this won’t really be a fair comparison… but you know, life isn’t fair, so eat your peas and do your homework.

Wow! My Patreon experiment has been successful enough that I’m finally obliged to write one post per month, and this is the first such post. Let us celebrate with a post about something near and dear to everyone’s heart: fonts. Or rather, about fontconfig.

fontconfig is a pretty impressive piece of work. If you’re on Linux, it’s probably the thing that picks default fonts, handles Unicode fallback, and magically notices when new fonts are installed without having to restart anything. It’s invisible and great.

And unfortunately once in a great while it’s wrong. There is no common GUI for configuring fontconfig, so you’re stuck manually editing XML configuration files — for which the documentation is atrocious.

Lucky for you, and unlucky for me, I have twice now had to delve down this rabbit hole. Here is my story, that others may be saved from this madness.

Do you just hate Git? Are you perfectly happy with Mercurial (or, yikes, Subversion) but once a month you have to brave Git because everyone and their damn dog is now using GitHub? Are you vaguely aware that about half of all Git commands will probably delete all your work forever, but you don’t know which ones and you don’t want to spend three weeks poring over the documentation?

Good news! I wrote you this amazing Internet post. I hope I can mash just enough Git into your face that you will be less likely to set things on fire, and also less terrified that you might set things on fire. This should also be enough to make the Git documentation a little more comprehensible; it’s extremely thorough, but also overwhelming and nonsensical if you haven’t already read half of it.

I’m trying to keep this brief but also potentially useful to people who have never touched version control at all, so there are some 101 bits sprinkled throughout. Fear not! I don’t actually think Mercurial users have no idea what a patch is.

Creating a programming language is apparently all the rage these days, and it’s got me thinking about what I would really like to see in one. I’m starting to suspect the things I want are either impossible or mutually incompatible, so I’d better write them down and let smarter people tell me why I can’t have everything and also a pony.

I’m strongly influenced by my love of Python, my aversion to C and C++, my fascination with Rust, and the bits of Haskell I understand. I very recently read an overview of Nim, which is part of what got my juices flowing. Also I have a lot of fond memories of what Perl 6 could have been, so, fair warning.

This is a brain dump, not a linear narrative, so some of this might be mutually referential or causally reversed or even complete nonsense. Please pardon the dust.

What a fucking clusterfuck.

In case you were absent from the Internet the past two weeks, or for some reason don’t care about tech shenanigans, the following has happened:

• Six years ago: Brendan Eich, inventor of JavaScript and founding member of Mozilla, donated $1,000 to support Prop 8, the amendment to the California state constitution that sought to define marriage to exclude same-sex couples, and that apparently would have voided 18,000 existing same-sex marriages.

• Two years ago: Someone dug up and published a list of everyone who donated to Prop 8. The tech community noticed Eich, CTO at the time, was on the list. There was some furor, enough that Eich responded on his blog. Nothing else really happened, and it blew over.

• March 24: Eich was made CEO. People still remember that whole Prop 8 thing. All hell breaks loose.

• April 1: OKCupid adds a big splash page for Firefox users, asking them to maybe switch browsers in protest of Eich’s appointment. All hell escalates.

• April 1: Eich does an interview with CNet in which he says he’s the best man for the job. The interviewer asks him pointedly about Prop 8 several times, and Eich deflects. All hell escalates further.

• April 3: Eich steps down as CEO. All hell becomes sort of morose quietness.

• April 3: The world outside the tech community catches wind of all this. All hell breaks loose, again.

I’m pretty interested in that last part, but first:

Game Freak, you are kinda bad at interfaces. Don’t get me wrong; Pokémon X and Y are absolutely the best in the series and even just showing the party Pokémon from within the bag is nothing short of genius.

But goddamn how does nobody working there notice these other things, some of which have been around since Red and Blue.

(Pretty light spoilers since this is just standard Pokémon gameplay stuff.)

• When learning a new move and choosing which one to delete (in battle, anyway), there is no way to see the Pokémon’s stats. So if I’m choosing between a 60 power physical move and a 70 power special move, which is not uncommon… I can’t see if the Pokémon has better Attack or Special Attack. I often end up having to consult veekun to guess at the stats of the Pokémon that’s already out!

• Getting from a move to its description (and damage class, ahem) in battle is ridiculously tedious. You can, of course, hold L and choose a move to see its properties. But this is never mentioned anywhere in the game, I’ve had this actually use the move once or twice, and anyway it doesn’t work if you have L=A turned on (and why wouldn’t you? one-handed mode wooo). Meanwhile, the X and Y buttons do absolutely nothing.

• The bag has five pockets. Four of them are nicely organized.

  • Key items, of which there are probably a few dozen at most.
  • Berries, which can’t hold more than… 65, I think?
  • TMs, of which there are 108.
  • Medicine, which probably doesn’t exceed 50.

  That leaves the “other crap” pocket, which gets filled with the following:

  • 20 types of Pokéball
  • 18 Arceus plates
  • 18 type-boosting held items (Charcoal and the like)
  • 18 type-boosting gems
  • A dozen evolution items
  • At least 17 species-specific held items (Stick, etc.)
  • At least two dozen general-purpose held items
  • Some untold number of (ahem) the new item type introduced in XY
  • Fossils, shards, loot, Heart Scales, Honey, mail, and god knows what else.

  That’s over 200 items in one pocket, twice the size of the next-biggest.

• The miscellaneous pocket of the bag no longer shows item grouping icons (like a Pokéball), which at least helped separate sections in BW.

• Neither the bag nor the in-battle item menu show item icons any more. The held item on a party or boxed Pokémon doesn’t, either.

• The “item” submenu on a party Pokémon offers to let me take or swap an item even from a Pokémon that doesn’t have one. It also doesn’t tell me what the item is before I take it, which is annoying when I’m trying to find where I left something like Amulet Coin or Lucky Egg.

• “Deposit” and “withdraw” on the Pokémon box system are useless. Using one or the other is slightly faster than going through “organize”, I guess, but anyone who runs with a full party of 6 (i.e. almost everyone) is going to want to do both together. It wouldn’t matter except that this is the one thing stopping me from merely mashing A to get to my boxes.

• For some reason trading has two ways to show a Pokémon: press A, or tap a weird icon or something at the bottom I forget. The next step from either of these to actually offering a trade is a little different, and I can’t figure out why.

• When viewing a party Pokémon, I can’t switch directly to my Pokédex and read a description of it. I have to remember its dex number and look it up manually.

• There is no list of all moves or abilities I’ve seen. :( Shouldn’t the Pokédex track what abilities Pokémon can have?

• The zoomed-out box grid view on the PC should really color each box icon according to its wallpaper. With 24 boxes this view is not very useful.

• There are two (and a half…?) completely separate and distinct interfaces for deleting a move. There are at least two completely separate and distinct interfaces for seeing a Pokémon’s summary. This is a little silly, but it’s been the case for a while.

• The summary view when perusing boxed Pokémon neither mentions its nature nor indicates which stats are raised/lowered.

• Learning a new move and forgetting an old one involves some three separate prompts. If you change your mind about learning the new move, the button to abort is helpfully labeled “QUIT” and just restarts the sequence of prompts over—more than once have I just mashed A and accidentally agreed to learn it again. Why not just say “here’s a fifth move, pick which one to lose”?

  (That’s an annoyance in a lot of games, particularly RPGs: the button I mash to advance through dialogue is the same button I use to agree to choices offered in dialog. My 3DS has like a thousand buttons I’m pretty sure you can spare a second one.)

• There are several places in the game where a list of items is intended to be scrolled by sliding on the touch screen, a la most phone interfaces (e.g. the bag, puffs in Amie), but either I’m a clumsy buffoon or something is not calibrated well because I very frequently end up tapping an item instead.

• I miss the Habitat List. Please stop adding neat UI features in doomed branch games and not bothering to port them to trunk of the next generation. :(

So hey all those people with uncles and cousins that work at Game Freak who’ve come out of the woodwork since XY was announced: if you could pass this along that would be fantastic.

(This article has been translated into Russian by Everycloudtech—thanks!)

I have a hilariously unfinished Python module I work on from time to time named sanpera. It’s an imaging library for Python, with the vain hope that it might replace PIL someday. But this isn’t about sanpera.

sanpera happens to be powered by ImageMagick. I distinguish this from being an “ImageMagick wrapper”, as it explicitly has nothing resembling the ImageMagick API, because said API is insane. But this isn’t about ImageMagick, either.

Using ImageMagick requires binding Python to C, and that’s what this is about. There are several ways to use C libraries from Python:

• Writing an extension module means the Python API is defined in C, so the library is used exactly as it was intended: with C code. Unfortunately this requires writing a lot of C, as well as a lot of careful Python refcounting. My C is passable, but I’ve done far more reading it than writing it, so this is not an appealing option.

• ctypes is a standard-library module that can load shared libraries and call functions from them without the use of a C compiler or any new C code. Convenient, especially if you hate dependencies (in which case why are you binding to C?), but all the ctypes-powered code I’ve read has been tedious and fiddly and ugly.

• Cython, a spiritual port/evolution/fork/something of the older Pyrex, is a language similar to Python that translates to C and then compiles into an extension module. Cython code can define Python classes and functions, but also call C functions and perform other C operations directly. Code with C semantics is translated fairly directly to C; code with Python semantics is translated to appropriate use of the CPython API; and Cython fills in all the bits to translate between the two.

I went with Cython because it looked interesting, it seemed to reduce the number of translation layers I’d have to care about, and it would even let me write hot loops (this is an imaging library) in C without actually writing any C. Plus, since it’s not actually Python code, it can compile to both Python 2 and Python 3 extension modules with very little effort on my part.

Here’s what Cython looks like:

This is a story of how nothing in this story is my fault.

I’ve got a coworker who’s super into Clojure, a Lisp-like that runs on the JVM. In particular he’s super into how it’s got notions of mutability (and, thus, immutability) all throughout.

More than once he’s lamented that Python lacks a frozendict—a dictionary type that can’t be changed. Dictionaries tend to crop up a lot in Python, and in a very large codebase, it’s very easy to end up with this scenario:

1. Some function somewhere generates a dict that’s only used by one caller. The interface is obvious since the keys and values are created right there in one place. No problem.
2. Other code comes along, notices this handy function, and starts using its return value. Some of this code may pass the ad-hoc dict up to callers, too.
3. Some of that other code needs more things added to the dict, but computing the extra data is expensive, so arguments are added to the function that optionally turn on certain keys.
4. Some code needs even more things added to the dict that are outside the purview of the original function, so they add helper functions that take the ad-hoc dict and add more things to it.
5. Since this has all now happened multiple times throughout your codebase, someone addresses the problem by writing adapter code that infers the original dict from some other object describing its structure, thus saving everyone from writing all these functions that return dicts.

A lovely spaghetti dinner. It’s now nigh impossible to trace what the dict contains or where half of it came from.

Returning an object in the first place would have avoided much of this, but when you’re sitting at step 1, that seems like a lot of effort just to return half a dozen things from a function you wrote to another function you wrote. Swapping out dict() for frozendict() is easy.

My opinion on frozendict had never grown stronger than “I guess that would be cool”, so I never sat down and wrote the class, and there must be enough subtleties that nobody else at Yelp has either.

Then today, PEP 416 came to my attention. This PEP proposed adding a frozendict type, but was rejected last year as being largely unnecessary. What’s interesting about it is that the rejection ends with almost a footnote suggesting that perhaps dictproxy ought to be exposed to Python-land, instead. And indeed this was done, and it exists in Python 3.3.