Kriston and I built Lady Ada’s Game of Life kit over at HacDC a while back, and it roused the interest of a few folks. Sommer asked that I be on the lookout for another kit that might be a good candidate for learning to solder. I eventually suggested another Lady Ada kit: the MintyBoost, a simple circuit that lets you top up your USB gadgets’ batteries through the use of a pair of AAs. By the time I got my act together, six(!) people had expressed interest in building the kit, pushing supplies of my tools (somewhat scarce) and electronics knowledge (decidedly meager) to their limits.

But it went surprisingly well! Everyone’s kit worked immediately; in fact, they even seemed to work with the iPhone 3GSes that were on hand — something that the MintyBoost can’t necessarily be relied upon to do.

You can find more photos here.

I think that my favorite part was watching everyone mess around with the Dremel. That’s some advanced nerdery for you.

I think that folks had fun; we might do this again, perhaps with more of an Arduino bent. If you want in, let me know.

I made my first EAGLE schematic!  I’m sure it’s horribly broken, but I’m still feeling pretty good about it.

If you’re interested in getting the source file, head over to the post on Sunlight Labs.  And if you do, please be gentle.  Advice on what I’ve gotten wrong would be welcome, though.

ALSO: Thanks to the diligent outreach efforts of Sunlight’s own Nicko Margolies, the original post has now been picked up by Hack-a-Day and MAKE. Neat! And better still, educational: I’ve already had a number of revisions suggested to me by the Hack-a-Day commenters.

ALSO ALSO: Engadget, too! Though I’ve gotta say: man are the commenters there morons.

The wages of internet success, I suppose.

AAAAAND: Gizmodo. That’ll just about do it, I think. Their commenters are mean. Which I like. I want the apparent EE to offer some more information, though.  I think he’s at least partly wrong.

Earlier this week my old boss, JP, sent me a note saying that the full-text RSS script I’d written was being shut down.  EchoDitto was nice enough to continue hosting it on their servers, but it had been consuming an increasing share of resources, and finally it needed to be killed.  Sad.

Then, horrifying.  JP had mentioned it idly, but until I saw the Google Alert for this I didn’t quite realize what had been going on.  The self-described black-hat search engine optimization crowd — the folks who assemble sites peppered with ads that are designed to attract search engine traffic, aka “link farms” — had been using my script to steal other people’s content and republish it on their own sites.  Using this sort of genuine content helped them snag traffic more effectively than they could with the gibberish that you sometimes see in spam, so they were sorry to see the script go.

Well, I appreciate the attention, but it’s not exactly what I had in mind for my work when I released it.  So I explained the situation, and bid them adieu:

I was at least somewhat aware of this danger, and consequently didn’t open-source the code (thank goodness).  It was still a bit of a shock to see the reality of what I’d wrought, though.

I still believe what I wrote in that initial post: it’s pointless for online publishers to try to control how their readers consume content.  If you wish to publish digitally, you need to accept the realities that come with doing so.  Pretending otherwise is just going to inconvenience your readers and slow your business’s necessary evolution.  Prolonging that process seems likely to increase the painfulness of the process, not eliminate it.

For those curious, the algorithm was exactly what many in comments had guessed.  I used some regular expressions to build a hierarchical structure representing all the <div> and <td> elements in each page associated with an RSS item.  These tags are the ones most often used to provide a page’s layout, and the full text of an entry can usually be found in a single instance of such a container.  I then traversed this structure, looking for the text excerpt from the original RSS item.  Once found, the rest of that container’s contents could be pulled out.  It’s a simple idea, though the realities of HTML — and the difficulty of preserving byte offsets between a sanitized working copy and the original — made the actual implementation require quite a bit more cleverness (and caching) than it may sound like.

The result worked pretty well. Still, there were a few problems with the approach.  For one thing, comments were frequently included in the same container as the main entry.  For another, the script would fail if the RSS entry text was a summary of the item rather than an excerpt. I think that both of these are surmountable problems: a better approach would examine the “textiness” of each container using a variety of scoring metrics.  Something similar could be used for detecting the start of comments (which tend to be peppered with timestamps, quotations of the original text, and occur after a big <h[1-6]> containing the word “comment”).  I took a stab at a new, Pythonic implementation using Adrian Holovaty’s templatemaker and a few other tools, but a lack of immediate success (and much higher computational demands than the original script) made the project fall by the wayside.  Now that I know how it might be used, I’m even less likely to pick it back up.

But I’d still love to see my algorithm adapted by the people who make RSS readers, and would be happy to talk to any interested and qualified parties about making that happen.  Those SEO morons don’t appear to be particularly technically proficient — I’m not too worried about them managing to steal content via a client-side app (though certainly some thought should be given to the matter before giving the store away, say via Applescript).

Tim points out that Vint Cerf’s advocacy for network neutrality doesn’t mean that every venerable internet expert agrees with it.  True!  Google for “Dave Farber” or “Bob Kahn” and “net neutrality” and you’ll find plenty of articles… from 2007.

I think this debate has stagnated.  People are continuing to act as if the FCC’s plan is a complete unknown. This enables the lazy “evil corporation/incompetent government” frame that we all know and love. At one point this was at least semi-appropriate.  People — including those like Cerf, Farber and Kahn — made guesses about what was likely to happen, largely based on their own position along the regulation/markets ideological spectrum. “The government is trying to outlaw QoS!” “The ISPs are going to charge for Google!”  Both of these positions were hyperbolic.

But look, there’s a difference now: the FCC has announced its NN principles, to thundering… well, there hasn’t been much reaction, actually.  Everyone’s just pretending that nothing happened and sticking to their guns.  “The principles are too vague!” they cry, even though those principles say that allowances will be made for necessary network management, that throttling heavy users will be okay, that malware can be filtered, that copyright law can be enforced, and that managed services can be developed and sold.  That wipes out a significant swath of the ISPs’ objections.

Yet no one seems to have noticed. Here’s Farber responding to the new principles — he doesn’t particularly object to them, he just thinks they’re too vague and that they might make ISPs wary of trying to innovate (as if the research costs to network management are so great that the potential for work product to be shot down makes it impractical; please, this is software).  He also says they’ll lead to a bunch of lawsuits, though he simultaneously says we should instead rely on the DOJ and FTC to handle this (in a rule-free environment!), which makes just about no sense to me.

I digress.  Look: the FCC has put some of its cards on the table.  Not all!  Rulemaking still needs to occur (if it didn’t, opponents would be even more outraged).  But the agency has signaled that it’s not actually going to do the incredibly idiotic things that neutrality opponents claimed it would. Those people need to acknowledge this fact, and come up with a clarified set of objections. What is it, exactly, that the FCC’s stated plans are going to stop you from doing? Right now I’m just seeing a bunch of now-irrelevant FUD; some talk about how great a monopolistic IPTV market would be; and some hand-waving about “new business models”, which I take to be a synonym for “rent-seeking”.  It’s time to admit that the FCC isn’t the one being vague.

It’s hardly worth noting, but this is a lousy editorial.  A couple of points, one of which I buried in the last post about net neutrality:

• Comcast’s reversal of its decision to fiddle with Bittorrent seems to be getting bandied about as an example of the market taking care of things. People should be less sanguine: it was not market pressure from clients but rather PR pressure from activists that forced Comcast to back down.  And if the net neutrality issue hadn’t been looming at the time, the outcome could easily have been different.  Mustering that level of coordinated outrage is not a sustainable model for policing ISP behavior.  Other ISPs offer evidence of gloomier scenarios: in Canada, Rogers has been throttling Bittorrent — and, in fact, all encrypted traffic, since it can’t distinguish encrypted BT from encrypted anything-else — since at least 2005.  Similar difficulties face customers of some Australian ISPs.  And there are other policies that have been in place so long that people have stopped complaining: unless something has changed, I’m still prohibited from running a server for personal use off my home connection (in principle a server of any type; in practice, port 80 is the only one that will be caught). And it was years — years! — before Comcast disclosed or even acknowledged the existence of its secret bandwidth caps.  Neutrality opponents are mistaking a single victorious battle for the absence of a war.
• The Post piece refers to the broadband market as “a vibrant and well-functioning marketplace”. In truth, it’s stagnated: in North America, prices remain steady — at a rate above what much of the developed world pays — while ongoing improvements in speed show little hope for of catching up with the networks of countries significantly poorer than the US.  If the Post wants to argue that net neutrality will make the situation even worse, fine. But they don’t even seem to realize that by global standards our domestic broadband marketplace is an underperformer.

Surely any Redskins fan will agree that foremost among Dan Snyder’s sins is his selection of a broadcast outlet that only provides its audio stream in Flash.

See, I’m going to be heading back from Philly this Sunday, and many Sundays besides, and I’d rather not miss the whole goddamn football season.  If WTEM provided an MP3 stream this would be no problem — the FStream iPhone app would let me listen to the game on stretches of I-95 well beyond the anemic reach of 980 AM.

That’s not the case, though.  WTEM uses a Flash-based streaming system, apparently through a vendor called streamtheworld.com.  That’s fine for the web browser, but the iPhone’s too stupid to deal with Flash.

So I set out to provide an alternate stream.  Amazon offers on-demand virtual servers through its EC2 service, and my first thought was to fire one of those up for each game.  At ten cents an hour it would be affordable, and I could use the more-competent server to transcode the Flash stream into a more iPhone-compatible MP3 stream.

I got far enough along this path that I might as well share my work.  Here’s a quick recipe for setting up a VLC-transcoding-capable server on EC2:

1. Launch the following Ubuntu AMI: ami-ed46a784
2. Make sure that you’ve opened port 8080 in the relevant security settings.
3. Execute these commands:
   

   apt-get -y update
   apt-get -y install libmp3lame-dev
   apt-get -y install ffmpeg libavcodec-unstripped-52
   apt-get -y install vlc vlc-plugin-esd mozilla-plugin-vlc

4. Execute something similar to, but not quite this:
   

   sudo -u nobody vlc -vvv “http://208.80.52.80/WTEMAM” –sout ‘#transcode{acodec=mp3,ab=64}:standard{access=http,mux=asf,dst=}’

This is close.  It’s really close.  If you run this and then load up VLC — the amazingly useful media player — and connect to the AMI on port 8080, you’ll get a 64 kbps transcoded MP3 stream of the WTEM flash audio stream.  It works well!

Unfortunately, I don’t know the specific incantation that makes this MP3 stream compatible with FStream on the iPhone.  I tried various playlist formats.  I tried various audio formats.  I tried various container formats.  There doesn’t seem to be any decent documentation on the internet telling me what to do.  It just says it’s connecting, forever.

But while searching for a solution I discovered something interesting: there’s an iPhone version of VLC!  True, it’s 13 megs, and yeah, it’s only available on jailbroken phones (via the Cydia package manager).  But it works great!  I pointed it at http://208.80.52.80/WTEMAM and it started spitting out incomprehensible sports radio babble almost immediately.  Admittedly, that was over wifi.  But I see no reason to expect it to perform any worse over 3G than an equivalent MP3 stream would.  I’ll be giving it a shot on Sunday during the Giants game. And while I’m sorry to not be providing an MP3 stream for other fans, I’m glad to offer another reason to jailbreak your phone.

Tim wades in to the debate about robots replacing low-skilled workers. I don’t think I made my case as strongly as I could have, and consequently I can’t resist responding, if only briefly.

1. I didn’t react to the “cost of materials and energy” argument as forcefully as I should have. I really, really, really don’t think this is going to be an issue. Machines are much more efficient than humans — that’s sort of the whole point. It’s not at all a subtle difference, but rather a massive leap (they don’t call the first sweeping wave of automation a revolution for nothing). The price of technology continues to fall. And the energy costs we’re talking about are slight: keeping a Roomba charged takes about a third as much power as illuminating the lamp on your nightstand — and that’s if the lamp is using a compact fluorescent bulb.
2. Tim’s discussion of our hardwired preference for other humans is an interesting one. But I’m not sure this is a problem that will be as resistant to engineering as he thinks. Consider Paro, the robot seal. By all accounts he works pretty well at soothing nursing home patients. Tim’s probably right that people would prefer — and pay for — a human R.N. rather than Rosie the Robot in nursing scrubs. But if a suite of relatively simple assistive technologies is cheaper than either — partially automated showers, biomonitors, pill reminders and the like — they’ll probably settle for that.

Human labor remaining competitive with robots seems to assume that one of two things will happen.

First, wages for unskilled labor could be pushed even further down. Presumably this is supposed to happen as average income continues to rise. I’m not sure how this can be considered feasible, either practically or ethically — we’re basically talking about reinventing the serf.

Second, robots could become dramatically less economical due to some sort of resource or energy shock. But given robotic efficiency (and the reality of their still-falling prices), it’s hard for me to envision this happening except in the sort of catastrophic scenario in which maintaining human employment levels would be the least of our worries.

Although it is arguably as much a business innovation as a technological one, there’s no doubt that the iPhone has radically altered the mobile device landscape, and, in the process, profoundly changed the way that Americans think about sitting on the toilet.

In fact, its uses extend beyond this crass but foremost example.  You’ve also got your standing-in-line downtime, the crucial waiting-for-coffee-to-brew period, and of course miscellaneous hours spent idly fiddling while in meetings. If I’m not mistaken, the Bureau of Labor Statistics is on record as saying that high scores from Sudoku games played on mobile devices account for most of the productivity gains of the past decade.

Personally, I like to use the device to catch up on my online reading.  But this crucial electronic-fucking-around is slowed by the need to enter URLs.  If you’re like me, typically you’ll have some long-form article open in a neglected browser tab: say, 2000 words from Sasha Frere-Jones on why your implicit acceptance of the latest top 40 hit means you’re more — and less! — racist than you thought.  It’s something I mean to read, but which the demands of the workday have made me put off.  As I leave my desk to pursue less productive activities (food acquisition; interaction with humans), I frequently find myself wishing I had that article loaded on my handset.  But the only way to move it there without tedious typing is to email myself the link, then check my email, then click on the link.  If only there was an almost perfectly equivalent but slightly faster way!

Well, you’re/I’m in luck!  I twittered this ridiculous first-world complaint the other day, and @tbridge and @jroo helpfully pointed me toward Prowl, a $2.99 iPhone app that takes the 3.0 firmware’s push notification capabilities and wraps them in a simple API.

This opens the door to creating a bookmarklet that grabs your browser’s current URL and pushes it to your phone.  The phone will buzz, you’ll click the “view” button, and you can then follow the link.  Easy!  Here’s the bookmarklet.  Just follow these steps (which, I should note, have only been tested in Firefox 3.5):

1. Buy Prowl.  Open it on your iPhone — you’ll need to register with its parent site and give the app permission to display notifications.
2. Using your newly-created credentials, log into the Prowl site. Go to the settings tab and create an API key by clicking the appropriate button.
3. Paste the API key into the form field below. Click the “Create Bookmarklet” button. NOTE: the customization of the bookmarklet is done in client-side Javascript, entirely within your browser — don’t worry, you won’t be sending me your API key.
4. Drag the newly-created button into your browser’s quicklaunch bar.

Simple!  Now when you click that bookmarklet a new window will be briefly opened.  It’ll submit a request to the Prowl site that contains the current URL that you’re looking at.  Shortly thereafter you should get a message on your phone with the link.

As for the new window: I admit, it’s inelegant to spawn a popup.  But the Prowl API only accepts POSTed requests, which, barring a sudden and deeply unwise decision on their part to host third-party scripts, rules out a more seamless AJAX solution.  The popup works well enough, although on especially slow connections the window may wind up closed before the request goes through.  A better solution would involve a timer that checks the spawned window for when its location.href property suddenly becomes inaccessible due to cross-domain security policies (indicating that the page it contains is now being served by the Prowl domain).  But my first crack at that didn’t work, so for now you guys are stuck with this.

Nextbus stops! Served by Google App Engine! Displayed on the iPhone!

Of course:

• The app crashes as soon as you try to do anything after the map loads.
• There are way too many pins. And I don’t want them to be pins anyway. And the stops aren’t likely to change often so I should actually move this operation off the network and onto the iPhone, which might mean reimplementing geohash, which I don’t yet fully understand (but know to be neat!).
• I haven’t written code to query Nextbus for predictions. This will be fairly easy, but exactly how I want to handle caching is an open question.

Still: wheels, motion, etc.

P.S. Objective C suuuuuucks

It seems like the Iranian cyberactivist movement has marked the entrance of the term DDoS into our culture’s shared vocabulary. There’s a big DDoS attack going on right now, in fact: the Post has coverage (they’re among the targets), and I heard a piece about it this morning on NPR. Governments are among the victims. People are starting to notice that the agencies responsible for dealing with this stuff don’t have the power to do much more than write whitepapers. Governments are going to start doing things about it.

I think this will likely take two forms:

• Legislated standards for ISPs. In particular, I imagine we’ll see ports for IRC and SMTP, if not all non-http ports, move to a default-closed state. I’m not sure how you’ll opt into opening them, but SMS verification of an online request seems to be increasing in popularity (my bank does this; Google does, too, when you start to use their App Engine service). There’ll also likely be incentives or mandates put in place for things like server-side antivirus scanning of email attachments. The details matter quite a lot — the possibility of a corporate power-grab that constrains citizens’ ability to interact securely is real — but I think such attacks can be fought off, and the result will probably be a good thing on the whole.
• These ISP standards could conceivably get written into trade agreements, at which point a more politically interesting possibility will arise: the imposition of economic or network sanctions against nations judged to have an out-of-control internet. Is there a damaging DoS attack coming from South Korea? Is that country’s government judged not to have adequate standards in place to fight or prevent it? Cut ‘em off the network, or throttle their traffic. There’s a constituency for this: the WIPO people would be only too happy to have this capability.
  
  
  This opens up other interesting possibilities. For one thing, the mutually destructive nature of sanctions would become immediately clear to internet users in a way that isn’t always obvious in the comparatively sluggish realm of trade. And for another, online false flag operations would become a more serious concern. A hacker group could conceivably blackmail a small, cloutless nation under the threat of eliciting an international network crackdown.

It’ll be interesting (and no doubt horrifying) to see deep packet inspection debated on the House floor. I think we’re headed that way, though.

Oh yeah: let me plug this article, which is the more entertaining narrative about a DoS attack that I’ve come across.