Random bits from a random nerd

Odd Packet Loss at End of Year

I was debugging a slow network and found a couple of odd problems that I thought worth sharing. Check this out - RRDTool graph from my pfSense firewall, showing latency and packet loss between the firewall and the Motorola SB6141 modem:

Packet loss graph

This is a pecular graph, designed to show link quality. The upper half is latency, in milliseconds, and the lower portion is packet loss, in percent. Note that packet loss goes from zero to 50% more or less overnight.


Next, notice that it happened right around Jan 1, 2014. At a guess, there’s date-related bug in the Motorola SB6141; a power cycle completely fixed the problem:

No more packet loss

That sorted, the network remained slow. I found this awesome Curl feature to debug things, and DNS was taking 5-6 seconds per lookup. A dive into /var/log/daemon.log produced lots of DNS errors of this form:

Error (network unreachable) resolving 'pdns5.ultradns.info/AAAA/IN'

Which I found on this post. All you have to do is disable IPv6 support in Bind9; the underlying problem is that DNS is trying to resolve IPv6 addresses when the link doesn’t support them. (I’m IPv6-capable here but have it disabled for now.)

So adding “-4” in the /etc/default/bind9 file under options fixed that.

And it was still slow, now because data took 6 seconds to return:

before fix

A pfSense reboot fixed that:

after fix

I’m assuming it’s a bug in pfSense, perhaps a memory leak or something related to the firewall rules. Very odd. There was minimal CPU and memory in use, hard to diagnose.

One last useful trick: I learned here that you can regenerate the root DNS file with a single command:

dig +bufsize=1200 +norec NS . @a.root-servers.net > named.root

Good tools to share. That curl bit for diagnosing HTTP is frickin’ awesome.

New SSL Certs Here and Www

My one-year certs were expiring, as expected GoDaddy jacks the price up for renewals. The original cost was $7 per cert for a year, upon renewal they are $69. Each.

Umm, hell no.

A quick search later I found this roundup of providers and off to Namecheap.com I went.

They have a nice website, I chose 2-year Comodo ‘Positive SSL’ certs ($7.95 each) for here and www.phfactor.net. Total cost $35.80, and the signup and install process was painless. And hey! Now I have two years before I have this to redo.


Local Android Group Presentation on Meltdown

I’ve been a slacker on posting here, a quick update on Meltdown. I’ve been coding on the next version, and have a working ContentProvider data store with Fragments UI in development. Should be faster, use less memory and start up much faster!

I volunteeered to talk about Meltdown to our local Android group, and it came out pretty well. Slides, links and details are here for your perusal and amusement.

Modaco Switch on the HTC One

HTC One product image

So I bought the developer edition HTC One Android phone a while ago, thought I should share some of what I’ve learned. The AnandTech review was the tipping point; nice review of an amazing phone.

Before you buy one, though, read this first:

Here's the problem: the Developer Edition has HSPA/WCDMA 850/1900/2100 and LTE  700/850/AWS/1900. If you don't know what that means but you're still cheering for unlocked phones, you need to get educated fast, because this is the problem with unlocked phones in the U.S.
This phone will only work properly on AT&T.
On T-Mobile, it will get 3G in some cities, but not in others; it'll be stuck on 2G in many places. It might get LTE in the future, but we're not sure. Ditto for Simple Mobile, Ready SIM, Straight Talk, and any other carrier that uses T-Mobile's network.

Super Short Review

Great phone. Good battery, excellent camera, excellent display, no regrets after three months. I tried T-Mobile prepaid, Solavei, now using net10 prepaid plan at appx $50/month. Net10 is an AT&T MVNO, so coverage is decent, not as good as Verizon. I do not have LTE, just HSPA and HSPA+; that suffices for me.

Unlike my Samsung Galaxy S3, audio quality on the HTC is excellent; as good as my iPhones and/or iPods. This has been a big annoyance for me, the S3 is a decent device but music sounds much worse, so even with Beats disabled the HTC sounds excellent on speakers or line-out. WIN.

Overall: Recommended.

The AT&T version is an alternative; they also have the 64G version and do have LTE. It’s locked, though.

Unlocked bootloader

One of the big appeals for me was the combination of great hardware with no-hacks-required reprogramming. I’ve less time, so a phone ready to hack was worth extra. (64GB was also a big selling point, as I like to carry lots of music with me when I travel.)

So what can you do with said bootloader? Replace the entire operating system!

I’ve taken my time in doing so. HTC has some custom apps I’d hate to lose - the improved audio (‘Beats’, as far as I can tell a custom DSP filter) and the camera app. The camera on this thing is excellent, especially for the indoor uses that predominate for me. So I was, and am, in no hurry.

There are multiple ROMs that will work on this - see this page for a listing, and be prepared to spend a few hours reading. The tradeoffs are complex, and the ROMs updated frequently. HTC has a nice page with a dictionary of terms and procedures that’s also well worth reading.

The stock Android version of the hardware, “Google Experience”, came out after I’d bought, or I might have bought that instead. Ahh well. If you just want stock Android, that’s where I’d start.

Ground rules and choices - root and S-OFF

I decided on MoDaCo Switch, an insanely clever ROM that allows the best of both worlds: You can toggle between stock Android and the HTC version. It self-updates over the air (“OTA updates”) as well.

As a long-time Unix geek, I was very cautious of anything requiring root or disabling device security. Mobile security is bad enough already, I don’t want to add more risk than necessary. I’m also looking for reliability, since this is my primary phone.

Installing MoDaCo Switch requires another app, a ROM assistant. The best options seem to be ClockworkMod and TWRP. I chose TWRP based on the MoDaCo instructions.

I’ve already got the ADK installed for Meltdown, so I’ve got the various tools ready to go. ADB, fastboot and the like.

I did not mess with S-ON/S-OFF. This unlocks lower levels of the device/radio firmware, and since it’s not required I see no reason to mess with it. Nor did I root the phone; same reasoning.

The steps

  1. Download SWITCH.Beta1.zip, SWITCH.Beta10.zip and SWITCH.S-ON.zip from here
  2. Download TWRP from here
  3. On the phone, install the TWRP manager from google play
  4. On the computer, run

    adb push SWITCH.Beta1.zip /sdcard/

    adb push SWITCH.Beta10.zip /sdcard/

    adb push SWITCH.S-ON.zip /sdcard/

  5. Unplug the phone and power down

  6. Hold volume-down and boot
  7. Select Fastboot and press power to enable
  8. Connect USB - display should change to ‘FASTBOOT USB’
  9. On the computer, run fastboot flash recovery openrecovery-twrp-
  10. Unplug the phone, select Recovery and press power to boot into recovery
  11. The phone should boot into TWRP!
  12. From TWRP, select and install the zipfiles, all menu driven.
  13. Once the phone boots, you should have a new app, ‘MoDaCo Switch’ that will switch modes.
  14. Profit!

Current status

I just got this installed today, and the browser tab collection was large, so this is just a dump of tabs and some notes. Complex process, more than one way to do it, and maybe of use to someone else as well. I just did my first switch from HTC mode to Google mode, and hey! It worked. App data preserved, too, and MoDaCo did a self-update to beta 12, all working as I had hoped. I’ll post again if needed.

Switched From DSL to Cable

I’ve been delighted with my current firewall, a Netgate m1n1wall 2d13 running pfSense. I’m happy to recommend both the hardware and software. It even comes in red:

firewall hardware

I’ve been using it for a year or so I think, on my AT&T U-Verse DSL connection. It’s been reliable and fast, able to handle the 24/3 (mbit) connection even with lots of devices and activity.

Then yesterday, I took advantage of a local promtion. Time Warner Cable is running a one-year discount on connectivity, so I can get about twice the speed for the same price. Specifically, $80/month for 50/5, plus $20 for VOIP.

So the living room stack is now cleaner, smaller and faster:

cabinet stack

(Ignore the audio mixer in the middle, that’s another post)

I should note that I went with the Wirecutter’s recommendation on cable modems and bought the Motorola Surfboard SB6141 via their affiliate link. Seems perfect, though I’m mildly annoyed we need a second box for VOIP, should’ve seen that coming.

I used a Debian ISO BitTorrent to test it out, as speedtest was giving me obviously-bogus 20-ish megabit numbers. A Debian torrent will beat the shit out of pretty much any Internet connection:

transfer graph

Looks like TWC provisioned us with almost sixty megabits of downlink. Damn.

Low ping too. Frickin’ awesome ping, in fact. I should get back into FPS games just so I can be a LPB again:

pinging google

As I did the BitTorrent, I kept an eye on the firewall as well. I was using 108 peers, at 58Mbits, but the firewall got seriously laggy. Looking at pfTop, it appears that just handling the interrupts took 70% of the CPU, so the box was running flat-out. This graph underestimates it a bit, but since I forgot to grab a screenshot it’ll do for now. That spike Monday is the BitTorrent, and note the lighter color shows time spent handling interrupts. We were slammed.

cpu graph

Looking at the pfSense docs:

If you require less than 10 Mbps of throughput, you can get by with the minimum requirements. 
For higher throughput requirements we recommend following these guidelines, 
based on our extensive testing and deployment experience. These guidelines offer a bit of 
breathing room because you never want to run your hardware to its full capacity. 
10-20 Mbps - No less than 266 MHz CPU
21-50 Mbps - No less than 500 MHz CPU
51-200 Mbps - No less than 1.0 GHz CPU

And my m1n1wall? 500Mhz AMD Geode LX800. Specified as max throughput of 85Mbits, and that’s probably with an optimized, zero-rule configuration.

Right, I need more firewall MIPS to handle this connection.

Current options for More pfSense Love

So what are my options? I want pfSense, gotta be fanless with no moving parts so it’s silent in operation. Low power a major consideration too. Need a GHz-class processor. Gigabit ethernet preferred, but not required.

The Netgate FW-525B looks good. Atom 1.8 GHz CPU, 2GB of memory, four gigabit ports, $400:

Netgate FW-525B

I’m a bit cautious about Atom chips though, having just upgraded from them on the server, so let’s see what else Netgate has available. How about the FW-754? Dual core Atom now, still 1.8GHz but six gig-ethernet ports. Nice form factor, too. $583, ouch.

Netgate FW-7541

Oddly, both ship with 60 watt power supplies, which is much higher than I’d like. The server is 22W max, and it seems stupid to use more on the firewall. Also, the 525 page page says

Regular operation on this fanless system is fairly high in temperature.

which sucks. Neither is a clear winner, so if you’ve an idea please leave a comment / email / Tweet. I can live with the m1n1wall for now, but I’m definitely looking to upgrade.

Anyone need a well-treated m1n1wall?

Notes for others

  • TimeWarner cable is, at least here, faster than advertised. You should notice faster web page serving, since the uplink went from 3 to 5 megabits.
  • So far, I have a fixed IP with all ports unblocked. No problems with serving HTTP, HTTPS, SMTP and SSH.
  • I use Dyn for DNS, and last week I reduced the TTL on my DNS records to 60 seconds so that when I switched it’d take right away. Highly recommended you do the same - services were back within a minute of the connection coming up. If TWC starts changing my IP, pfSense has hooks to update Dyn as it happens; very sweet.
  • The Motorola modem seems good. Not paying rent for a TWC modem is a big win.
  • pfSense remains the best firewall in my opinion.
  • Gawd but I’d love to get Verizon FIOS or Google Fiber here in San Diego!

Most importantly, firewalls require a fast hardware platform. That generic Linksys or vendor-provided box is very likely a reason why your connection is slower than it could be. I recommend the m1n1wall for most connections up to 40Mbit, and certainly pfSense is a wonderful system for home or business use.

An Aphorism in Memoriam

We lost a good friend Sunday night.

Some Richardson:

To feel an end is to discover that there had
been a beginning.  A parenthesis closes that
we hadn't realized was open).

Ghost Blogging

Forgot to mention this, but close after losing all the blog images in the server migration, I found and funded Ghost on Kickstarter. It looks modern and awesome, so rather than spending more time on Octopress I’ll see how Ghost does. Supposed to have Jekyll/Octopress import out-of-the-box.

As a side note, I tried to find images on archive.org but only found a measly six. Ahh well.

Really hoping that Ghost makes blogging easier, as the Rake/Octopress process is a real deterrent to casual writing.

WWDC 2013

WWDC foyer

Yep, I got to go to WWDC 2013, Apple’s sold-out-in-70-seconds developer conference. A week of amazement!

It was a great trip, and I’m delighted to have gone. My first time, and I hope to go again, though given the glitches and superfast sellout, I’m by no means sure I’ll be able to get a ticket again. This year, I tried to register but got an error on the Apple site, and was called back a day or two later; I’m not sure why I was chosen.

Apple and secrecy; they go together like Smurfs and buckshot.

My tips

  • I flew Virgin America, my favorite carrier these days, morning flight on Monday from SAN to SFO. There was fog, so I missed the first 20 minutes of the keynote. Flights both ways were packed but otherwise fine.
  • Hotel was the Pickwick, which is OK - nice and close, price reasonable and the lack of AC was no problem.
  • Be sure you have your Ethernet adapter; the xcode and beta downloads are gigabytes and Apple has a wired network that’s optimal.
  • SubEthaEdit is AWESOME for note-taking. Having ten people doing the work lets you keep up and still end up with a very detailed set of notes. Met some nice people this way, too.
  • I wish the session were not under NDA, as I’d love to write about them, but that’s Apple’s choice to make. Ahh well.

This guide was super helpful, well worth a read.


SFO: SFO on Friday

WiFi was superbly run, fast and reliable. WWDC wifi

I spent a few hours talking to the folks at Xamarin, which involved seeing some nice sights.

SF, iconic.

The after-hours party had good food, I left early as crowds are not my thing.

WWDC party

It was great. Go if you can. Not as intense/packed as PyCon, but a more sustainable pace. After PyCon, I was ‘tech hangover’ for a week; after WWDC I had no such effect. Longer sessions, longer breaks, better balance.

I Am Such an Idiot - Images Lost.

Yes, all of the inline blog images are gone. Toast. Kaput. One hundred percent operator error.

I made a mistake in copying over data from the previous server to the new one, and some of the contents of the all-important /usr/share/wordpress/wp-content directory tree didn’t get copied.

Then I compounded the fault by not doing enough checking post-migration.

Then I compounded that by re-using the backup drive on the new system.

I am deeply embarassed by this. I know this is a low-traffic site, but this is the sort of amateur-hour error that has me questioning my compentence.

There are 849 posts in the blog, so it’s not feasible to manually try and fix each, so I may just declare Blog Bankruptcy and start over.


Server Migrated

Yep, finally retired the Atom 330 box and moved everything over to the new server I got for my birthday a few months ago. It’s lots faster and uses even less power - under 20W max versus 90W for the last one. Smaller, nearly silent, SSD (256 mSATA), Core i3 CPU, 16GB of memory, just a delightful little box.

And it’s tiny, this is the Intel NUC hardware. Recommended.

Anyway, think I’ve got most of the services and such migrated, ping me if there’s things broken.