Or: How I Learned to Stop Worrying and Love the Memory Leak

I received a "high memory usage" alert. Already panicking, I logged into New Relic and saw this terrifying graph:

Memory Leak?

That's a graph of memory usage, starting from when the server was created. For the uninitiated, when memory usage grows and grows and grows like that, chances are very, very high that you've got a nasty memory leak on your hands. Eventually, your server is going to run out of memory, and the system will start killing processes to recover some memory and keep running -- or just crash and burn.

The funny thing about this particular server is that I had already identified that this server was leaking resources, and I thought I'd fixed it.

Issue Closed

So, I started to investigate.

Running free -m confirmed that nearly all the memory was in use. But top (sorted by MEM%) indicated that none of the server processes were using much memory. Huh?

After some time on Google and Server Fault, I ran slabtop and saw that nearly all server memory was being cached by the kernel for something called dentry. This server has 16GB of RAM -- I'm no expert, but I'm pretty sure it does not need 14GB of cached directory entries. I know I can free this RAM, and with some more help from Google I find the magic incantation is:

echo 2 > /proc/sys/vm/drop_caches

After 5 terrifying seconds during which the server seemed completely locked up, the memory had been freed! But apparently, something about the way this server was acting was causing the kernel to keep all these directory entries cached. In other words, this was probably going to keep happening. I didn't want to have to create a cron job to manually clear the cache every 4 hours, but I wasn't above it.

More reading told me that maybe I was worried about nothing. Looking closely at the peaks of that graph, I saw that the kernel was freeing up memory.

Not a leak!

So maybe I was worried about nothing! Still, I didn't want New Relic alarms going off all the time. And what if the server needs memory more quickly than the kernel can free it? It seemed like something I shouldn't have to worry about.

Yet more Google-noodling, and I found that you can indeed tell the kernel how aggressively to clear the caches. (That latter post captured practically my thoughts exactly, and seemed to trace my experience tracking down this issue to a tee.)

So, after some tweaking, I settled on setting the following sysctl configuration in /etc/sysctl.conf (edit the file, then load it with sysctl -p):


It seemed like the higher I set the vm.vfs_cache_pressure, the earlier (lower memory usage) it would free up the cache.

Here's a sweet graph showing three states:

  • [A] untweaked
  • [B] manually clearing the cache with echo 2 > /proc/sys/vm/drop_caches
  • [C] memory usage using the tweaked sysctl configuration

Slab Annotated

Those saw teeth on the right? That's the kernel freeing memory. Just like it was doing before, but more aggressively. This is a "memory leak" I can live with.

So you want to move your Homebrew folder

By default, Homebrew gets installed under /usr/local. This is great, because it doesn't require you to use sudo to install and upgrade packages. But the downside is that it turns your /usr/local directory into a git repository. If you don't have any conflict with this, then by all means, stick with the default.

I had a conflict. Specifically, I use nave for node version management. Unfortunately, both Homebrew and nave drop a README.md in /usr/local, which means nave frequently modifies a file that's under Homebrew's version control and brew update breaks.


I decided to "move" my Homebrew directory to ~/Homebrew. Here are the steps I followed:

I didn't document this as I did it. Hopefully, I didn't forget anything.


Managers, Goals, and Performance

Key take away for me from "What Great Managers Do to Engage Employees" was this:

Performance management is often a source of great frustration for employees who do not clearly understand their goals or what is expected of them at work. They may feel conflicted about their duties and disconnected from the bigger picture. For these employees, annual reviews and developmental conversations feel forced and superficial, and it is impossible for them to think about next year’s goals when they are not even sure what tomorrow will throw at them. (emphasis mine)

This is something we've been struggling with at my job. We use quarterly OKRs to set goals for all employees, but many of our OKRs (for me and my direct reports) lose relevance after 2 or 3 weeks.

So, I'm still looking for ways to help me (and my team) measure our performance and set clear, relevant goals.

History v. State

I want to reprogram the way I think about the state of my data models.

Think of a blog post. Before I publish it, it's unpublished. After I publish it, it's published. If I unpublish it, it's unpublished again. Maybe I edit it and republish it. Published again.

I (and a lot of programmers, I think) tend to think of a thing like a blog post as having a state. Maybe in the MySQL database, blog posts are in a posts table having a state or status column. It's an ENUM type, probably.

But state is really just whatever the most recent change represents. Like git -- state is like HEAD. And too often, I don't think about saving the history of states when I should.

So, resolved: consider whether any new data model having a state or status should instead (or also) have a history.

Things I did today

Among other things, in the past 24 hours, I've:

  • set up an RDS MySQL instance
  • made an RDS instance a replication slave of our database hosted with Linode
  • fiddled with my bash prompt and other .bash_profile goodies
  • made breakfasts and lunches for my 2 little kids
  • made myself a lovely salad for lunch
  • dropped my kids at school (bigger task than it sounds like)
  • gone to work, where I
    • held scrum
    • reviewed and merged pull requests
    • reviewed and rejected pull requests
    • finished a bug fix
    • reverted a major feature deployment
    • communicated a plan to migrate our database off of Linode and into RDS
    • debugged and tested our database migration plan
  • went to a meetup
  • fixed my home printer not printing
  • plunged a toilet
  • cleaned a toilet
  • written this blog post

Still to do:Update: Also Done

  • review and script our migration plan
  • migrate our database off of Linode and into RDS

What is it about Twitter?

I like people. I swear I do. It's just that personal interactions tend to be extremely tiring. They take a lot out of me. And the larger the group of people, the more severe the energy drain. Same with familiarity: the less familiar I am with people, the more draining it is to interact with them. So, most of the time, I avoid large gatherings and being with people I don't know very well.

One thing I've noticed, though, is that most interactions I have with people online are not at all draining. Twitter, IRC, IM, GitHub, email -- I thrive in these communities and get energized when I can interact with people online. Twitter and GitHub are my favorite ways to communicate with people.

Twitter is great because it's a really light lift. You don't even know who's listening. A lot of the time, it feels like making wise-cracks from the back of the classroom. But there's also information sharing. I like to tweet links to blog posts I've seen. Often, there are common threads that tie my Twitter timeline to in-person conversations I've been having. But I think the key is that no response is expected. Okay, sometimes I look forward to replies and favorites and retweets. If I post a funny tweet, I hope someone faves it -- it's the same as going for a laugh in conversation, but without the awkwardness when I flop.

GitHub is completely different than Twitter. By definition, on GitHub I am interacting with people on a specific programming project, and we are discussing code. At my company, we also do this in Pivotal Tracker, but the interaction there is terrible. On GitHub, it's incredibly easy to translate my thoughts into a new issue or comment on an existing issue. And when we get a good conversation going on GitHub, that same ease of communication can morph into something more than a debate over the merits of one approach to a problem over another. Specifically, it's really easy to post images and add emojis. The humor and fun that enables can turn an otherwise boring or contentious comment thread into experiences that I remember with the kind of fondness that I imagine other people have for great parties.

These tools enable me to experience the positives without the negatives. I get all the joys of interpersonal interaction without draining my energy.

And I realized tonight that that enablement -- providing that bridge from my personality island to the mainland of other people -- makes these tools dear to me in an intensely personal way. It's like they augment my personality. Or give me superpowers. Or something.

I haven't quite put my finger on it. But I'm pretty sure that that's why I get raging mad when Twitter shoves shitty advertising in my face or GitHub refuses to enable notifications for gist comments. They're fucking with my shit when they do that. These tools -- they're not a trifle to me. They've become a part of me because of the deep interactions they've enabled me to achieve. It's like I've integrated them with my own personality. So when you fuck with them, you're fucking with me.

And I don't like when you fuck with me.

Yosemite Upgrade Changes Open File Limit

OSX has a ridiculously low limit on the maximum number of open files. If you use OSX to develop Node applications -- or even if you just use Node tools like grunt or gulp -- you've no doubt run into this issue.

To address this, I have this line in my $HOME/.bash_profile:

ulimit -n 1000000 unlimited

And a corresponding entry in /etc/launchd.conf:

limit maxfiles 1000000

That solved the problem until I upgraded to OSX Yosemite, after which I began seeing the following error every time I opened a terminal window:

bash: ulimit: open files: cannot modify limit: Invalid argument


Luckily, I a little Google foo yielded this Superuser post (and answer).

So it was a quick fix:

$ echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
$ echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ ulimit -n 65536 65536    

Then I updated my $HOME/.bash_profile to change the ulimit directive to match that last command, above, and I was back in business.

Easily prune your ssh known_hosts file

At some point, you've probably seen this message when you try to log in to one of your servers:

ssh failure message

This is really common when you have Amazon EC2 instances behind Elastic IPs because the IP address stays the same (and probably the hostname, too), but as new instances replace old instances, the new instances' ssh keys are probably different.

But if you look carefully, you'll see that the failure message tells you how to resolve this problem:

Offending RSA key in /Users/[username]/.ssh/known_hosts:5

That means that line no. 5 of the known_hosts file contains the problematic key. So, assuming that you are sure this is NOT in fact a security breach, you can remove that line.

It's a bit of a pain-in-the-butt to manually edit this file, though. You can use sed to do it easily, but if you're like me and you don't use sed all the time, you need to look at the man pages every time you want to use it. That's why I wrote this quick bash script to do it automatically.

Drop that in your PATH and make it executable. Then you can simply type ssh-purge-host 5 to remove line 5 from your known_hosts file.

Hope that's useful!

Could JXCore Be An Awesome Deployment Tool?

JXCore allows you to turn Node.JS applications into stand-alone executables. One possible use case would be to package up your entire application in an executable and deploy it to production servers, skipping the usual dance with git and npm. If performance is good, this could make for an interesting deployment tool. Deploy by Dropbox? Yup, you could! It's on my list of things to try with a hobby project.