Posts by John Wood

In 2009 we were were growing fast. Very fast. So fast in fact that we were beginning to crush our MySQL database while running queries against a few tables that had grown to contain between 10 and 30 million rows. We moved the data from some of our largest tables into CouchDB to help us reduce the load on our MySQL server (see this series of blog posts for more info).

The move to CouchDB went well. Pages in our web application that would occasionally time out were now loading in a couple of seconds. And, our MySQL database was much, much happier. We liked CouchDB so much that we started planning a feature that would make heavy use of … Keep reading

Monitoring and reporting are a critical part of any business that relies on technology. At any point in time you should know exactly what state your system is in. This window into the health of your system helps you diagnose and troubleshoot problems, plan for the future, and most importantly, prevent issues from happening in the first place.

We use several different monitoring and reporting tools here at Signal, all serving their distinct purpose.

Scout

Scout is a hosted server monitoring service that allows you to collect stats on just about anything. Scout uses a plugin-based approach to determine what stats you would like to collect for each of your servers. After setting up your sever profile in Scout, you … Keep reading

Ensuring that customers get your message is a big deal to us. We’ve spent the last five years scaling our infrastructure so that messages can be sent quickly. A good example of this is with our customer redbox, who sends out a time sensitive rental coupon to more than 3 million subscribers once a month. Speed is critical and any latency can create customer service issues.

Just as important as speed though is the ability to get the message delivered, which means taking time to analyze delivery trends over time and making constant improvements. One change recently implemented with our SMS infrastructure is to perform post delivery analysis and remove subscribers in which message delivery repeatedly fails. SMS delivery … Keep reading

This week we released some exciting updates to our SMS message search functionality. While most of the changes were done to help us manage the messages behind the scenes, we were able to introduce a few new features that our customers will be able to take advantage of.

Search Messages Older Than 90 Days

Before the update, we limited you to a 90 day message search timeframe. This was because our previous message search solution got a little grumpy when you asked it to search for messages over too wide of a timeframe. That is no longer the case. You can now search all SMS messages sent within the past year.

Search Messages By Type

Sometimes it’s useful to filter … Keep reading

We will be performing some maintenance on two of our database servers on Saturday 10/29, starting at 12:00am (midnight Friday night / Saturday morning). The maintenance should be quick, and isn’t expected to take longer than 5 minutes. However, during that time, any operations that read from and write to the databases hosted on these servers will fail.

Operations that may be impacted include:
* Contact attribute updates
* Creating new subscriptions
* Sending a scheduled message
* Short URL visit tracking
* Various user generated reporting via our web interface

These operations may fail during the maintenance window, and should be retried when service is restored.… Keep reading

One Monday morning about a month ago, I was browsing through open issues in our bug tracker looking for something to work on. It was my week on “technical support”. At Signal, the engineer on technical support spends a week working on non customer facing issues with our infrastructure, process, etc, that are too large to simply knock out as you encounter them.

An issue entitled “Add monitoring to detect jobs that do not start” caught my attention. We already had monitoring in place to alert us when a job fails. However, we had nothing to alert us if that job never starts in the first place. We have many scheduled jobs that run in the background, some pretty important, … Keep reading

Recently we added a new series of reports aimed at making it easier to manage multiple accounts. In the blog post announcing those reports, we made it clear that they were far from complete, and that more awesomeness was right around the corner.

Since then, we’ve added reports to display the email and sms messaging volume for your accounts and the ability to export data for all of your accounts (instead of seeing results for just the top 6). But today we took these reports a step further, and added the ability to view the stats for an individual account in detail.

The new account detail report shows you all sorts of subscription and messaging related statistics about the … Keep reading

Note: This entry has been cross-posted from the personal blog of Signal engineer John Wood.

On Tuesday, July 26th, I attended my first code retreat. The code retreat was led by Corey Haines and Tyler Jennings, sponsored by Obtiva, and held to coincide with Tech Week. Simply put, it was invaluable experience, and I can’t wait for the chance to do it again.

What the hell is a code retreat?

For those who have never attended a code retreat, the format is simple. The group is given a problem. In our case the problem was Conway’s Game of Life. The code retreat is then broken up into several 45 minute sessions. Each session you … Keep reading

When we built the email messaging channel, we focused on making it incredibly easy to send an email blast to one or more lists of subscribers, or to a segment of those subscribers (like only those who live in Chicago, perhaps). But, one request we kept getting from customers was for the ability to send a one-off email to single email address.

Our customers were continuously finding themselves in a situation where they needed to send an email to a single recipient. Perhaps that recipient joined the list the day after an important blast was sent, and you don’t want them to miss out. Or, perhaps your customer accidentally deleted the email with your 50% off coupon, and they’d … Keep reading