Types of tests:

Recommendation: we first focus on automated functionality / behavior driven tests to validate that things work as expected at runtime.

Unit tests can come after that. We could follow a maxim that if we touch a method (function), then we have to write a unit test for that method and any necessary mocks, fakes, or stubs as required. As others have pointed out on Wikimedia technical mailing lists, rigorous code review can also serve as a substitute for unit tests, or at least allow for pared down unit tests. The code review and actual manual functional testing of all introduced code can also suffice to flesh out the different scenarious involved with new or changing code.

Unit tests
Live testing using Beta Labs (HTML Scraping), CrossBrowserTesting.com (screenshots for well defined URLs on multiple user agents), and possibly Sauce Labs (headless browser interaction) infrastructure
[Other types of tests]

Testing control interface

Notifications / Reports
* 10 minute healthcheck, email gripe on failures only
* Pacific Time health status report: 6:30 AM, 10:30 AM, 2:30 PM, 5:30 PM
* Daily email with hyperlinks to screenshots for URLs on different devices. In time, maybe even a zip file of the screenshots scraped from browser testing site or PhantomJS or both.

Differences between Admin vs Partner access, control

Other things I've overlooked
* Analytics: Step 1: Minute and Hourly granularity, and Hourly job runs to fill Limn graphs
* Analytics: Step 2: Pivots on Language, High Level Device Type, Specific Device Type, HTTP Response Code, HTTP Request Type