Types of tests:
Recommendation: we first focus on automated functionality / behavior driven tests to validate that things work as expected at runtime.
Unit tests can come after that. We could follow a maxim that if we touch a method (function), then we have to write a unit test for that method and any necessary mocks, fakes, or stubs as required. As others have pointed out on Wikimedia technical mailing lists, rigorous code review can also serve as a substitute for unit tests, or at least allow for pared down unit tests. The code review and actual manual functional testing of all introduced code can also suffice to flesh out the different scenarious involved with new or changing code.
Live testing using Beta Labs (HTML Scraping), CrossBrowserTesting.com (screenshots for well defined URLs on multiple user agents), and possibly Sauce Labs (headless browser interaction) infrastructure
- * Tests focus on
- [connection between MobileFrontend and ZeroExtension object interfaces
- [WAP] object interface
- Positive and Negative testing
- In some future state, maybe fuzz testing
[Other types of tests]
- * Test that validates each section of the page plus directly accesses ESI page fragment generators
- Positive validation based on characteristics of request: X-CS (missing, empty, incorrect, correct), User-Agent & X-Device (missing, empty, incorrect, correct), Subdomain, Language, forged source IP plus real source IP (we can add a bot IP to Varnish)
- Expected text format
- Expected rewritten URLs, no unexpected chargeable URLs
- Negative validation based lack of X-CS, X-Device, Subdomain, Language
- * Test that validates upon simulated user interaction the proper result occurs. Sauce Labs (Selenium "unit tests"), locally-run PhantomJS "unit tests", or some combination of these two can be used for this purpose.
- Test that does screen captures on CrossBrowserTesting.com.
- &cachebuster=<random> should be added to initial requests to try to obtain non-cached responses from the origin. Ideally, different articles should be accessed, maybe using the Random feature to get the article title at the beginning of a test.
- * Test that provides diff from previous test output
- https://www.mediawiki.org/wiki/Wikipedia_Zero/Test_cases lays out some of the historical tests which could largely be automated. Some probably need to be revisted for accuracy.
Testing control interface
- Scheduled jobs from Beta Labs and, as supported, CrossBrowserTesting.com and Sauce Labs. The remote testing services could conceivably be kicked off from Beta Labs or from Tool Labs if scheduling isn't already in place on them.
- Web interface on Tool Labs using HTTP BASIC authentication. Would simply call the "backend" Beta Labs server to kick off appropriate job.
Notifications / Reports
* 10 minute healthcheck, email gripe on failures only
* Pacific Time health status report: 6:30 AM, 10:30 AM, 2:30 PM, 5:30 PM
* Daily email with hyperlinks to screenshots for URLs on different devices. In time, maybe even a zip file of the screenshots scraped from browser testing site or PhantomJS or both.
Differences between Admin vs Partner access, control
- HTTP BASIC authentication would be the easiest way to do this. If the user is one of us, all tests are run. If it's a partner, the partner only gets the test results related to the partner.
- The backing authentication mechanism does not need to be complicated. An .htpasswd file would be easiest.
Other things I've overlooked
* Analytics: Step 1: Minute and Hourly granularity, and Hourly job runs to fill Limn graphs
* Analytics: Step 2: Pivots on Language, High Level Device Type, Specific Device Type, HTTP Response Code, HTTP Request Type