Finding a library

Chrome Debugger Protocol bindings:

• chromewhip
• PyChromeDevTools
• pychrome

WPT's constraints:

• Operating systems: GNU/Linux, macOS, and Windows
• Platform: Python 2
• License: BSD 3-clause compatible
• functional integration with the WebSocket protocol supported by Chromium/Chrome (e.g. version and protocol extensions)

Finding a library

Generic WebSocket clients implementations:

• Lomond
• pywebsocket - "pywebsocket is intended for testing or experimental purposes."
• wspy - client too primitive to interface with Chrome
• WebSock - does not expose a client abstraction
• gevent-websocket - Last released: Mar 12, 2017
• ws4py - unmaintained
• Autobahn & Twisted (Autobahn also integrates with asyncio)
• websockets - requires Python 3
• websocket-client - LGPL licensed
• Tornado

Finding a library

Generic WebSocket clients implementations satisfying WPT's constraints:

• Lomond
• gevent-websocket
• ws4py
• Autobahn & Twisted

Subjective priorities:

• probability of support/maintenance
• number and size of dependencies

Initial experimentation: https://github.com/bocoup/wpt-cdp-experiment

Winner: Lomond

Finding a library

Polling

Lomond checks for automatic pings and performs other housekeeping tasks at a regular intervals. This polling is exposed as Poll events. Your application can use these events to do any processing that needs to be invoked at regular intervals.

The default poll rate of 5 seconds is granular enough for Lomond’s polling needs, while having negligible impact on CPU. If your application needs to process at a faster rate, you may set the poll parameter of connect().

Note If your application needs to be more realtime than polling once a second, you should probably use threads in tandem with the event loop.

https://lomond.readthedocs.io/en/latest/guide.html#polling

Finding a library

Runner-up: wspy

wspy is a standalone implementation of web sockets for Python, defined by RFC 6455. The incentive for creating this library is the absence of a layered implementation of web sockets outside the scope of web servers such as Apache or Nginx. wspy does not require any third-party programs or libraries outside Python's standard library. It provides low-level access to sockets, as well as high-level functionalities to easily set up a web server. Thus, it is both suited for quick server programming, as well as for more demanding applications that require low-level control over each frame being sent/received.

https://github.com/taddeus/wspy

API stability

Ideal:

wptrunner <-- WebDriver --> Chromedriver <-------- CDP ---------> Chrome
wptrunner <--------------------- CDP ---------------------------> Chrome

API stability

Ideal:

wptrunner <-- WebDriver --> Chromedriver <-------- CDP ---------> Chrome
wptrunner <--------------------- CDP ---------------------------> Chrome

Actual target:

                                           .--- stable CDP ---.
wptrunner <-- WebDriver --> Chromedriver <-+ experimental CDP +-> Chrome
                                           '- deprecated CDP -'
wptrunner <----------------- stable CDP ------------------------> Chrome

Semantics

WebDriver: Navigate To

1. If the current top-level browsing context is no longer open, return error with error code no such window.

2. Let url be the result of getting the property url from the parameters argument.

3. If url is not an absolute URL or is not an absolute URL with fragment or not a local scheme, return error with error code invalid argument.

4. Handle any user prompts and return its value if it is an error.

5. Let current URL be the current top-level browsing context’s active document’s document URL.

6. If current URL and url do not have the same absolute URL:

   1. If timer has not been started, start a timer. If this algorithm has not completed before timer reaches the session's session page load timeout in milliseconds, return an error with error code timeout.

7. Navigate the current top-level browsing context to url.

8. If url is special except for file and current URL and URL do not have the same absolute URL :

   1. Try to wait for navigation to complete.

   2. Try to run the post-navigation checks.

9. Set the current browsing context to the current top-level browsing context.

10. If the current top-level browsing context contains a refresh state pragma directive of time 1 second or less, wait until the refresh timeout has elapsed, a new navigate has begun, and return to the first step of this algorithm.

11. Return success with data null.

source (retrieved 2018-12-10)

Semantics

Chrome DevTools Protocol: Page.navigate

Navigates current page to the given URL.

source (retrieved 2018-12-10)

diff --git a/tools/wptrunner/wptrunner/executors/reftest-wait_webdriver.js b/tools/wptrunner/wptrunner/executors/reftest-wait_webdriver.js
index c1cc649..f0ba2bc 100644
--- a/tools/wptrunner/wptrunner/executors/reftest-wait_webdriver.js
+++ b/tools/wptrunner/wptrunner/executors/reftest-wait_webdriver.js
@@ -1,6 +1,11 @@
 var callback = arguments[arguments.length - 1];
 function root_wait() {
+  if (document.readyState != "complete") {
+    setTimeout(root_wait, 10);
+    return;
+  }
+
   if (!root.classList.contains("reftest-wait")) {
     observer.disconnect();
@@ -37,8 +42,4 @@ var observer = new MutationObserver(root_wait);
 observer.observe(root, {attributes: true});
-if (document.readyState != "complete") {
-    onload = root_wait;
-} else {
-    root_wait();
-}
+root_wait();

-var root = document.documentElement;
-var observer = new MutationObserver(root_wait);
-
-observer.observe(root, {attributes: true});
-
-root_wait();
+var root, observer;
+
+(function begin() {
+  root = document.documentElement;
+
+  // This script may be evaluated before the document element is available.
+  if (!root) {
+    setTimeout(begin, 0);
+    return;
+  }
+  observer = new MutationObserver(root_wait);
+
+  observer.observe(root, {attributes: true});
+
+  root_wait();
+}());

Semantics

Chrome DevTools Protocol: Navigation-related events

Page.domContentEventFired

Page.frameScheduledNavigation - Fired when frame schedules a potential navigation.

Page.frameStartedLoading - Fired when frame has started loading.

Page.frameStoppedLoading - Fired when frame has stopped loading.

Page.lifecycleEvent - Fired for top level page lifecycle events such as navigation, load, paint, etc.

Page.loadEventFired

Page.navigatedWithinDocument - Fired when same-document navigation happens, e.g. due to history API usage or anchor navigation.

Architecture overview

Concepts

• browser - web browser process which runs a WebSocket server that implements the Chrome Debugger Protocol; this process is managed by the user of Pyppeteer
• pyppeteer.Connection - abstraction around a WebSocket connection to a running browser process
• pyppeteer.Session - interface for interacting with a browser window

                    .---------------- Pyppeteer ---------------.
    .---------.     |     .------------.           .---------. |
    | browser | 1 <---> * | Connection | * <---> 1 | Session | |
    '---------'     |     '------------'           '---------' |
                    '------------------------------------------'

Code hosted on GitHub.com: https://github.com/bocoup/wpt/tree/wptrunner-cdp

Change set:

15 files changed, 1042 insertions(+), 103 deletions(-)

API usage

$ grep -Ehro 'API status:.*' tools/pyppeteer/pyppeteer | sort | uniq -c
      3 API status: deprecated
      5 API status: experimental
     28 API status: stable

Settings

All trials:

• Used WPT at commit 8eab58f51c93f0075f4cc5e8e6d5b4fb2c4c4919
  • Date: 2018-12-10
  • Total tests: 31,101
• Used the wpt-docker-worker (only available from tasks initiated from the web-platform-tests/wpt repository)
• Did not restart after unexpected test results

Discrepancies

master to CDP:

• 361 tests with differing statuses

• 35 tests with differing subtest results

  (361 + 35) / 31101 => 1.27% discrepancy in patch

Discrepancies

master to CDP:

• 361 tests with differing statuses

• 35 tests with differing subtest results

  (361 + 35) / 31101 => 1.27% discrepancy in patch

master to itself:

• 21 tests with differing statuses

• 9 tests with differing subtest results

  (21 + 9) / 31101 => 0.10% discrepancy due to flakiness

Discrepancies

master to CDP:

• 361 tests with differing statuses

• 35 tests with differing subtest results

  (361 + 35) / 31101 => 1.27% discrepancy in patch

master to itself:

• 21 tests with differing statuses

• 9 tests with differing subtest results

  (21 + 9) / 31101 => 0.10% discrepancy due to flakiness

revert-gh-13419 (gh-13419) to CDP:

• 64 tests with differing statuses

• 32 tests with differing subtest results

  (64 + 32) / 31101 => 0.31% discrepancy in patch (adjusted)

Duration

master:

• Testharness: 17.88 minutes (σ = 8.11)
• Reftest: 12.16 minutes (σ = 5.29)

revert-gh-13419 (gh-13419):

• Testharness: 12.07 minutes (σ = 4.33)
• Reftest: 11.63 minutes (σ = 4.90)

CDP:

• Testharness: 12.65 minutes (σ = 4.58)
• Reftest: 7.81 minutes (σ = 3.47)

Test improvements

gh-14109: [paint-timing] Fix test for non-automated contexts

gh-14110: [paint-timing] Avoid race condition

gh-14132: [FileAPI] Remove reference to non-existent file

Future work

Contribute support for the per-message deflate WebSocket extension to wspy; switch to wspy

• More efficient use of transport -- potentially faster and more stable
• Small time/effort investment (presumably)

Future work

Migrate wptrunner to Python 3 and build on top of a more ergonomic library.

Future work

Work with Chrome DevTools maintainers to nail down the semantics of the most stable methods.

Summary

Experiment a success!

• Proved concept
• Suggested potential improvement in efficiency (decreased time-to-results)
• Rough edges with tractable path to resolution (~100 tests invalidated)
• Identified risk for production use: complexity and stability (could be mitigated through the use of a CDP-aware library)

https://github.com/bocoup/wpt/tree/wptrunner-cdp

Appendix: Flaky tests identified during this research

/client-hints/accept_ch_lifetime_same_origin_iframe.tentative.https.html
/content-security-policy/inheritance/blob-url-in-child-frame-self-navigate-inherits.sub.html
/cookie-store/cookieStore_delete_arguments.tentative.https.window.html
/cookie-store/cookieStore_delete_basic.tentative.https.window.html
/cookie-store/cookieStore_event_basic.tentative.https.window.html
/cookie-store/cookieStore_get_arguments.tentative.https.window.html
/cookie-store/cookieStore_get_set_basic.tentative.https.window.html
/cookies/samesite/form-get-blank-reload.html
/cookies/samesite/form-post-blank-reload.html
/fetch/api/request/destination/fetch-destination-worker.https.html
/html/webappapis/scripting/events/compile-event-handler-settings-objects.html
/paint-timing/first-image-child.html
/service-workers/cache-storage/common.https.html
/service-workers/cache-storage/window/sandboxed-iframes.https.html
/service-workers/service-worker/controller-with-no-fetch-event-handler.https.html
/service-workers/service-worker/fetch-event-redirect.https.html
/service-workers/service-worker/navigate-window.https.html
/storage/storagemanager-persist.https.window.html
/storage/storagemanager-persisted.https.any.html
/webvtt/rendering/cues-with-video/processing-model/audio_has_no_subtitles.html
/xhr/send-after-setting-document-domain.htm