Speed Test Methodology

What is your Speed Test Methodology? (Approach for Measuring VPN Speeds)

Perhaps one of the most significant factors of a quality VPN, besides encryption and security levels, is the speed of the network.  A slow VPN can make browsing painful, and make streaming video (or even audio) basically impossible.  For this reason, our editors go to great lengths to conduct in depth and full speed tests of VPN networks.

There are basically two approaches our editors use when conducting a speed test of a VPN network.  The first method utilizes the providers in-house software to test VPN speeds.  The second method utilizes a standard speed test website.  We generally prefer the site SpeedTest.net.

Our full speed test of HideMyAss VPN is an excellent example of an approach using the providers in-house software.  The full speed test of iVPN.net is an example of utilizing SpeedTest.net to conduct the tests.

In general, all tests for a single provider are conducted by the same editor with the same internet connection.  We also have that editor start and finish the tests in one block of time.  By keeping the tests with the same editor, on the same connection, at the same time of day we feel confident that our test results will not be biased.  We would be introducing significant bias if we were to run tests by different editors, with different connections, at different times of day.

Right now our US based East Coast editor is our primary speed tester.  This means we are using a standard cable modem connection.  We feel this best represents the majority of high speed internet customers.  All speed tests will identify the location of the editor, his/her native connection (type and speed), as well as the time of day in which the tests were performed.

Below are details on our specific approaches.

Standard Approach Utilizing SpeedTest.net

Our test methodology is as follows:  first, connect to the network to be tested (L2TP, OpenVPN, PPTP, etc.); second, open SpeedTest.net and begin test; third, allow SpeedTest.net to select best server based on ping; fourth, record upload/download/ping during two test runs; and fifth, where there are significant differences between the test runs (>450KB/s difference) complete another test run.

Basically we are utilizing SpeedTest.net and completing multiple runs on each VPN server.  By performing multiple runs, our results should better reflect the speed of the VPN network and hopefully control for issues with editor’s connection or possible hiccups.

Approach Utilizing Provider In-House Software

A number of the providers include the ability to speed test their VPN servers.  The providers recognize that customers wants high speed connections from the VPN, so they include this feature in their VPN software.  This ensures that a customer will know which VPN server is fastest and makes VPN server selection much easier.  In this case, because the in-house software will differ from provider to provider, our editor will go into great detail about how the software completes its speed test.

Our editors do evaluate the way the in-house software completes the test.  We look to make sure the ping, download, and upload speeds are measured and reported.  We also look at how the speeds are measured.  Since the industry standard method on sites like SpeedTest.net is to test speed by by downloading and uploading a test file, we like to see this from the in-house software as well.  As long as the software completes the speed test using a test file, and upload, download, and ping are reported we feel confident that we are getting an accurate speed test.

If for some reason we don’t feel confident in the in-house software, we will fall back to the standard approach utilizing SpeedTest.net and identify this in the test report.

Displaying the Results

We like to display the test results in raw format.  This means we will show screenshots, and raw data from the speed tests.  In terms of screenshots, you can expect to the see the images from SpeedTest.net.  In terms of raw data, you can expect to see spreadsheets and tables.  We also like to do some analysis on the results and compare the speed results to our streaming recommendation.

If you want more information on our streaming recommendations, please visit our Learn article for a complete listing of the recommendations and supporting documentation. We also have a good Learn article on network speeds and bandwidth units (e.g. Mbps/Kbps/KBps/etc.)