2021-01-31

A formal test on 4 phones (2 iOS, 2 Android) carried out on 31 Jan 2021 by Adam Fowler of the Herald Protocol development team as a pre-release test for Herald v1.2.0-beta3. These are the same 4 phones used for all Herald minor version and beta version performance regression tests.

Herald v1.2.0-beta3 introduces cross-protocol interoperability, enabling capture of payload data and RSSI measurements for the Herald protocol, and other connection and advert based protocols, such as OpenTrace, at the same time. Changes were also introduced to reduce memory and computation overheads, and improve consistency in event callbacks across Android and iOS.

Continuity has remained at > 99.9% between the 4 phones in the test during a full 13.5 hour period. Mean time between RSSI readings is 1.93s. Battery usage was reduced to 1.2%, 1.6%, and 1.6% per hour for the Samsung A10, A20, and iPhone 6+, down from 1.9%, 2.2%, and 2.5% in previous test (see 2020-12-08). The iPhone 6S battery is faulty, thus excluded from battery usage analysis (see 2020-12-08 for details).

The test was deliberately ran on known problem phones. A Samsung A10 on Android 29, Samsung A20 on Android 28, iPhone 6+ on iOS 12.4.5, and iPhone 6S on iOS 12.1.4. The Samsung A10 and A20 both rotate their BLE MAC address every few seconds, and the iPhones 6+ and 6S are test devices with worn batteries that have been in use since 2014 and 2015. The raw log data is available here for analysis and inspection.

Completeness has not been recalculated, but is likely to be higher. We have used the current known value for this until a full 10 device test can be ran.

Results

Measure	Result	Notes
Overall Efficacy	41.29%	Effective (UK MAX 46.57%)
Probability of detection	94.19%	Formal RISK detection
Phone Detection	100%	Includes iOS in the background
Continuity	99.95%	Full continuity
Completeness RT1	98.36%
Completeness RT2	TBD	Not yet calculated from data
Completeness CE	95.95%
Accuracy	95.95%	RSSI accuracy
Longevity	99.86%	13.5 hour test
Mean time between readings	<=1.93s
Country	UK
Have BLE Phones	72.53%	OFCOM reports
Hardware Support	98.19%	BBC data
Software/OS Support	98.00%	BBC data
Overall Pspec	69.79%
Pspec squared	48.71%

Phones tested

PhoneId	Phone Make and Model	OS and version	Known issues
1	Samsung A10 (SM-A105FN)	Android 28	A10s knowingly rotate their BLE MAC address with every request. Herald uses an ephemeral ID that rotates every 15 minutes as a performance improvement.
2	Samsung A20 (SM-A202F)	Android 29	Same issue as A10, above.
3	iPhone 6S	iOS 12.1.4	Phone name is Denmark. Battery is faulty and near end-of-life as it loses charge in steps.
4	iPhone 6+	iOS 12.4.5	Phone name is Netherlands.

Fair Efficacy Stats Summary

Phones	Max Pairs	Pairs Detected	Detection Percent	Max 30 second windows	30 second windows achieved	Missed windows percentage	Longevity change over 13.5 hours
4	12	12	100%	20040	20029	0.0549%	0.139%

Phone detection and distance estimation reports

The below 4 charts show discoveries, ID (data payload) reads, and RSSI (distance estimation) readings as individual dots.

The below charts show RSSI variation over time. In a formal 12 hour test these will mostly be static.

Battery usage

The below table and chart shows battery charge remaining over time. Note the iPhone 6S has a faulty near end-of-life battery that loses charge in steps (see 2020-12-08 results).

	Samsung A10 SM-A105FN	Samsung A20 SM-A202F	iPhone 6S ‘Denmark’	iPhone6+ ‘Netherlands’
Start time	2021-01-31 19:26:40	2021-01-31 19:27:36	2021-01-31 19:28:13	2021-01-31 19:28:14
End time	2021-02-01 08:52:24	2021-02-01 08:53:20	2021-02-01 08:55:34	2021-02-01 08:55:43
Start battery	100%	100%	100%	100%
End battery	84%	78%	66%	78%
Elapsed	13h 25m 44s	13h 25m 44s	13h 27m 21s	13h 27m 29s
Usage per hour	1.19146%	1.63826%	2.52679%	1.63471%
Notes	N/A	N/A	End of live Battery	N/A