# A script to compare two Macrobenchmarks runs

In [Statistically Rigorous Android Macrobenchmarks](https://blog.p-y.wtf/statistically-rigorous-android-macrobenchmarks), I laid out a methodology for rigorously comparing the outcome of two [Jetpack Macrobenchmark](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-overview) runs. To summarize the article:

* Remove sources of variations until the distribution fits a normal distribution with a stable standard deviation.
* Then compute the confidence interval for a difference between two means.

When I published the article, I also shared a [Google Spreadsheet template](https://docs.google.com/spreadsheets/d/1fqR99ROoSYtMrkoL5PWkUWDsW0E3JEXSTcXQqZJJCow/template/preview) that did some of that work.

Later on, a colleague (thanks Aaron!) shared a [Github repo of kscripts](https://github.com/kaushikgopal/kotlin-scripts) from Kaushik Gopal, and I realized I could easily turn my spreadsheet into a small Kotlin script to make it easier for other Square Android developers to play with comparing Macrobenchmark runs. So I did that.

Then I went on paternity leave and forgot about this, until recently when Saket Narayan reminded me that it could be worth sharing with the community.

Without further ado, [here's the script](https://gist.github.com/pyricau/07fd9598c5cdec0bc9f62505b6329df7).

You first need to install Kotlin, download the script and make it executable

``` bash
# Install kotlin
brew install kotlin

# Download the comparison script
curl -O https://gist.githubusercontent.com/pyricau/07fd9598c5cdec0bc9f62505b6329df7/raw/977b2a84532758fd614f6cc44dab43a242922cdb/compare.benchmarks.main.kts
chmod u+x compare.benchmarks.main.kts
```

Then you can run the comparison script:

```bash
# Compare the json output in run1 and run2 folders.
compare.benchmarks.main.kts run1/com.example.macrobenchmark-benchmarkData.json run2/com.example.macrobenchmark-benchmarkData.json

###########################################################################
Results for com.example.InteractionLatencyBenchmarks#openHomeScreen
##################################################
NavigationMs
#########################
DATA CHECKS
✓ All checks passed, the comparison conclusion is meaningful.

Data checks for Benchmark 1
- ✓ At least 30 iterations (100)
- ✓ CV (5.26) <= 6%
- ✓ Latencies pass normality test

Data checks for Benchmark 2
- ✓ At least 30 iterations (100)
- ✓ CV (4.32) <= 6%
- ✓ Latencies pass normality test

- ✓ Variance less than doubles (0.66)
#########################
RESULT
Mean difference confidence interval at 95% confidence level:
The change yielded no statistical significance (the mean difference confidence interval crosses 0): from -6 ms (-2.36%) to 1 ms (0.3%).
#########################
MEDIANS
The median went from 259 ms to 231 ms.
DO NOT REPORT THE DIFFERENCE IN MEDIANS.
This data helps contextualize results but is not statistically meaningful.
#########################
```

While I'd love to get feedback and ideas for improvements ([hit me up!](https://androiddev.social/@py)), I'm providing this script as is, with no guarantees and no intention to maintain it. Do whatever you want with it!