Review
This Review of Statistical Methodology used in producing Small Business Benchmarks was undertaken by David Heath, Director of Cumpston Sarjeant Pty Ltd and Fellow of the Institute of Actuaries of Australia. It was delivered to us on 30th September 2013.
Introduction
The following is the report's introduction. You can download the complete report Review of statistical methodology used in producing small business benchmarks (PDF, 1.5MB)This link will download a file.
You have requested that we review the statistical methodology employed in the derivation of Small Business Benchmarks (SBB).
I understand this assignment arose following a report published by the Inspector General of Taxation, 'Review into the ATO’s use of benchmarking to target the cash economy' ('the IGT report'), in July 2012.
I am aware that the IGT report included several recommendations following their investigation, and that the ATO has acted on many of these recommendations.
In recommendation 3.1 in the report, the IGT recommends improvement in community understanding and confidence in the benchmarks, including: (b) assurance from an independent party with statistical expertise about the robustness of the benchmarking methodology
It is important to define the scope of our investigation. The recommendation in the IGT report and our subsequent brief call for an independent assessment of the statistical methodology used in the SBB process. However, we believe that it is not possible to confine our investigations and report to the mere calculation of the SBB, and the statistical methodology employed. The derivation of the benchmarks, the decisions on their publication, and their use by the ATO and other stakeholders reflect policy decisions that are wider than mere statistical techniques. The statistical methodology must be assessed within this context of the use of the benchmarks.
It should be remembered that while the process of the calculation, publishing and use of the SBB include statistical methodology, that methodology is merely a tool as part of the overall process and objectives.
Similarly, the assessment of the validity or otherwise of the statistical methodology employed must consider the context of the ultimate use of the SBB. As statistics are merely a tool, they still contain significant levels of judgment in their use. For example, in using confidence interval testing (a commonly applied statistical technique) the user must decide on an appropriate level of confidence.
The Inspector General of Taxation (IGT) report was a review into the ATO’s use of benchmarking to target the cash economy.
That report took a broad approach in considering the derivation and use of the benchmark ratios. In our review of the statistical methodology, we are unable, nor are we required to consider all aspects of the benchmarks, particularly in the usage of the final published ratios by the ATO.
However, where appropriate we may make comments in areas that are not purely statistical in nature.
For example, the use of benchmarks in issuing default or amended assessments – this is really a policy issue so not really within our assessment of statistical methodology. However, in discussions with representatives of the ATO, we understand that in applying the benchmarks, taxpayers subject to such assessments receive some 'benefit of the doubt'. Default assessments may be made where a taxpaying entity is unable to produce adequate records. In applying a benchmark in such cases, the ATO chooses the higher end of the relevant benchmark range.
This has the effect, of assuming the taxpayer in question has below average profitability; indeed the default assessment effectively assumes a profitability close to the lower 35% of profitability of comparable taxpayers.
The choice to use a particular ratio, and the resultant degree of 'leeway' in applying the chosen ratio (rather than an average, or some ratio that implicitly considers the taxpayers in question in a punitive manner) is a policy choice. As part of the scope of our assignment, we can only comment on our understanding of derivation of applied ratios.
Material used in our investigation
We began our investigation by reading the July 2012 report, published by the Inspector General of Taxation, 'Review into the ATO’s use of benchmarking to target the cash economy' ('the IGT report').
We also perused the relevant sections of the ATO website, which contained the published Small Business benchmarks, as well as additional background material.
Having gained this initial understanding we had meetings with relevant ATO staff, and at our request were supplied with further documents, as well as answers to particular questions. We chose six industries in order to verify the methodology and practise by the ATO, and were supplied with relevant datasets that enabled our own calculations.
Perhaps the most useful document for our investigation was an internal document , 'Small Business Benchmarks, A summary of the small business (performance) benchmark development process'. I am aware this is an internal document, and that a version entitled 'Small Business Benchmarks methodology and ratio calculations' shall be published on the ATO website. References below are still to the internal document.
The SBB document provides a clear and inherently sensible description of the process of the development of published benchmarks. I understand this is an internal document. Clearly its focus is the description of the development and derivation of the benchmarks, rather than their application. Accordingly it forms a good description of the process that we are required to assess, being the statistical validity robustness of the derivation of the benchmarks.
Rather than be concerned with coding our broad process has been to understand and assess the process whereby SBB are derived, and then perform some specific testing.
We have relied on this document to provide an outline of the framework used to determine the benchmarks. Having gained an understanding of the methodology used, we then took several steps:
- asked for clarification of some areas of the process, both during meetings with ATO representatives and by email
- gained an understanding of the derivation of the population to be used in the calculation of benchmark ratios and ranges. In particular we looked at the number of businesses before and after each data cull
- tested sample data, in particular that used to derive the ranges.
The Small Business benchmarks document describes twelve steps. We have described our investigations in terms of these twelve steps.