Automated scoring and assessment of online tests with minimal human intervention is well covered within the scope of OIDAR Services: Ruling reversed by AAAR on department's appeal.

While deciding case AAAR mentioned that - since there are no guidelines in Indian laws regarding the concept of minimum human intervention in electronically provided services, we refer to the European Commission VAT Committee Working Paper No 896 wherein the notion of ‘minimal human intervention’ was discussed in the context of determining whether or not a service can be said to fall within the definition of electronically supplied services. The European VAT Committee had agreed that for the assessment of the notion of ‘minimal human intervention’, it is the involvement on the side of the supplier which is relevant and not that on the side of the customer.

This is an appeal filed by the Department against the ruling given by the AAR.

The appeal is on the limited aspect of the classification of the Type-3 test administrative solution offered by the Respondent Company to its clients in India. The lower Authority had held that the Type-3 test is out of the purview of OIDAR Services. The Department is aggrieved by this decision and has come before us in appeal. The Appellant Department has contended that the Type-3 test has all the ingredients of an OIDAR service and the lower Authority has erred in not classifying the same as OIDAR service.

Facts of Case

Now let us examine the exact nature of the activity which takes place in a Type-3 test conducted by the Respondent. This candidate registers for the test online and remits the registration fees also online. The test is taken by the candidate at designated test centres in India where the candidate is assigned a computer workstation and the entire duration of the test is administered and supervised by a physical invigilator as well as an online proctor. The candidate accesses the test electronically via the internet at the test centre. The format of a Type-3 test involves a mixture of multiple-choice questions and analytical writing assessment questions i.e essay-based questions. On completion of the test, the Quantitative and Verbal elements of the test (multiple-choice questions) are scored based on a computer algorithm and the candidate is immediately given an indicative score report which provides the score only for the multiple-choice questions of the test. The score of the essay-based questions involving Integrated Reasoning and Analytical Writing elements, do not form part of the indicative score. The essay responses are sent by the Respondent to their scoring entity in the United States of America where the evaluation of the essays is done independently by a professional human scorer as well as a computer program known as an Automated Essay Scoring system (AES).

The essay scoring process by the scoring entity in the United States of America starts by evaluating if the information provided by the Respondent is complete or needs additional information before moving on to the next step. If the information provided is not sufficient, or an error occurs, a notification is sent back to the Respondent. Teams within the scoring entity and the Respondent work together to resolve any errors. If the essay information has all of the required information the next step is to evaluate if this is (i) a new essay record, (ii) a request to replace an existing record, or (iii) a request for a rescore of the essay. If the essay request is a new request or a request to replace an existing record, then the essay is routed to both a human scorer as well as to the AES for evaluation. This is the standard evaluation process for the Type-3 Test. Once the scorers (human as well as the AES) have completed scoring the essay, then the final score is an average of the human score and the AES score if the scores are within one-point difference. For example, if the human scorer returns a score of 5 and the AES rates the essay a 4, then the final score will be a 4.5. If the difference between the human scorer and AES is more than one point, then the essay is always routed to an expert human scorer and the expert scorer’s decision becomes the final score that is returned to the test taker.

Held by AAAR

There is no dispute on the fact that there is an element of human intervention involved in the process of scoring the essay responses in the Type-3 test. What needs to be decided is whether the extent of human intervention is ‘minimum’ or not.

"Since there are no guidelines in Indian laws regarding the concept of minimum human intervention in electronically provided services, we refer to the European Commission VAT Committee Working Paper No 896 wherein the notion of ‘minimal human intervention’ was discussed in the context of determining whether or not a service can be said to fall within the definition of electronically supplied services. The European VAT Committee had agreed that for the assessment of the notion of ‘minimal human intervention’, it is the involvement on the side of the supplier which is relevant and not that on the side of the customer."

We have already detailed the entire process involved in conducting the Type-3 test and it is seen that scoring by a human scorer is just one of the processes involved in a computer-based test. One of the major benefits of a computer based test is the facility of obtaining immediate grading. While grading of multiple-choice questions is done instantaneously using an algorithm, grading of essays involves the use of AES (Automated Essay Scoring) which is a specialized computer program to assign grades to essays. The Respondent has an entity in the United States which has developed an AES for reliable scoring of essay responses in a computer-based test. How does one know that the automatic scoring system works well enough to give scores consistent with consensus scores from human scorers? Any method of assessment must be judged on validity, fairness and reliability. An AES would be considered valid if it measures the trait that it purports to measure and it would be considered reliable if its outcome is repeatable. Before computers entered the picture, essays were typically given scores by two trained human raters. If the scores differed by more than one point, a more experienced third rater would settle the disagreement. In this system, reliability was measured by the degree of agreement among the human raters. The same principle applies to measuring a computer program’s performance in scoring essays. An essay is given to a human scorer as well as to the AES program. If the AES score agrees with the score given by the human scorer, the AES program is considered reliable. A machine-human score correlation serves as a good indicator whether the AES is returning a stable consensus score of the essay. Therefore, the role of the human scorer is in effect a means to ensure the reliability of the AES program. We do not discredit the importance of a human scorer in the process of assessment of essay responses. However, the focus here is on a computer-based test where the intent is to also assess the performance of the candidate using an automated system. The reliability of the AES is validated by the near agreement to the score given by the human scorer. For this reason, we hold that the involvement of the human element in the assessment of essay responses is well within the realm of ‘minimum human intervention’. Further, even from the perspective of the candidate, the human involvement is minimum in the entire process of the Type-3 computer-based test starting from the manner of registering for the test, the actual test-process and the outcome of the test, as all stages are automated. No doubt at times the candidate seeks a revaluation or rescoring of their essay responses and such revaluation task is given to a human scorer. However, even in such cases there is no direct human interaction of individualistic nature between the evaluator and the candidate. The Respondent accepts the electronic request for a rescore of the essay and returns the result to the candidate electronically. The candidate who is the service receiver has received a fully digitally provided service. When the Type-3 computer-based test is viewed as a whole, the scoring done by the human scorer is to be regarded as being within the realm of minimum human intervention. As such the ingredient of ‘minimum human intervention’ required to classify the service as OIDAR is also satisfied. We therefore, disagree with the decision of the lower Authority that the Type-3 test is not an OIDAR service.

We allow the appeal filed by the Principal Commissioner of Central Tax, Bangalore West Commissionerate and set aside the ruling given by the Authority for Advance Ruling with regard to the classification of the Type-3 test.

We hold that service provided for the Type-3 test is classifiable as an OIDAR service.

taxreply.com © 2024. All Rights Reserved