Note – We are proud to inform you that a prior version of this article was cited in a decision of the Supreme Court of New York, Appellate Division. The information here was reviewed on April 18, 2005, and is believed to be accurate through that date. We’ll try to update this periodically. Please use the information here as an overview and a guideline only. Someone with expertise should analyze the facts of any individual situation before important decisions are made, particularly in questioned or challenged SAT score situations (euphemisms used by ETS when accusing a student of cheating on the SAT). At times ETS changes these policies without notification to anyone outside ETS. So, the current policies of ETS should be reviewed very carefully.
This article is focused upon the SAT’s math and verbal (now called “Critical Reading”) scores. A writing section has recently been added to the SAT in 2005; as of the date of this article, there has been no experience with a challenge to a writing section score. Some of procedures described below apply to ETS’s GRE and GMAT (the GMAT will be developed by ACT in the future), offered on computer; moreover, those tests have introduced some additional difficulties, including the occasional computer scoring glitch. As for the ACT, their procedures differ in some important respects, although many of the general principles in this article still apply.
I. Introduction & Testing Conditions
Test-takers have wide-ranging interactions with Educational Testing Service ("ETS"). The beginning of the SAT relationship usually occurs when a student mails to ETS a registration form and a check. More and more students are now registering for the test via www.ets.org. If everything proceeds normally (and it does for the vast majority of test-takers), the test-taker receives an admission ticket by mail before the SAT, takes the test under good testing conditions and receives a score by mail in 3-5 weeks. Scores are usually available by telephone, for an extra fee, 2 weeks after the test, and are now available via the internet.
The relationship between the student and ETS can go awry, but this happens rather infrequently. If an SAT Admissions Ticket (confirming registration for the test) isn't received by the student one week before the test, or if the Ticket is lost, the student should call ETS Customer Service to confirm registration. Usually the pre-registered student's name is on the test center's list; moreover, a student almost always can take the SAT on a stand-by basis, which requires the payment of an extra fee--check the procedures in the SAT Registration Bulletin. After the test the student can figure out what happened with the admission ticket.
Complaints about test center irregularities (noise, mistiming, etc.) range from the trivial to the significant. If test-takers are shorted one minute or less, it's generally not worth taking on ETS over it. Likewise, minor noise or momentary disturbances will be hard to fight; however, on very rare occasions there are significant disruptions to a quiet test situation, and those should be pursued vigorously.
Where significant disruptions or timing discrepancies occur at the test center, the first consideration is whether the student was sufficiently distracted so that a score cancellation should be requested. Some students assume that ETS will add points to their score to reflect the disruption or discrepancy. This will NEVER happen. On rare occasions ETS will schedule a special retest within a few weeks for a group of students, but usually only if a large number of students complain. If a student is unclear as to whether his/her score was impacted, he/she should request to be told the score BEFORE having to decide whether to cancel (this requires an extension of the right to cancel until the score is generated). ETS seldom permits this, but never offers it where it isn't requested.
Most complaints to ETS should be handled by a faxed letter with details and requested resolution (don't wait for snail-mail) along with courteous but persistent telephone follow-up, as one would do with any other business entity. Sometimes test-takers have difficulties with other testing agencies, like ACT or Law Services. Some of their procedures are significantly different from ETS procedures. Try to get advice from someone who is knowledgeable about the testing company with which you are dealing.
II. Questioned Scores, or Cheating Accusations
Delays in receiving test scores sometimes occur. When a student is notified that there is a delay in processing an SAT score, most often (perhaps 3/4 of the time) this results in the release (i.e., not canceling) of the score; however, there is also a 1/4 possibility of the most difficult dispute with ETS--questioned or challenged scores. These are euphemisms for ETS accusing a test taker of cheating, or otherwise unfairly obtaining a test score.
The following will focus on questioned or challenged scores. Again, most other disputes with ETS should be addressed by a letter and telephone follow-up. It is in the area of questioned score disputes that the stakes for the test-taker are the highest, and that ETS has its own arcane procedures which apply.
A student or parent or counselor may wonder exactly what the challenged score was. Sometimes ETS personnel will give the score over the telephone; sometimes they won't. Yet the score is on the first page ("Score Review Summary") of the student's file held by ETS. Just ask for the file (see Section IX below) to find out the questioned score.
III. ETS Attitude Toward Questioned Scores
When a test-taker makes a mistake regarding the SAT, the test-taker pays a price. When ETS makes a mistake regarding the SAT, the test-taker pays a price. There are two ways for a test-taker to lose in this game, and you don't necessarily have to be at fault to suffer some consequences.
Here's the nicest thing I can say about ETS--most people who work there genuinely care about being fair and trying to be responsive to students; however, the most well-motivated of ETS employees are stuck within a system that limits their flexibility. ETS has a "party line," or company attitude, that often dictates what will happen to a test-taker.
Here are some of the significant aspects of ETS’s attitude toward questioned scores:
- ETS requires at least two items of evidence before questioning scores, the most common of which are a large score difference (dramatic improvement from a prior test) combined with similar answers to a test-taker seated nearby.
- If ETS statistical analysis (which is limited in scope) or their handwriting analysis combines with a large score difference to indicate cheating, ETS employees take the position that they have enough evidence to support a decision of cheating, and that the student actually cheated.
- ETS employees believe that their system treats students fairly (which it sometimes doesn't).
- Therefore, in questioned score situations, ETS employees can, on occasion, be arrogant and/or patronizing to the very small percentage of students accused because they (ETS) fully believe that they are correct, that the test-taker cheated and that they (ETS) are bending over backward to be fair and protect the test-taker's rights.
- ETS asks for evidence from the test-taker, but routinely ignores it (unless it addresses a few, very limited issues such as a hand injury affecting handwriting).
- The most common evidence submitted by test-takers consists of character reference letters from teachers and other school personnel, which are given virtually no weight in the ETS test security process.
- Usually the best way to overcome a questioned score accusation is to obtain the ETS file and vigorously attack ETS evidence, which requires a significant degree of expertise.
IV. Types of Questioned Score Situations
There are two basic categories of questioned score situations: group situations and individual situations. Group situations involve the cancellation of scores for an entire group taking the test, and are usually based on test-takers getting access to test booklets before the test begins. Group situations are very complex and relatively rare, and will not be addressed here. Individual situations arise when an individual test-taker has his or her score questioned and threatened with cancellation.
Individual questioned scores come in two types: "unusual agreement" and "disparate handwriting." Of the two types, the former is by far the most common. Unusual agreement refers to unusual agreement of answers, and is an ETS euphemism for cheating by copying or otherwise communicating answers between two individual test-takers. Disparate handwriting is usually a euphemism for sending in an impostor to take one's test.
Because unusual agreement is the far more common basis for questioned scores, it will be the primary focus of the rest of this paper, although some comments will be made about impostor accusations.
V. What To Do If The Score Report Is Delayed
Often the first indication of questioned scores is that a score report will be delayed. Test-takers usually are in touch with others who have taken the test on the same day, and will find out when other students have received their scores. If a test-taker's score is delayed more that one week, one possibility is that scores are being questioned.
ETS will often send a letter which says, "Your SAT I: Reasoning Test scores are delayed because they are under routine administrative review." As previously stated, perhaps 3/4 of these situations result in a released score, with perhaps only 1/4 resulting in questioned scores. Also, there are other unusual possibilities--on rare occasions, ETS loses an answer sheet or the score report gets mailed to the wrong address or is somehow lost.
If the test-taker receives this "scores are delayed" letter, he/she should call ETS and ask specifically where they are in the review process, and what will happen next. Often the ETS person will reply that there are general processing delays, but the test-taker should ask for specifics. If asked point-blank whether the two items of evidence exist for a score to be questioned, ETS employees will almost always acknowledge it if that is the case.
In calling ETS, always keep careful notes of the dates and times calls are made, write down the name (and correct spelling) of the ETS employee to whom you speak, summarize what was said and try to follow-up with that person in subsequent calls.
If an ETS employee tells you that a score is being questioned, immediately request a letter stating the basis for questioning the score, and ask when you can expect to receive this letter. This may help expedite the process by a week or two. Then, seek advice ASAP from someone with expertise.
VI. ETS Questioned Score Procedure
Step #1--FLAGGING: The ETS Test Security Office (TSO) reviews a test-taker’s score, which has been flagged by large score difference or an inquiry of some type (sometimes an anonymous tip). This review most often occurs before the score is reported back to the student and to the other score recipients (i.e. colleges), but sometimes occurs after a score is reported.
The most common event causing ETS to review a score is that the test-taker has increased his or her score significantly from a score obtained at a prior administration. ETS does computer comparisons of current and prior scores, and "flags" a score for review if the increase is above a certain amount. With SAT scores, if a student increases 350 points or more in combined math + verbal scores, or increases 250 points or more in either math or verbal, the score is flagged for review. This will constitute a questioned score if ETS finds a second item of evidence to support cancellation.
Also, if an outside person (another student, counselor, admissions officer or coach) contacts ETS and says that cheating may have occurred, even if this is done anonymously, ETS will review the score.
Step #2--SIMILAR ANSWERS OR HANDWRITING: The TSO looks for handwriting differences in the two test forms of the same individual (current and previous), and does a statistical analysis of similar answers comparing the test-taker's answers with those of everyone seated nearby. If the TSO finds neither handwriting differences nor similar answers, the score is considered valid, and released and reported.
Step #3--"QUESTIONED SCORES LETTER": The "Questioned Scores Letter" (not to be confused with the "scores are delayed" letter, described above) contains a cursory summary of the evidence supporting canceling the scores, along with a list of options for the test-taker. These options will be discussed below. This letter invites the test-taker to make one submission of additional information that the ETS Board of Review might consider.
Step #4--COPY OF THE FILE: The test-taker has the right to request copies of all documents in his/her file. The vast majority of test-takers do not avail themselves of this right. This will be discussed further below. The Board of Review makes its decision based upon evidence in the file, so that it is imperative for a test-taker to see this evidence in order to understand why the ETS TSO is recommending score cancellation. Also, since the test-taker can make only one submission of additional information, it is highly recommended that the test-taker examine a copy of the file before making his/her submission of additional information. It is possible, but not guaranteed, that ETS will permit a second submission to the Board of Review if the first submission was done before a copy of the file was requested.
Step #5--ADDITIONAL EVIDENCE: The test-taker may choose to submit additional evidence to the Board of Review. It seems that in too many questioned scores situations, the test-taker submits additional evidence, the Board remains convinced that cancellation is appropriate and the test-taker ends up choosing another option. The most common mistake test-takers make is that they submit character reference letters to convince the Board of Review of their innocence. As stated before, the Board gives virtually no weight to these kinds of letters.
Step #6--BOARD OF REVIEW: After additional information is received, the test-taker's file is submitted to the ETS Board of Review. The Board decides whether or not to cancel scores. The Board can release the scores if it chooses (only one member of the three-member panel need "vote" for release of the score and the score will be validated and reported); however, if the Board decides to cancel scores, the TSO sends another "options" letter to the test-taker.
Step #7--OPTIONS: The test-taker may challenge the Board's decision to cancel scores either in court or in arbitration, or (sometimes) can ask a university to which he/she is applying to independently evaluate a score cancellation to determine whether the university agrees with the ETS position. Alternatively, the test-taker can take a free retest at a special administration set up by ETS. See comments below on the various options.
VII. Options Given by ETS
Pre-Option--Provide additional information. Although a student usually desires to provide character evidence, this is virtually always ignored by the Board of Review. Usually the only effective evidence is information that attacks ETS’s own statistical analysis. One advantage of providing additional information is that the only downside is loss of time; all other options are still available. The primary disadvantage is that the disregarding of the test-taker's evidence can be disheartening. High school seniors need to be particularly careful with delays, as they can adversely impact the college admission process.
Option #1--Free Retest. The disadvantages of retests are several, and usually involve increased pressure and anxiety due to the following factors:
1. A retest is usually taken by the test-taker alone in a room with an individual proctor;
2. A cloud of suspicion surrounds the test-taker;
3. The retest is usually non-disclosed, so it cannot be checked by the test-taker; and
4. The test-taker may have spent weeks or even months preparing for the original test, but then has to achieve a similar level of preparation with only perhaps a week or two notice of the retest.
One advantage of a retest is that it can be arranged rather quickly and scored right away, which is particularly important in admissions deadline situations. Another very important advantage of a free retest is that a student can score significantly lower (100 points per section, math & verbal) than the questioned test and still receive his/her questioned scores, while retaining the ability to score even higher than the questioned scores. Yet another retest option to be considered (if there is sufficient time) is retaking the test in a more typical, regularly-scheduled public administration, in which case a student would simply ask for a refund of the fee from the questioned test, and a cancellation of that score.
A former president of ETS, Nancy Cole, has stated in a letter that one retest might not be fair, and a student has a basis for arguing for a second retest.
Option #2--Refund. ETS will be happy to resolve the dilemma by refunding the test fee and canceling the questioned scores. This amounts to the test-taker "walking away" from the situation. The test-taker can take the test again at a regularly scheduled administration.
Option #3--Asking a college to review. This option has ETS canceling the scores, but then sending the whole file to a college (not all colleges will even consider doing this) to make its own decision. In past guides to test-takers, ETS discouraged test-takers from choosing this option, implying that it would be a very rare occasion indeed in which a college would accept a score as valid in the face of ETS's having decided that the score was not valid. SPECIAL NOTE TO STUDENT ATHLETES--often student athletes with challenged scores think that if their college accepts their score under this option, then the NCAA will be satisfied. WRONG! If the NCAA requires an SAT score, the NCAA will only take a score that ETS considers valid.
Option #4--Arbitration. While most non-lawyers and lawyers have a positive feeling about arbitration, ETS arbitration is a different kettle of fish. If arbitration is chosen as an option, ETS requires the test-taker to first sign an agreement to arbitrate. This agreement rigs the arbitration in ETS's favor in several ways:
- It specifies the standard for review, that ETS had substantial evidence to question scores, rather than the more neutral general civil standard of "preponderance of the evidence";
- It disallows in-person appearances, meaning that the arbitration decision will be decided solely on the submission of documents; and
- No new evidence can be submitted to the arbitrator that wasn't previously submitted to ETS's Board of Review. A very small percentage (this author estimates 10% or fewer) of students win these arbitrations against ETS.
Option #5--Judicial Review. This is an option many test-takers favor until they are confronted with the enormous cost and time factors involved. Going to court is portrayed as easy to do on television shows, but test-takers never realize that, in this country, only the very wealthy or the seriously physically injured are able to litigate. Furthermore, ETS has won nearly all of its approximately 30 test security cases, except for two that were structured and shaped by the author of this article (with Dalton vs. ETS being the most important case in this field). Finally, many judges are inclined to rule in ETS's favor because to do otherwise can be seen to be improper judicial meddling in affairs better left to testing experts. In the opinion of this author, the best single use of judicial review would be, in certain impersonation situations, to seek a mandatory injunction requiring ETS to submit a test booklet and answer sheet for fingerprint analysis. This might be expensive, but it could generate important evidence for a test-taker where impersonation had been substantiated by the opinion of an ETS handwriting expert.
In theory, evidence in questioned scores situations consists of information that tends either to support or undermine a score cancellation decision. Evidence can come in many forms--everything from the mental state of the student at the exam to prior test scores to an analysis of the correlation between the student's grade point average in school and the test score.
ETS's Board of Review and ETS's TSO both have their own very peculiar ways of considering evidence. Their weighing of evidence can involve a double standard: evidence provided by ETS that favors cancellation of the score is usually given great weight; moreover, evidence provided by the test-taker which supports the validation of the score tends to be given little weight.
Here are some specific examples of the ETS double standard in action:
Proctor vs. proctor--If the proctor submits an irregularity report, ETS weighs it very heavily as evidence of cheating; however, if the proctor notices no cheating and in fact submits an affidavit supporting the student, such evidence is routinely discounted and usually ignored by the ETS Board of Review.
Handwriting expert vs. handwriting expert--The ETS Board of Review virtually never accepts a test-taker's handwriting expert's opinion if it differs from that of the ETS handwriting expert; furthermore, if something is unclear to the Board in the ETS expert's report, the TSO will call for clarification. In the same situation involving the test-taker's expert's report, the Board will conclude that any lack of clarity is a flaw in the report itself.
Whether a student was in the room in general--The ETS Board of Review seems not to be interested in independent proof of whether a test-taker was in the test room in impostor accusation situations where the ETS handwriting expert has opined that the test-taker did not take the test.
Prior test results--ETS views results on its prior tests as valid and reliable, even where the test-taker proves illness or injury as a factor lowering a prior score. Prior tests that happen to be proctored practice tests administered in test preparation courses are routinely disregarded as valid prior test results.
Fingerprint evidence--If a test-taker wants to submit this as proof that he/she took the test, ETS will routinely deny the test-taker access to the original documents. Yet ETS asks the test-taker for a thumbprint to verify authenticity in retest situations. Fingerprint evidence, therefore, is only relevant to ETS if it helps ETS or favors their position, and it is not relevant if it helps or might favor the position of the test-taker. NOTE: in a recent questioned score situation, the mere request to send documents for fingerprint analysis to the fingerprint expert in the Dalton case appeared to be sufficient to get the scores released.
Confirmation that the student increased scores as a result of taking a test preparation course--This is an area of considerable controversy. The author has information from two different test preparation companies that, in the past, their submitting a letter was helpful in getting a student's score released. However, it was apparent in the Dalton case that detailed information submitted by The Princeton Review was not helpful (the student ultimately won), perhaps because The Princeton Review criticizes ETS and provides students unique assistance like the information in this essay. Consider these contradictory facts: ETS personnel believe that test preparation courses don't improve scores, but the College Board, their partner in the SAT, has set up a for-profit division to provide online test preparation courses!
Take note! In calling and/or writing to ETS, it is not helpful to refer to this article, or to The Princeton Review, or to The Princeton Review Foundation. There may still be some ETS employees who have a negative attitude toward our student advocacy positions, and/or us, and you don't need to have that negativity applied to your situation. Make use of the information you read here, but just avoid mentioning this article to ETS.
IX. Getting and Analyzing ETS's Evidence
Anyone desiring to see the evidence in their ETS file needs to be prepared to send a series of letters requesting that evidence. Here's a recommended "First Request" letter:
"First Request" letter in an "unusual agreement" (copying) case, to be faxed to the ETS TSO who signed a "scores are delayed" or "questioned scores" letter:
Note: You should call and ask for the best fax number. Don't use the general fax number on ETS letterhead, which is in another building.
Please send to me, as soon as possible, a copy of the entire ETS file on (name of test-taker), including copies of the seating chart, (test-taker's) original test booklet and answer sheets for both (test-taker) and Candidate B. Kindly include all the evidence considered by the TSO in deciding to cancel (test-taker's) scores. I make these requests under Standard 8.11 of the Standards for Educational and Psychological Testing of the American Psychological Association. In addition, I request a reasonable extension of your deadline date of __________ so that I will have enough time to review the evidence that you're sending me before I reply to it.
After receiving the above letter, ETS's TSO will respond by sending documents. Which documents? It seems to depend on their mood. Always they send the "Score Review Summary," and what they call their "Response Analysis." The Summary is an overview of the file, and has, on the first page, the scores of the test that are being questioned. This can be significant to many students who are not able to find out over the phone what their questioned scores actually were. Sometimes the Summary will indicate what important information is not in the file--for example, the seating chart may be missing.
The "Response Analysis" shows ETS's analysis of the answers of the test-taker compared to the answers of "Candidate B," the designation of another person from whom the test-taker typically is suspected of copying. This document can be hard to read, understand and interpret, and expert assistance should be sought. Often the most significant information on this document is the number of identical incorrect answers between the test-taker and Candidate B.
Often the file will contain an Erasure Analysis, and an analysis of the variable, or experimental, section. The evidence contained in these parts of the file needs to be evaluated by an expert to assess its strength and possible impact on the decision of the Board of Review.
In copying cases, one of the best methods of establishing the validity of the questioned score is to analyze the answer sheets for items that are dissimilar. Some of the best evidence in proving validity is finding test questions, preferably difficult questions, which the test-taker answered correctly and Candidate B answered incorrectly. Also markings (called scratchwork) in the test-taker's original test booklet that can be tied to the test-taker's answers can be excellent evidence of validity.
If a student decides to retest in a regular testing administration after a questioned score, it is particularly important that there be copious scratchwork in the test booklet that connects to the test-taker's chosen answers.
In mentioning a comparison of answer sheets and a review of the test booklet, it needs to be emphasized that ETS will usually not provide these documents unless specifically requested, and, in many cases, not unless specifically requested several times.
In summary, analyzing the evidence in the test-taker's file held by ETS involves the subtle weighing of many different factors such as matching wrong answers, erasures, the variable section, differing correct answers, scratchwork and any unique circumstances, all of which are interrelated. The test-taker should seek a review of his/her evidence and file by someone with experience and expertise to assist in drawing sound conclusions.
The following is an example of a submission to the ETS Board of Review that led to their release of the student's score. All identifying information has been removed, and ETS procedures have changed slightly since this 1996 case. Please note that each case turns on its own facts, and the document below should be used only to get a general idea of the kind of details into which an analysis of the student's file should go in order to make a meaningful impact upon the Board of Review.
BEFORE THE BOARD OF REVIEW OF EDUCATIONAL TESTING SERVICE
RESPONSE OF ----- -------
Pursuant to the letter of January 12, 1996 from Ms. X, Senior Test Security Specialist, ----- ------- hereby submits her response to Educational Testing Service (ETS) relative to the SAT exam taken by ----- ------- on November 4, 1995.
----- -------, by filing this response, does not intend to waive any of her rights. Specifically, ----- ------- reserves all rights to submit additional information in this or any other forum as a result of ETS's refusal, through its Test Security Office, to provide ----- ------- with all of the evidence relevant to this matter. As detailed in Section IV.E. of this Response, ----- ------- has made a request to the TSO for a copy of her actual test booklet, and, while a copy has been supplied, it omits the variable section.
II. HISTORY OF THIS MATTER
Before the -------s received any letters from ETS in this matter, there had been a plethora of oral communications between -----'s parents, Mrs. Y and Mr. Y, and ETS. Mrs. Y first called ETS on November 27, 1995, requesting that three more colleges be added for score reporting. She was asked to call back. The following day, November 28, 1995, Mrs. Y called again, asking why ----- hadn't received her November SAT scores since many students had already received their results. Mrs. Y was told by an ETS representative that scores wouldn't be sent for a few more weeks.
Mrs. Y called ETS again on December 12, 1995, and first spoke to Ms. X. Ms. X explained to Mrs. Y that -----'s file had to be cleared because of a significant score increase, and that this was routine procedure. Ms. X suggested that Mrs. Y call back on December 27, 1995. On that date Mrs. Y called, and Ms. X told Mrs. Y that -----'s file had not yet been cleared. Mrs. Y expressed her concern over the delay, emphasizing that ----- is a senior with college applications due. Ms. X suggested that Mrs. Y call back on January 5, 1996.
Mrs. Y called ETS on that date, and spoke again to Ms. X, who said that -----'s file still was not ready for release. Again Mrs. Y expressed concern over deadlines, this time deadlines for scholarships, and again Ms. X asked Mrs. Y to call back, this time on January 10, 1996.
Because of a blizzard on January 10th, Mrs. Y waited until January 12th to call ETS. There was no answer by ETS on either January 12th or January 15th, which probably was a holiday for ETS. On January 16th Mrs. Y spoke again to Ms. X, who stated that a letter would be going out in this matter. Ms. X reported that ----- received a 1200, but that the score was not released. She mentioned that there was no misconduct reported by the proctor, and she further recommended that Mrs. Y send in -----'s school transcript and supporting letters indicating -----'s ability.
On January 17, 1996, ----- ------- wrote to ETS, stating that her test scores improved as a result of her commitment to preparation after receiving disappointing scores from the May SAT. She enclosed letters from her SAT tutors, her math teacher, college recommendations and other material. On January 19, Mr. Y called to confirm that -----'s packet of information had been received at ETS, and Ms. X confirmed that it had.
On January 20, 1996 (delivery had been attempted on January 19th), the -------s received their first letter from Ms. X, which had been dated January 12, 1996. It should be noted that this letter was received exactly 11 weeks after -----'s November SAT. This letter raised four issues which suggested copying by ----- so as to possibly affect the validity of ----- -------'s November, 1995 SAT score:
1. unusual amount of agreement between -----'s answers and those of another test taker;
2. -----'s performance on similar sections of her test was inconsistent;
3. a substantial number of her answers had been erased and changed so that they matched the answers of another test taker; and
4. there is a substantial difference between -----'s scores on the November test and her scores on earlier tests.
On January 24th Mr. Y called Ms. X, and learned that the Board of Review had decided against releasing -----'s November scores. Mr. Y requested a copy of the Board's report, arbitration guidelines and a seating chart. Ms. X said that the seating chart information was confidential, but she would send the other information.
On January 30, 1996, Mr. Y faxed a letter to Ms. X requesting a copy of the entire ETS file on ----- -------, citing Standard 16.9 of the APA's Standards for Educational and Psychological Testing.
Ms. X, on January 31, 1996, replied by letter, stating that the Board of Review had considered the written information previously submitted by ----- ------- and still believed that there was substantial evidence to cancel -----'s score. ----- was asked to respond by February 28, 1996, or her scores would be canceled. Ms. X also enclosed various items from the ETS file in this matter, including a copy of -----'s original test booklet; however, the test booklet as supplied omitted section 4, the variable section.
----- ------- took the SAT I on three occasions, achieving the following SAT scores:
April, 1995: Verbal: 430 Math: 420 Total: 850
May, 1995: Verbal: 500 Math: 400 Total: 900
Nov., 1995: Verbal: 600 Math: 680 Total: 1200
III. ISSUES RAISED BY ETS
The January 16, 1996 letter from Ms. X to ----- ------- states the position taken by Educational Testing Service. Essentially, ETS has accused ----- ------- of cheating on the exam by copying from another test taker's paper.
Before addressing the issues raised by ETS in this matter, it is necessary to state clearly what has not been alleged relative to -----'s November SAT. Assuming that the -------s have at least been supplied with all documents and information relevant to this matter (and the -------s have not yet received a copy of the variable section from -----'s actual test booklet), it is apparent that none of the proctors assigned to the November, 1995 exam at the location where ----- ------- took the exam reported any misconduct on the part of either ----- or Candidate B.
Thus, there is no allegation that representatives of ETS observed any inappropriate, improper or disallowed behavior on the part of ----- ------- during the November, 1995 SAT.
ETS has based its action of withholding the results of the November, 1995 exam on supposed similarities between ----- -------'s answers and those of Candidate B. In total, Ms. X had indicated in her letter of January 16, 1996 four issues leading to a conclusion that copying or communication may have occurred: unusual agreement of answers; inconsistent performance; substantial erasures; and, the disparity between the scores earned by ----- ------- in May, 1995, and those earned by her on the November, 1995 exam.
It is our understanding that had ----- -------'s scores not increased beyond the ETS "threshold" for increases from the May, 1995 to the November, 1995 test, this entire process would not have been initiated. The position taken by ETS seems to be that an increase of 250 or more points on either the verbal or math parts of the test, or an aggregate increase of 350 or more suggests cheating. The "trigger" for a review of -----'s test appears to be her 250 point improvement in math.
A. There is no direct evidence of copying or collusion.
As noted above, ETS has produced no evidence of any kind of direct misconduct at the examination site as observed by proctors or anyone else. The statistical issues will be discussed separately.
B. ----- -------'s 250 point math increase is accompanied by a total of only 3 similar wrong answers out of 56 math answers supplied by -----, along with no erasures, no inconsistent performance and no other statistical evidence of cheating; moreover, there is substantial contrary evidence supporting -----'s contention that her November math score was the product of her own math work.
A careful examination of all the evidence contained in ETS's Response Analysis and -----'s test booklet shows that the weight of this evidence strongly supports the conclusion that ----- did her own work on the math sections of the November SAT, and did not copy from another test taker.
Of the four issues raised by Ms. X in her letter dated January 12, 1996, both the erasure and inconsistent performance categories support, rather than question, -----'s November math score. First, ETS's own erasure analysis shows absolutely no erasures on any of -----'s three math sections; second, -----'s math performance on each of the three sections of the November test is consistent with the others, as follows:
Math 1: 22 correct, 2 wrong
Math 2: 21 correct, 3 wrong
Math 3: 8 correct, 0 wrong
As to the issue of "unusual agreement" between -----'s math answers and those of Candidate B, ETS's own computer listing has only two relevant asterisks in math sections indicating, as set forth in ETS's "Discussion of Computer Listing," that "copying or communication may have occurred." These asterisks show that 1 of -----'s 2 wrong answers on Math 1, and 2 of -----'s 3 wrong answers on Math 2, matched wrong answers of Candidate B. Though these percentages exceed the 45% threshold used to generate an asterisk in these circumstances, the raw numbers themselves are far too small to support a conclusion of copying, and coincidence is just as reasonable an explanation for these 3 similar wrong answers.
Certainly many of the math answers of ----- and Candidate B are similar. This is so because they are both adept at SAT math, and both answered the large majority of math questions correctly. However, a closer inspection of the answer sheets themselves reveals a number of significant differences in the math performances of these two students:
1. On Math 1, ----- gave an incorrect answer to question #23 and omitted answering question #24, while Candidate B answered both questions correctly;
2. On Math 2, ----- omitted answering question #12, while Candidate B answered it correctly; also, ----- gave an extremely close (she answered "101," while the correct answers were 102 through 108), but incorrect, answer to grid-in question #19, while Candidate B answered it correctly; and
3. Most importantly, on Math 3, ----- answered question #7 correctly, while Candidate B omitted answering it; also, ----- omitted answering question #9, while Candidate B answered it correctly.
It is unclear whether any members of the Board of Review did the kind of analysis set forth above. In any event, the evidence on these three math sections of -----'s November, 1995 SAT supporting a charge of copying is limited to the large score difference and just a mere few wrong answers that are similar, while contrary evidence (that ----- did her own work) is abundant, and features a math question of medium difficulty which ----- answered correctly and Candidate B didn't answer at all.
C. ----- -------'s poor performance on the May, 1995 exam resulted from her taking the test during tennis season without any preparation.
When ----- ------- took the SAT in both April and May, 1995, she had not prepared for it either time. She took the SAT again in May primarily because the Question & Answer service was available for that test, and she knew that she would be able to review her performance by obtaining the Q & A. As stated in her letter of January 17, 1996, ----- was "shocked" and "disappointed" with her May scores. She explains the determination with which she prepared for the November SAT, and has submitted letters from her tutors affirming the intensity of her preparation efforts.
----- intentionally bypassed the October SAT and took the November SAT so that she could have extra time to prepare. ----- went to unusual lengths over many months to thoroughly prepare for the November, 1995 SAT.
D. ----- ------- had extensive SAT tutoring prior to the November, 1995 exam.
----- responded with the same resolve and discipline in preparing for the November SAT that she shows on the tennis court. She embarked on a several-month plan of intensive SAT preparation. While no claim is made that her SAT tutoring was exclusively responsible for the improvements in -----'s performance, a cursory review of the applicable facts suggest that it most likely contributed significantly to her improvement.
Ms. A, in her letter to the College Board dated January 17, 1996, describes the intensive and extensive tutoring she gave to -----. She began with an analysis of -----'s performance on the May SAT. Ms. A attests to -----'s hard work and improved performance. In addition, Mr. B of the Math Department at -----'s high school, in a letter also dated January 17, 1996, emphasizes -----'s accomplishment and diligence in a high-level math course at the school.
Ms. C, in a letter to ETS dated January 17, 1996, describes how she tutored ----- for the verbal sections of the November SAT. -----, a journalist, displayed a classic SAT test-taking deficiency - she was too creative in answering verbal questions. -----'s journalistic efforts are evident from material she has submitted in this matter, including a letter dated January 17, 1996 from Ms. D. Based upon -----'s diligence and Ms. C's experience, Ms. C felt that ----- would score in the 600's on the verbal part of the November SAT.
Educational Testing Service need not endorse SAT tutoring to recognize the potential for some benefit to some students. Considering -----'s background and frustrating experience on the May SAT, she seems to be the type of student who would have benefited significantly from a structured tutoring approach, and reason requires that the benefits be recognized in the context of this matter.
E. -----'s 80 point verbal increase is supported by substantial evidence that she did her own work on the scored verbal sections of the SAT
It should be noted that -----'s 80 point verbal increase, from May SAT to November SAT, is less than 1/3 the 250 point increase that is used to trigger a review of a student's score based upon large score difference. Of the four issues set forth in Ms. X's letter of January 12th, large score difference clearly does not apply to -----'s verbal performance.
In addition, although Ms. X's letter cites a "substantial number of (-----'s) answers have been erased and changed so that they match the answers of another test taker" as a second issue supporting copying, the number of erasures noted by ETS's own erasure analysis totals a paltry 3 out of the 72 verbal answers given by -----. 3 is hardly substantial, and the erasure evidence supporting copying by ----- on the verbal sections of the November SAT is trivial.
With large score difference and erasures not supporting copying on verbal, the remaining issues requiring analysis are unusual agreement of answers and inconsistent performance. As to unusual agreement of answers, there is an asterisk next to Verb 1 on the ETS Computer Listing for -----'s November SAT. Once again, as was done for -----'s math sections above, the -------s carried out a careful analysis of the actual answers given by the two students on their answer sheets.
This analysis of actual answers revealed substantial evidence that copying did not occur. The asterisk next to Verb 1 on the Computer Listing was generated because 7 of -----'s 9 incorrect answers on Verb 1 matched those of Candidate B; however, this was the only verbal item asterisked. It should be noted that the parallel category of "Run" was not asterisked because the highest run of -----'s matched incorrect answers was only 4, less than the number which would indicate copying.
Again, both students did well on verbal, so that many of their correct answers match on Verb 1. The seven incorrect answers that match on this section are questions #5, #8, #10, #21, #23, #27 & #32. On at least 3 of these questions (#5, #8 and #27), the answers given by both students appear to be the second-best answers available, which is consistent with the high-level abilities shown by the two students. On 3 of the other questions (#10, #21 and #23), the difficulty level of "5" assigned by ETS (on its "Correct Answers and Difficulty Levels" Sheet) indicates that these were 3 of the 5 very hardest questions on the section. It is understandable that even the best students will answer questions of this difficulty level incorrectly, and coincidence certainly can explain their choices of the same answers on these 3 questions.
More important, however, are the following significant differences in performance of ----- and Candidate B which emerge from a careful look at their answer sheets for Verb 1:
1. On questions #16 and #18, ----- gave incorrect answers, while Candidate B answered both questions correctly;
2. On questions #17 and #20, ----- omitted answering, while Candidate B answered both questions correctly; and
3. On question #14, ----- gave a correct answer, but Candidate B gave a different, incorrect answer.
Again, it is unclear whether any members of the Board of Review did this kind of analysis on -----'s Verb 1 section. The several differences noted, particularly -----'s answer to question #14, viewed in totality with the kind and quality of incorrect answers given to the "matching answer" questions on Verb 1, actually support the conclusion that -----'s verbal performance on this section was the product of her own efforts, and not of copying.
With substantial evidence favorable to ----- as just noted regarding the "unusual agreement of answers" issue for Verb 1, the only issue left to discuss regarding -----'s verbal performance is "inconsistent performance." -----'s verbal performance on each of the three scored sections of the November test is consistent with the others, similar to the consistency of her math performance:
Verb 1: 23 correct, 9 wrong
Verb 2: 23 correct, 5 wrong
Verb 3: 10 correct, 2 wrong
The -------s must assume that Ms. X, in the part of her letter dated January 12, 1996 citing inconsistent performance, referred to -----'s accomplishments on the scored verbal sections as compared to her variable section. Certainly it appears that she did not do that well on her variable section according to ETS's "Agreement of Responses" analysis. On the variable section she appears to have answered only about half of the questions correctly. On the other hand, and in -----'s favor, ETS's analysis shows only 26% of -----'s variable section responses match those of Candidate B, a percentage only slightly above chance and supportive of the conclusion that ----- did not copy. Also, because the variable section is pretesting new questions, it is often more difficult than scored sections, and sometimes contains ambiguous and faulty questions.
The -------s, for reasons detailed below, strongly object to any information regarding the variable section being used to question -----'s score. Their reason is that ETS has not provided ----- with the underlying data (i.e. a copy of the variable section of -----'s test booklet, along with the difficulty levels of the questions) that would enable her to do an independent analysis of this evidence. All the -------s have available here is summary, secondary information generated by ETS. Only in a Star Chamber proceeding would it be fair to use against ----- "evidence" that she is not permitted to see.
F. The ETS statistical analysis is flawed and unreliable in general.
The sole basis for the action taken by ETS relative to ----- -------'s November, 1995 exam is statistical data apparently produced because of the large score difference in the scores from the May, 1995 and November, 1995 exams. ETS has failed to provide anything beyond raw statistical data to support its conclusions of an unusual number of matched responses. No explanation of the process by which conclusions were drawn or validated has ever been provided.
However, a meaningful critique of ETS's statistical data would not seem to be necessary where, as here, a careful analysis of the two students' answer sheets provides an abundance of evidence supporting -----'s scores as valid. Should -----'s analysis not be deemed sufficient by the Board of Review to release scores, the -------s reserve the right to submit a critique (by an expert of their choosing) of ETS's statistical methodology and the lack of published, scientific backup for ETS's statistical approach used in questioning scores.
V. ISSUES RAISED BY ----- -------
A. ETS's refusal to provide a copy of the variable section of ----- -------'s actual test booklet constitutes misrepresentation and deception, and any use of variable section information to question -----'s score is fundamentally unfair.
It appears to be TSO policy in general to refuse to provide copies of all evidence to test takers when requested, contravening the alleged policy of ETS as announced to the public. Such action constitutes misrepresentation and deception by ETS.
The Standards for Educational and Psychological Testing of the American Psychological Association, provide, in Standard 16.9, that "All evidence considered in deciding upon the intended action, including evidence that might lead to a contrary decision, should be made available to the test taker on request."
Specifically citing this standard, the -------s made a request for copies of -----'s entire file, including a copy of her original test booklet. Evidence made apparent by an examination of the actual test booklet might lead to a contrary decision. The -------s have no way of knowing this unless they can examine the entire actual test booklet. ETS, while providing a copy of the test booklet, withheld a copy of the variable section of the booklet.
Clearly the entire test booklet is in -----'s ETS file in this matter. The members of the Board of Review have had an opportunity to examine it, including its variable section. Even if each and every member of each Board of Review who has examined -----'s file has carefully studied the variable section in the actual test booklet and has specifically determined that nothing contained therein was considered in making his or her decision to question -----'s scores, the intent of the Standards is to give ----- an opportunity to review the test booklet for evidence that would lead to a contrary result and support her position that her November, 1995 SAT score was valid and fairly earned.
Finally, the -------s are not now able to comment on ETS's confidentiality policies and ETS written internal policies regarding providing requested documents in their entirety (i.e., with variable sections) to test takers. Copies of these policies were requested by Mr. Y, and not provided by ETS. In the event that the Board of Review continues to question -----'s score after considering this Response, then the -------s request that the Board of Review supply ----- with the variable section of her test booklet, the ETS policies mentioned above, and copies of any other new evidence or relevant data supplied to or generated by the Board so that the -------s may supplement this Response.
B. ETS's Board of Review is not independent, and ----- ------- reserves the right to appeal directly to ETS's Board of Trustees.
ETS letters and documents repeatedly refer to members of the Board of Review as being "independent." It is ----- -------'s position that as well-intentioned as the members of the Board of Review might be, they are not independent, simply because they are all full-time, paid employees of ETS with significant vested interests in upholding ETS procedures, ETS's use of certain statistical analytic tools and the preferences of ETS management and legal counsel.
----- ------- reserves the right to appeal any adverse decision of the Board of Review to ETS's Board of Trustees for two reasons:
1) ETS's Board of Trustees are independent to the extent that they are the only individuals within ETS's institutional structure whose occuMrs. Yions, wages, careers and livelihoods do not depend on making decisions consistent with ETS's management preferences or current procedures; and
2) ETS's Board of Trustees should have the opportunity to confront, first hand, the serious discrepancies between their announced policies and the actual procedures employed by ETS's TSO and Board of Review in cases like this one.
C. The TSO's delay in this matter is both unconscionable and violative of ETS's own guidelines.
Section II. above details the history of many phone calls and contacts by the -------s attempting to resolve this situation, and the unconscionable delay by the TSO in proceeding to question -----'s scores.
The -------s have been told that ETS's own guidelines allow a 10 week period for ETS to question scores. Students typically get their scores back within 5 weeks, and it is unclear why the TSO even needs 10 weeks to generate the statistical information that is used to question scores. Let it suffice to note that it was a full 11 weeks in this matter before the -------s actually received formal written notice of the questioning of -----'s scores, along with the specific reasons for questioning as asserted by the TSO.
Should the Board of Review cancel -----'s scores in this matter, it will give tacit approval to the TSO's delay beyond ETS's own guidelines in a time period during which seniors desperately need SAT scores to meet college application deadlines.
D. ETS's seating chart may be inaccurate.
----- ------- recalls that the seat immediately in front of her at the November SAT was empty. Because the ETS seating chart in this matter shows that a student with booklet #447 was sitting immediately in front of -----, the -------s contest the accuracy of the seating chart.
VI. SUMMARY AND CONCLUSION
ETS's TSO cites four bases for questioning -----'s November SAT scores: large score difference, unusual agreement of answers, inconsistent performance and substantial erasures. With regard to -----'s math score, where a large score difference exists, there is no inconsistent performance and absolutely no erasures; furthermore, the careful analysis of answer sheets detailed above generates substantial and convincing evidence that ----- did her own math work, contradicted primarily by ETS's finding of a trivial number (3) of matched wrong answers. On math, the TSO asserts a large score difference alone, and virtually nothing more. Accordingly, -----'s evidence as set forth in this Response gives the Board of Review a clear and compelling basis for releasing her November SAT math score.
It should be even easier for the Board of Review to release -----'s November SAT verbal score, because a large score difference doesn't even exist on verbal. Nevertheless, the above analysis shows, for verbal, a trivial number (3) of erasures and an unusual agreement of answers situation in which a careful comparison of answer sheets generates substantial differences. ----- has clearly overcome the slim evidence that the TSO fairly asserts on verbal. The only evidence that ----- can't easily counter is inconsistent performance based upon the variable section, and that ETS evidence should be disregarded because ----- is unable to examine the underlying documentation.
Viewing math and verbal separately, very little statistical evidence of even the slightest probative value indicates that ----- ------- cheated in any way on the November, 1995 SAT. In fact, the total weight of the evidence set forth in this Response strongly supports the conclusion that ----- ------- did not receive help or act improperly on the November, 1995 SAT.
The evidence is also clear that the events surrounding ----- -------'s taking of the SAT in May, 1995, negatively affected her performance on that test. Further, -----'s extensive SAT preparation and tutoring prior to the November, 1995 SAT no doubt contributed to improvement in her performance on that test.
In summary, given the minimal statistical data in support of the ETS position, the significant factors in support of -----'s position and the totality of the circumstances, evidence and lack of evidence, fairness and reason compel the conclusion that the Board of Review should release ----- -------'s scores achieved on her November, 1995 SAT.
Dated: _____________ _____________________
Mr. Y