Election officials in a small county in California discovered by chance last week that the tabulation software they used to tally votes in this year's general election dropped 197 paper ballots from the totals at one precinct. The system's audit log also appears to have deleted any sign that the ballots had ever been recorded.
An investigation shows that the paper mail-in ballots were scanned properly by officials into the central-count optical-scan system made by Premier Election Solutions (formerly Diebold Election Systems) -- a receipt printed out by the machine at the time they were scanned on November 1, three days before the election, indicates that the machine recorded the ballots. The ballots even showed up in preliminary tallies counted on election night on November 4 and in a report printed out on November 23. But some time after this point, the tabulation software inexplicably deleted the ballots without election officials ever knowing.
Premier has acknowledged that a problem with its software caused the system to delete votes. The company has apparently known about the problem since 2004 and provided some election officials with a workaround, though Humboldt County election director Carolyn Crnich said she'd never been told of the problem. The issue involved a programming error that caused ballots to be randomly dropped from the tabulation software, without providing any indication to officials running the system that it was doing so.
A former subordinate, who left Humboldt last year to join the election staff of another California county, told a local paper that he had been told about the workaround in an e-mail sent by Premier, but never recorded the information in Humboldt County's written procedures and never told his boss about the workaround.
Crnich told Threat Level the issue has made her question her confidence in the voting system because, even though the company provided officials with a workaround, the problem indicated a fundamental flaw in the company's programming. She said she'd heard a lot of stories from other election officials about problems with voting machines, but never thought they applied to California.
"I've always sort of listened to those anecdotal incidents with a jaundiced ear because California has some very stringent requirements of election systems that are in use here as well as some very strict security procedures and I didn't think those things affected us here," she said. "But this has sort of put a cloud over any confidence that I had in the Premier equipment that's been in this department since 1995."
It's important to note that Crnich would never have discovered the problem through her standard canvassing procedures -- since the votes were still in the system after the canvas was completed -- nor would she have discovered it while conducting a mandatory manual audit that California counties are required to do.
The audit requires every county to hand-count ballots in 1 percent of randomly chosen precincts in order to compare the totals against digital tallies. But the audit involves only ballots cast physically at a precinct, not mail-in ballots, which are the ballots that the Premier/Diebold system dropped in Humboldt. Even if the Humboldt ballots had been precinct-cast ballots, Crnich would not have known they'd been dropped from the system if they weren't cast in a precinct that was included in the 1 percent of precincts that were hand counted.
Crnich discovered the missing ballots only because she happened to implement a new and innovative auditing system this year that was spearheaded by members of the public who helped her develop it.
Humboldt County, which is a small county located in northern California near the Oregon border, implemented the Transparency Project, whereby every paper ballot (Humboldt uses only paper ballots) gets digitally scanned by a separate commercial scanner, not made by a voting machine company, so that the ballot images can then be posted on the internet for anyone to examine and conduct their own independent recounts. (See this post for more about how the Transparency Project works.)
It was through the Transparency Project that Crnich and Mitch Trachtenberg, a volunteer who helped design part of the project, discovered the problem with the Premier software on November 30th after they finished scanning all of the ballots through the Transparency Project's commercial scanner two days before the county was required to certify its election results. After the county had already scanned and tabulated the 60,000+ ballots with the Premier voting system and created the official tally, the Transparency Project workers then spent 65 hours scanning the ballots into a Fujitsu scanner and creating digital images of each ballot. They discovered in doing so, that they had 216 more ballots recorded than the number of ballots that were counted by the Premier tabulation system.
Crnich said she initially thought they had inadvertently scanned some ballots twice, so she didn't hesitate to certify the official election results to her board of supervisors on the morning of December 1. But that afternoon, Trachtenberg discovered that 197 of the extra ballots all belonged to one precinct in the city of Eureka. After examining the ballots from that precinct, Crnich realized the 197 ballots had not been included in the official results she'd certified to supervisors, although they had been included in preliminary results that were recorded on election night and in a report printed on November 23rd. (Crnich still has not accounted for the outstanding 19-ballot discrepancy but believes the number may stem from ballots that were scanned twice.)
Convinced she'd done something wrong, Crnich contacted Premier to find out what she'd done to make the ballots disappear from the system. She said she was only worried at that time about "the embarrassment that this was going to cause me because I certified incorrect information to the secretary of state."
But after examining copies of her database, Premier told her the problem wasn't her but its Global Election Management System software (also known as GEMS) which is used to tabulate votes from all of the company's voting systems -- optical-scan machines as well as touch-screen machines.
Premier explained that due to a programming problem, the first "deck" or batch of ballots that are counted by the GEMS software sometimes gets randomly deleted if any subsequent deck is intentionally deleted. The GEMS system names the first deck of ballots "deck 0", with subsequent batches called "deck 1," "deck 2," etc. For some reason "deck 0" is sometimes erased from the system if any other deck is erased. Since it's common for officials to intentionally erase a deck in the normal counting process if they've made an error and want to rescan a deck, the chance that a GEMS system containing this flaw will delete a batch of ballots is pretty high.
The system never provides any indication to election officials when it's deleting a batch of ballots in this manner. The problem occurs with version 1.18.19 of the GEMS software, though it's possible that other versions have the problem as well. Crnich said an official in the California secretary of state's office told her the problem was still prevalent in version 1.18.22 of Premier's software and wasn't fixed until version 1.18.24.
Neither Premier nor the secretary of state's office, which certifies voting systems for use in the state, has returned calls for comment about this.
After examining Humboldt's database, Premier determined that the "deck 0" in Humboldt was deleted at some point in between processing decks 131 and 135, but so far Crnich has been unable to determine what caused the deletion. She said she did at one point abort deck 132, instead of deleting it, when she made a mistake with it, but that occurred before election day, and the "deck 0" batch of ballots was still in the system on November 23rd, after she'd aborted deck 132. She couldn't recall deleting any other deck after election night or after the 23rd that might have caused "deck 0" to disappear in the manner Premier described.
The deletion of "deck 0" wasn't the only problem with the GEMS system. As I mentioned previously, the audit log not only didn't show that "deck 0" had been deleted, it never showed that the deck existed in the first place.
The system creates a "deck 0" for each ballot type that is scanned. This means, the system should have three "deck 0" entries in the log -- one for vote-by-mail ballots, one for provisional ballots, and one for regular ballots cast at the precinct. Crnich found that the log did show a "deck 0" for provisional ballots and precinct-cast ballots but none for vote-by-mail ballots, even though the machine had printed a receipt at the time that an election worker had scanned the ballots into the machine. In fact, the regular audit log provides no record of any files that were deleted, including deck 132, which was deleted when Crnich intentionally aborted it. She said she had to go back to a backup of the log, created before the election, to find any indication that "deck 0" had ever been created.
Parke Bostrom, one of the Transparency Project volunteers, wrote in a blog post about the issue, "This means the audit log is not truly a 'log' in the classical computer program sense, but is rather a 're-imagining' of what GEMS would like the audit log to be, based on whatever information GEMS happens to remember at the end of the vote counting process."
Two other California counties use the same version of GEMS (version 1.18.19) used in Humboldt. The two counties received the 2004 e-mail from Premier/Diebold explaining the workaround for the problem, though the e-mail doesn't specifically say that not following the procedure will result in lost ballots. Threat Level obtained a copy of the e-mail, which was sent from Tari Runyan, Diebold's western regional support manager at the time. The e-mail says, "I have attached a document that details using Central Count for
November - specifically beginning Central Count and the Deck 0. It is very important that you follow these instructions - please contact Rob or myself if you have any questions."
The attached document reads:
California Secretary of State's office told a local paper that it had not been told about the problem by Premier in 2004. The secretary of state's office also found no problem with the software when it certified version 1.8.19 (.pdf) or when it conducted a top-to-bottom review of voting systems in 2006.
The GEMS software is used in several other states. At least nine counties in Florida (Dixie, Gilchrist, Glades, Hernando, Manatee, Polk, Seminole, St. Lucie and Wakulla) use version 1.18.19. Officials in those counties were unavailable for comment, since they're all out of town attending a statewide elections meeting. An elections worker in Polk County, however, said he'd never heard of the "deck 0" problem and had worked in Polk's elections office since 2003. Numerous other Florida counties also use the GEMS software, though they use version 1.20.2.
Two states, Maryland and Georgia, use the GEMS software statewide. Maryland did not respond to a call asking about the version of GEMS it uses, though a 2006 report about its voting system indicated the version of GEMS being used at that time was 1.18.24.
A Georgia spokesman said his state uses GEMS version 1.18.22G. The "G" indicates a version that is particular to Georgia. Merle King, executive director for election systems at Kennesaw State University, which tests voting systems for Georgia, didn't recall hearing about the problem with either versions 1.18.19 or 1.18.22. King wasn't familiar with the "deck" nomenclature and said that Georgia's system may operate differently than California's, since his state uses mostly touch-screen machines and doesn't upload ballots to GEMS in batches. Instead, Georgia election officials simply upload memory cards to the GEMS system.
Prior to the election this year, Premier announced a different problem with its GEMS software that caused the program to drop votes if officials tried to upload too many memory cards at once.
UPDATE: I was able to get in touch with one Florida official who was attending the conference that all election officials in Florida are attending this week. Ion Sancho, supervisor of elections in Leon County, which uses GEMS version 1.20.2, told me that he believes the issue in Humboldt County pertains only to counties that use the Premier/Diebold central-count high-speed scanner. Counties that use touch-screen machines and precinct-based scanners would likely not have the problems with GEMS that Humboldt experienced, even if they use the GEMS version 1.18.19.
Sancho said the precinct-based scanners he uses with GEMS version 1.20.2 do not produce "decks" in the GEMS machine because the ballots are not scanned in batches into one machine. Instead, they are scanned one-by-one at the precinct after each voter casts his ballot and are then uploaded on memory cards into the GEMS system. Sancho said that the Premier/Diebold central-count system is not certified for use in Florida and therefore the nine counties that are using GEMS version 1.18.19 are using precinct-based scanners, not the central count that is mentioned in the Premier/Diebold e-mail.
(Hat tip: John Gideon)
See also: