Diebold GEMS central tabulator contains a stunning security hole

Thu, 08/26/2004 - 11:43. Investigations
Issue: Manipulation technique found in the Diebold central tabulator -- 1,000 of these systems are in place, and they count up to two million votes at a time.

By entering a 2-digit code in a hidden location, a second set of votes is created. This set of votes can be changed, so that it no longer matches the correct votes. The voting system will then read the totals from the bogus vote set. It takes only seconds to change the votes, and to date not a single location in the U.S. has implemented security measures to fully mitigate the risks.

This program is not "stupidity" or sloppiness. It was designed and tested over a series of a dozen version adjustments.

Public officials: If you are in a county that uses GEMS 1.18.18, GEMS 1.18.19, or GEMS 1.18.23, your secretary or state may not have told you about this. You're the one who'll be blamed if your election is tampered with. Find out for yourself if you have this problem: Black Box Voting will be happy to walk you through a diagnostic procedure over the phone. E-mail Bev Harris or Andy Stephenson to set up a time to do this.

For the media: Harris and Stephenson will be in New York City on Aug. 30, 31, Sep.1, to demonstrate this built-in election tampering technique.

Members of congress and Washington correspondents: Harris and Stephenson will be in Washington D.C. on Sept. 22 to demonstrate this problem for you.

Whether you vote absentee, on touch-screens, or on paper ballot (fill in the bubble) optical scan machines, all votes are ultimately brought to the "mother ship," the central tabulator at the county which adds them all up and creates the results report.

These systems are used in over 30 states and each counts up to two million votes at once.

(Click "read more" for the rest of this section)

The central tabulator is far more vulnerable than the touch screen terminals. Think about it: If you were going to tamper with an election, would you rather tamper with 4,500 individual voting machines, or with just one machine, the central tabulator which receives votes from all the machines? Of course, the central tabulator is the most desirable target.

Findings: The GEMS central tabulator program is incorrectly designed and highly vulnerable to fraud. Election results can be changed in a matter of seconds. Part of the program we examined appears to be designed with election tampering in mind. We have also learned that election officials maintain inadequate controls over access to the central tabulator. We need to beef up procedures to mitigate risks.

Much of this information, originally published on July 8, 2003, has since been corroborated by formal studies (RABA) and by Diebold's own internal memos written by its programmers.

Not a single location has yet implemented the security measures needed to mitigate the risk. Yet, it is not too late. We need to tackle this one, folks, roll up our sleeves, and implement corrective measures.

In Nov. 2003, Black Box Voting founder Bev Harris, and director Jim March, filed a Qui Tam lawsuit in California citing fraudulent claims by Diebold, seeking restitution for the taxpayer. Diebold claimed its voting system was secure. It is, in fact, highly vulnerable to and appears to be designed for fraud.

The California Attorney General was made aware of this problem nearly a year ago. Harris and Black Box Voting Associate Director Andy Stephenson visited the Washington Attorney General's office in Feb. 2004 to inform them of the problem. Yet, nothing has been done to inform election officials who are using the system, nor have appropriate security safeguards been implemented.

On April 21, 2004, Harris appeared before the California Voting Systems Panel, and presented the smoking gun document showing that Diebold had not corrected the GEMS flaws, even though it had updated and upgraded the GEMS program.

On Aug. 8, 2004, Harris demonstrated to Howard Dean how easy it is to change votes in GEMS, on CNBC TV.

On Aug. 11, 2004, Jim March formally requested that the Calfornia Voting Systems Panel watch the demonstration of the double set of books in GEMS. They were already convened, and the time for Harris was already allotted. Though the demonstration takes only 3 minutes, the panel refused to allow it and would not look. They did, however, meet privately with Diebold afterwards, without informing the public or issuing any report of what transpired.

On Aug. 18, 2004, Harris and Stephenson, together with computer security expert Dr. Hugh Thompson, and former King County Elections Supervisor Julie Anne Kempf, met with members of the California Voting Systems Panel and the California Secretary of State's office to demonstrate the double set of books. The officials declined to allow a camera crew from 60 Minutes to film or attend.

The Secretary of State's office halted the meeting, called in the general counsel for their office, and a defense attorney from the California Attorney General's office. They refused to allow Black Box Voting to videotape its own demonstration. They prohibited any audiotape and specified that no notes of the meeting could be requested in public records requests.

The undersecretary of state, Mark Kyle, left the meeting early, and one voting panel member, John Mott Smith, appeared to sleep through the presentation.

On Aug. 23, 2004, CBC TV came to California and filmed the demonstration.

On Aug 30 and 31, Harris and Stephenson will be in New York City to demonstrate the double set of books for any public official and any TV crews who wish to see it.

On Sept. 1, another event is planned in New York City, and on Sept. 21, Harris and Stephenson intend to demonstrate the problem for members and congress and the press in Washington D.C.

Diebold has known of the problem, or should have known, because it did a cease and desist on the web site when Harris originally reported the problem in 2003. On Aug. 11, 2004, Harris also offered to show the problem to Marvin Singleton, Diebold's damage control expert, and to other Diebold execs. They refused to look.

Why don't people want to look? Suppose you are formally informed that the gas tank tends to explode on the car you are telling people to use. If you KNOW about it, but do nothing, you are liable.

LET US HOLD DIEBOLD, AND OUR PUBLIC OFFICIALS, ACCOUNTABLE.

1) Let there be no one who can say "I didn't know."

2) Let there be no election jurisdiction using GEMS that fails to implement all of the proper corrective procedures, this fall, to mitigate risk
This problem appears to demonstrate intent to manipulate elections, and was installed in the program under the watch of a programmer who is a convicted embezzler.

According to election industry officials, the central tabulator is secure, because it is protected by passwords and audit logs. But it turns out that the GEMS passwords can easily be bypassed, and the audit logs can be altered and erased. Worse, the votes can be changed without anyone knowing, including the officials who run the election.

Multiple sets of books

(Click "read more" for the rest of this section)

The GEMS program runs on a Microsoft Access database. It typically recieves incoming votes by modem, though some counties follow better security by disconnecting modems and bringing votes in physically.

GEMS stores the votes in a vote ledger, built in Microsoft Access. Any properly designed accounting program will allow only one set of books. You can't enter your expense report in three different places. All data must be drawn from the same place, and multiple versions are never acceptable. But in the files we examined, we found that the GEMS system contained three sets of "books."

The elections official never sees the different sets of books. All she sees is the reports she can run: Election summary (totals, county wide) or a "Statement of Votes Cast" (totals for each precinct). She has no way of knowing that her GEMS system uses a different set of data for the detail report (used to spot check) than it does for the election totals. The Access database, which contains the hidden set of votes, can't be seen unless you know how to get in the back door -- which takes only seconds.

Ask an accountant: It is never appropriate to have two sets of books inside accounting software. It is possible to do computer programming to create two sets of books, but dual sets of books are prohibited in accounting, for this simple reason: Two sets of books can easily allow fraud to go undetected. Especially if the two sets are hidden from the user.

A hidden trigger The data tables in accounting software automatically link up to each other to prevent illicit back door entries. In GEMS, however, by typing a two-digit code into a hidden location, you can decouple the books, so that the voting system will draw information from a combination of the real votes and a set of fake votes, which you can alter any way you see fit.

That's right, GEMS comes with a secret digital "on-off" switch to link and unlink its multiple vote tables. Someone who tests GEMS, not knowing this, will not see the mismatched sets of books. When you put a two-digit code into a secret location can you disengage the vote tables, so that tampered totals table don't have to match precinct by precinct results. This way, it will pass a spot check -- even with paper ballots -- but can still be rigged.

How and when did the double set of books get into GEMS?

Black Box Voting has traced the implementation of the double set of books to Oct. 13, 2000, shortly after embezzler Jeffrey Dean became the senior programmer. Dean was hired as Vice President of Research and Development in September 2000, and his access to the programs is well documented through internal memos from Diebold. The double set of books appeared in GEMS version 1.17.7.

Almost immediately, according to the Diebold memos, another Diebold programmer, Dmitry Papushin, flagged a problem with bogus votes appearing in the vote tables. The double set of books remained, though, going through several tweaks and refinements. From the time Jeffrey Dean was hired in September, until shortly before the Nov. 2000 election, GEMS went through over a dozen changes, all retaining the new hidden vote tables.

For four years, anyone who has known how to trigger the double set of books has been able to use, or sell, the information to anyone they want.

Black Box Voting Associate Director Andy Stephenson has obtained the court and police records of Jeffrey Dean. It is clear that he was under severe financial stress, because the King County prosecutor was chasing him for over $500,000 in restitution.

During this time, while Jeffrey Dean was telling the prosecutor (who operated from the ninth floor of the King County Courthouse) that he was unemployed, he was in fact employed, with 24-hour access to the King County GEMS central tabulator -- and he was working on GEMS on the fifth floor of the King County Courthouse. (Dean may now be spending his nights on the tenth floor of the same building; after our investigations appeared in Vanity Fair and the Seattle Times, Dean was remanded to a work release program, and may be staying in the lockup on in the courthouse now.)

Jeffrey Dean, according to his own admissions, is subject to blackmail as well as financial pressure over his restitution obligation. Police records from his embezzlement arrest, which involved "sophisticated" manipulation of computer accounting records, report that Dean claimed he was embezzling in order to pay blackmail over a fight he was involved in, in which a person died.

So now we have someone who's admitted that he's been blackmailed over killing someone, who pleaded guilty to 23 counts of embezzlement, who is given the position of senior programmer over the GEMS central tabulator system that counts approximately 50 percent of the votes in the election, in 30 states, both paper ballot and touch screen.

And just after he is hired, multiple sets of books appear in GEMS, which can be decoupled, so that they don't need to match, by typing in a secret 2-digit code in a specific location.

Dr. David Jefferson, technical advisor for California voting systems, told Black Box Voting that he could see no legitimate reason to have the double set of books in a voting program. He surmised that it might be incredible stupidity.

Dr. Jefferson should speak to Jeffrey Dean's partners and those who worked with him. "Stupid" is not how he is described. The descriptions we get, from Dean's former business partner, and from others who worked with him, are "sophisticated," "cunning," "very bright," "highly skilled," and "a con man."

This is the man who supervised the programming for GEMS when the multiple set of books was installed. Diebold, however, is the company that did nothing about it.

Internal memos show that Dean was sent the passwords to the GEMS 1.18.x files months after Diebold took over the elections company. Diebold clearly did not examine the GEMS program before selling it, or, if it did, chose not to correct the flaws. And after exposing this problem in 2003, Diebold still failed to correct it.

Elections were run on this tamper-inviting system for more than three years, and anyone who knew could sell the vote-tampering secrets to anyone they wanted to, at any time.

It has been a year since this report was first printed, and Diebold has never explained any legitimate reason for this design, which is rather elegant and certainly is not accidental.
But do new security measures solve the problem?

The MS Access database is not passworded and can be accessed illicitly through the back door simply by double-clicking the vote file. After we published this report, we observed unpassworded access on the very latest, GEMS 1.18.19 system in a county elections office.

Some locations removed the Microsoft Access software from their GEMS computer, leaving the back door intact but, essentially, removing the ability to easily view and edit the file.

However, you can easily edit the election, with or without Microsoft Access installed on the GEMS computer. As computer security expert Hugh Thompson demonstrated at the Aug. 18 California Secretary of State meeting, you simply open any text editor, like "Notepad," and type a six-line Visual Basic Script, and you own the election.

Some election officials claim that their GEMS central tabulator is not vulnerable to this back door, because they limit access to the GEMS tabulator room and they require a password to turn on the GEMS computer.

However...

(Click "read more" for the rest of this section)

Any county that uses modems to transfer votes may inadvertently be giving control of the entire central tabulator to anyone who gets at the computer through the modem phone lines (even if it is NOT attached to the Internet). This allows Diebold, or any individual, to manipulate votes at their leisure, from any personal computer anywhere in the world.

Let's talk about getting at the central tabulator through telephone lines: Mohave County, Arizona, for example, has six modems attached to its GEMS computer on election night. King County, Washington has had up to four dozen modems attached at once.

You will hear that the GEMS machine is stand alone, and is never connected to the Internet. It does have an Internet component, called "jresults," but nowadays most counties say that they do not hook GEMS up to the Internet. They say that they remove the disk from the GEMS computer and physically take it to another computer, from whence the Internet feed comes. Very nice -- BUT:

You can access a computer through phone lines as well as through the Internet. In fact, famous hacker Kevin Mitnick liked to hack through telephone lines, not the Internet.

If you have the dial-in numbers, it is possible to get at the GEMS computer from anywhere, using RAS. The dial-in protocols are given to poll workers, many people in Diebold have them, lots of temps have them, and the configurations have been sitting on the Internet for several years.

What if your county doesn't use any modems at all? That's excellent, but here's what we found: Harris & Stephenson visited county elections officials to ask for lists of names. We asked who was allowed to access the central tabulator, after it was already turned on, and who is given a password and permission to sit at the terminal?

Several officials told us they don't keep a list. Those who did, gave us the names of too many people -- County employees (sometimes limited to one or two). Diebold employees. Techs who work for the county, like county database technicians, also get access to GEMS. Printshops who do the ballots have some access also.

Diebold "contractors," who are temporary workers hired by subcontractors to Diebold were also reported to have gained access to the GEMS tabulator. (Diebold accounts payable reports obtained by Black Box Voting indicate that Diebold advertises for temps on Monster.com, hotjobs.com, and uses several temporary employment firms, including Coast to Coast Temporary, Ran Temps Inc, and also works with many subcontractors, like Wright Technologies, Total Technical Services, and PDS Technical Services.)

What if there is a password even to get onto the GEMS computer itself?

There usually is. The problem is this: Once that computer is open and running GEMS (on election night, for example), that password doesn't much matter. Votes are pouring in pell-mell, and they aren't about to shut that computer down until hours later, sometimes days later.

Also, Black Box Voting found another problem with the design of GEMS: Check out the Audit Log, which is supposed to record everything that happens. In every database, you find everyone logging is as the same person, "admin."

There is a reason for this. We did not find a way in GEMS to log in as a new user unless you close GEMS and reopen the file. Now who, on election night, with votes pouring in, is going to close and reopen the file? They don't. Instead, everyone calls themselves the same name, "admin," thereby ruining the audit log (which can be easily erased and changed anyway.)

What about counties that limit access to just one person, the county elections supervisor?

We've found nowhere that actually does this. The reason: Elections officials are dependent on the vendor, Diebold, during the election.

Suppose we have a computer whiz county official who is the ONLY person who can access GEMS?

Unlikely, but if you do: "Trust, but verify." We should never have to trust the sanctity of a million votes to just one person.

The following things can be done when you go in the back door in GEMS using Microsoft Access:

1) You can change vote totals.

2) You can change flags, which act as digital "on-off" switches, to cause the program to function differently.

According to internal Diebold memos, there are 32 combinations of on-off flags. Even the programmers have trouble keeping track of all the changes these flags can produce.

3) You can alter the audit log.

4) You can change passwords, access privileges, and add new users.

Let's talk about passwords

How many people can have passwords to GEMS? A sociable GEMS user can give all his friends access to the vote database. We added 50 people, and gave them all the same password, which was "password" -- so far, we haven't found a limit to how many people can be granted access to the election database.

Election meltdown:

We found that you can melt down an election in six seconds, simply by using the menu items in GEMS. You can destroy all data with two mouse clicks, and with four mouse clicks, you can destroy the configuration of the election making it very difficult to reload the original data.

Does GEMS even work as advertised? According to testimony given before the Cuyahoga Elections Board, the Microsoft Access database design used by Diebold's GEMS program apparently becomes unstable with high volume input. This problem, according to Diebold, resulted in thousands of votes being allocated to the wrong candidate in San Diego County in March 2004.

The Audit Log

Britain J. Williams, Ph.D., is the official voting machine certifier for the state of Georgia, and he sits on the committee that decides how voting machines will be tested and evaluated. Here's what he had to say about the security of Diebold voting machines, in a letter dated April 23, 2003:

"Computer System Security Features: The computer portion of the election system contains features that facilitate overall security of the election system. Primary among these features is a comprehensive set of audit data. For transactions that occur on the system, a record is made of the nature of the transaction, the time of the transaction, and the person that initiated the transaction. This record is written to the audit log. If an incident occurs on the system, this audit log allows an investigator to reconstruct the sequence of events that occurred surrounding the incident.

Since Dr. Williams listed the audit data as the primary security feature, we decided to find out how hard it is to alter the audit log.

We went in the front door in GEMS and added a user named "Evildoer." We had Evildoer perform various functions, including running reports to check his vote-rigging work, but only some of his activities showed up on the audit log. When we had Evildoer melt down the election, by hitting "reset election" and declining to back up the files, he showed up in the audit log.

No matter. It was a simple matter to eliminate Evildoer. We went in through the back door and simply deleted all the references to Evildoer.

Microsoft Access encourages those who create audit logs to use auto-numbering, so that every logged entry has an uneditable log number. Then, if one deletes audit entries, a gap in the numbering sequence will appear. However, we found that this feature was disabled, allowing us to write in our own log numbers. We were able to add and delete from the audit without leaving a trace.

Could the double set of books be legitimate?

From a programming standpoint, there might be reasons to have a special vote ledger that disengages from the real one. For example, election officials might say they need to be able to alter the votes to add provisional ballots or absentee ballots. If so, this calls into question the training of these officials. If election officials are taught to deal with changes by overwriting votes, regardless of whether they do this in vote ledger 1 or vote ledger 2, this is improper.

Also, if it was legitimate, it would be a menu item in the GEMS program, not executed in a hidden location triggered by a secret 2-digit code. Nothing in the GEMS documentation describes the use of any feature like this whatsoever.

Here's why we need to involve CPAs in vote tabulation regulations, procedures, and design:

If changing election data is required, the corrective entry must be made not by overwriting vote totals, but by making a corrective entry.

It is never acceptable to make changes by overwriting. Data corrections should not be prohibited, but must always be done by indicating changes through a clearly marked line item that preserves each transaction.

However, according to elections officials we interviewed, GEMS is improperly designed, and cannot perform an adjustment, and you can't journal changes that occur for weird reasons that really happen. (For example, a poll worker might accidentally run ballots through twice. You need to be able to correct this and still show your work.)

Instead of doing an adjustment and showing the explanation, retaining a permanent record of everything that happened, a common procedure is to wipe out the mistake, and simply overwrite it with new data. This is completely improper, from an auditing standpoint.

It is certainly improper to have the summary reports come from the second ledger, while pulling the spot check reports from the first ledger, with a provision in the back door to allow these two ledgers to be mismatched.

But there is more evidence that these extra sets of books are illicit: If the extra set of books is legitimate, the county officials, whose jurisdiction paid for and own the voting system, should be informed of such functions. Yet Diebold has not explained to county officials why it is there at all, and in most cases, never even told them these functions exist.

As a member of slashdot.org commented when we originally published this information: "This is not a bug, it's a feature."
Recommendations:

County officials should be required to maintain the following procedures to mitigate risk:

- Control access to the central tabulator through key logs and access cards

- Get rid of all modems and any wireless communications. The use of the digiboard modem bank attached to GEMS has got to stop.

- All corrective entries should be journaled and documented and publicly available, whether or not "they would change the outcome of the election."

(Click "read more" for the rest of this section)

- Maintain a list of everyone who enters the central tabulator room, with log in and out times and dates

- Any Diebold techs or county IT people who are allowed access the central tabulator room should be formally deputized or certified and sworn as election officials. Their names and credentials should be available to the public. The names of all individuals allowed access to central tabulators should be posted publicly during elections, and all individuals who have access the central tabulator should be available to citizens through public records requests.

- Physical control, in addition to keys to the room, should include blocking off access through ceiling panels and limiting physical access through all other means.

- In Diebold counties especially, the touch screens have got to go. The combination of a central tabulator that can be hacked six ways from Sunday in seconds, including the option of melting down the entire election, destroying the data, cannot be combined with unauditable electronic systems which keep no physical record of the vote.

Short term corrective action for touch screen counties: All counties who have touch screens also have central count machines for paper ballots, for counting absentee votes. In November, use paper ballots and count them all in the high speed central count machine used for absentees.

Polling place tapes State officials should require all counties to post polling place tapes containing all results before votes are transmitted to the central count facility.

- Two copies should be printed, one to be posted at the polling place and the other to be attached to the vote data, sealed, and transported to the county in front of at least two witnesses.

- A one hundred percent audit of all polling place tapes against the data in GEMS should be performed. This must include summing up the data on all polling place tapes, to compare totals from polling place vs. central tabulator. Remember: The way GEMS is constructed, it will pass the polling place comparison unless data is also totalled on both reports.

The importance of the second copy: The first copy is sent privately to the county elections official. A second copy is needed in order to make an audit set of data available to the public simultaneously.

- Quit co-mingling of data. Absentee and provisional/challenge/early votes must not be mixed together with polling place votes, but must be accounted for as a separate line item.

- There must be consequences for failure to follow risk reduction procedures.

- Taxpayers should demand that their local government dump Diebold and seek restitution of their money under consumer protection laws.

WAYS TO GET TAXPAYER RESTITUTION:

Attorneys: Black Box Voting may join in your county, state, or federal Qui Tam actions, waiving our right to the whistleblower bounty, retaining your own for attorneys fees if possible, providing the evidence we have (and it fills a small warehouse by now), in order to get taxpayer restitution for the purchase of this system.

What about the Qui Tam requirement to seal the evidence?

We believe that in this case, the fraudulent claims cases should be filed anyway, with a refusal to seal the evidence, to recover money for the taxpayer.

Yes, there are some who say that to prevail with a false claims act, the evidence must be sealed, and some have kept quiet about what they are gathering, saying "nothing can be done until after the election." We disagree. We, all of us, have an obligation to head off this train wreck.

ALL evidence must be put into the hands of the public, so that we can have a fair election. Let us go forth with preventive actions instead of sabotaging the election in order to profit on the back end.

Consumer fraud cases are needed to achieve taxpayer restitution. The evidence must not be sealed, because it is needed in order to put approprate security procedures in place to protect the election.

California is expected to announce on Sept. 6 whether they will help seek taxpayer restitution in the existing Qui Tam.

We predict that the California Attorney General will reject the effort to seek taxpayer restitution. Instead, they will try to rehabilitate Diebold.

Two members of the California Voting Systems Panel have told Black Box Voting that they intend to deal with Diebold after the election.

Diebold has just demonstrated its "voter verified paper ballot" to California. Yet, this system really doesn't matter, if you don't have security in place, don't audit, and can hack the central tabulator.

More predictions

Diebold (and many public officials) will claim, again, that they have corrected the problem. Public officials will omit any mention of the messy little business where the embezzler put the election-manipulation program into the central tabulator, or the uncomfortable fact that Diebold left it there for years, for anyone to use or sell.

The GEMS software will remain secret, and even the county officials won't examine it, because they are forbidden to do so by their contract with Diebold. (See our consumer report on contracts)

While we are walking local officials through the problems with GEMS over the phone, showing them it exists, we expect high ranking officials and the Diebold company to justify their decision to do nothing by attacking the messenger, (Black Box Voting). We will be called nuts, kooks, and cranks.

How much taxpayer money is involved?

You can't run the multimillion dollar Diebold voting system without GEMS.

State of Georgia: $52 million
State of Maryland: We hear it is up to $70 million by now.
State of Arizona: Approx. $50 million
State of California: In total, approx. $100 million
All in all, the Diebold system is used in about three dozen states, and the amount of money spent nationwide is between 1/2 and 3/4 billion.

This nasty situation reminds us of the Savings and Loan crisis in the 1980s, in that it is such a boondoggle that one hardly wants to admit that it exists. But, like the S&L scandal, the train wreck is approaching.

It's not too late.

* Voters want and deserve security procedures to protect the integrity of their vote this fall.

* Taxpayers want and deserve their money back.

* Public officials must be informed, and if they refuse to look, it must be documented so that they can be held accountable.

* Anyone who looks has a moral obligation to do something about this. Any public official who looks has a legal obligation to take the appropriate steps.