Hacking blind spot: States struggle to vet coders of election software
Politico (archive.ph)
By John Sakellariadis
2024-09-01 11:00:00GMT
In a six-month investigation, POLITICO found there is little oversight of the supply chain that produces crucial election software, leaving financially strapped state and county offices to do the best they can with scant resources | Joe Raedle/Getty Images
When election officials in New Hampshire decided to replace the state’s aging voter registration database before the 2024 election, they knew that the smallest glitch in Election Day technology could become fodder for conspiracy theorists.
So they turned to one of the best — and only — choices on the market: A small, Connecticut-based IT firm that was just getting into election software.
But last fall, as the new company, WSD Digital, raced to complete the project, New Hampshire officials made an unsettling discovery: The firm had offshored part of the work. That meant unknown coders outside the U.S. had access to the software that would determine which New Hampshirites would be welcome at the polls this November.
The revelation prompted the state to take a precaution that is rare among election officials: It hired a forensic firm to scour the technology for signs that hackers had hidden malware deep inside the coding supply chain.
The probe unearthed some unwelcome surprises: software misconfigured to connect to servers in Russia and the use of open-source code — which is freely available online — overseen by a Russian computer engineer convicted of manslaughter, according to a person familiar with the examination and granted anonymity because they were not authorized to speak about it.
The company that conducted the scan, ReversingLabs, has also warned about those issues in a blog post and a talk at a hacking conference last year, though it did not specify the state and the vendor where the issues were found.
New Hampshire officials say the scan revealed another issue: A programmer had hard-coded the Ukrainian national anthem into the database, in an apparent gesture of solidarity with Kyiv.
None of the findings amounted to evidence of wrongdoing, the officials said, and the company resolved the issues before the new database came into use ahead of the presidential vote this spring.
This was “a disaster averted,” said the person familiar with the probe, citing the risk that hackers could have exploited the first two issues to surreptitiously edit the state’s voter rolls, or use them and the presence of the Ukrainian national anthem to stoke election conspiracies.
The supply-chain scare in New Hampshire — which has not been reported before — underscores a broader vulnerability in the U.S. election system, POLITICO found during a six-month-long investigation: There is little oversight of the supply chain that produces crucial election software, leaving financially strapped state and county offices to do the best they can with scant resources and expertise.
The sun sets as sign holders stand outside a polling station in Brookline, New Hampshire, on Nov. 3, 2020. The recent supply-chain scare in New Hampshire underscores a broader vulnerability in the U.S. election system. | Joseph Prezioso/AFP via Getty Images
The technology vendors who build software used on Election Day face razor-thin profit margins in a market that is unforgiving commercially and toxic politically. That provides little room for needed investments in security, POLITICO found. It also leaves states with minimal leverage over underperforming vendors, who provide them with everything from software to check in Americans at their polling stations to voting machines and election night reporting systems.
Many states lack a uniform or rigorous system to verify what goes into software used on Election Day and whether it is secure. When both state and federal officials have tried to bring greater attention to these flaws, they’ve had to contend with critics who resist “federalization” of state election processes.
After Russia’s attempts to disrupt the 2016 presidential race through hacking and disinformation, the Obama administration declared state election systems to be critical infrastructure.
But eight years later, the fragility of that system remains a cause for alarm among those familiar with the cobbled-together programs that control voting in many states.
The fear, current and former election officials said, is less a far-reaching hack that flips enough votes to swing an election than small, localized errors or attacks that undermine confidence in the ballot — or empower dishonest candidates to mount legal challenges to the results.
“I don’t think I agree that we’re 100 percent in a better place” than eight years ago, said Vanessa Le, a former special adviser to Director of National Intelligence Avril Haines.
Eight years after Russia's attempts to disrupt the 2016 race, the fragility of elections remains a cause for alarm among those familiar with the cobbled-together programs that control voting in many states. | Lynne Sladky/AP
The most vulnerable targets
If the last two elections offer any indication, voter registration databases are one of the most vulnerable targets for foreign hackers because — unlike voting machines — they must be accessible over the internet in order to operate.
In 2016, Russian hackers probed election systems in all 50 U.S. states and breached voter registration databases in at least two, according to a bipartisan Senate Intelligence Committee report. Four years later, Iranian hackers penetrated inside an unnamed state’s database, then used data stolen during the hack to mount a targeted voter intimidation campaign, the Justice Department found.
In the worst-case scenario, hackers could manipulate a state’s voter list, adding fictitious people to the rolls, changing real voters’ information or directing voters to the wrong polling places on Election Day.
That might have only a modest or indirect result on the vote. Most state and county election officials have safeguards and back-up procedures in place, ranging from interstate voter registration tools that would be harder to fool to paper-printed voter rolls, provisional ballots, audits and hand recounts.
But Le said that efforts by allies of former President Donald Trump to deny the legitimacy of the 2020 election have made voters more likely to believe exaggerated claims of an Election Day hack — especially if the outcome isn’t to their liking.
“You don’t have to actually compromise something to drive hysteria,” said Le, who was the lead investigative counsel for the majority staff on the Senate Intelligence Committee report.
Many state and federal election officials insist there has been significant progress in securing elections since 2016.
State and federal officials are in more regular communication than they were eight years ago, when many states balked at the Obama administration’s initial efforts to warn them about the threat from Russia. The Cybersecurity and Infrastructure Security Agency, now the lead federal agency on election security, didn’t even exist back then.
Perhaps most importantly, more than 95% of U.S. voters now vote by hand or on machines that leave some type of paper trail, which officials can audit after Election Day.
That has led many U.S. officials to make bold claims about the security of the 2024 elections.
In January, Paul Nakasone, the then-head of the National Security Agency and U.S. Cyber Command, testified that this year’s election will be “the most secure to date.” And in May, the director of CISA, Jen Easterly, told the Senate Intelligence Committee that U.S. election infrastructure is the “most secure in history.”
But even as federal officials moved to fill the gaps of the past, new threats have emerged. Declassified intelligence community reports from the 2020 and 2022 elections found that a broader array of foreign adversaries — among them cybercriminal hackers and influence peddlers from China, Russia, Iran, the Lebanese militant group Hezbollah and Cuba — were seeking to sway the American electorate than in previous cycles.
Congress has appropriated just $625 million in election security funding for the states since 2018 — though roughly two-thirds of that bankrolled changes to voting procedures made during the 2020 pandemic.
“Congress has a terrible record here,” said Adrian Fontes, Arizona’s top election official, during an event on election security in Washington, D.C., in June. “The states hold elections for federal offices under federal rules on federal ballots and federal case law and federal everything — except federal dollars.”
More than 95% of U.S. voters now vote by hand or on machines that leave some type of paper trail, which officials can audit after Election Day. | Michael Ciaglo/Getty Images
A need for more oversight
Congress and the states have also failed to set rules of the road for election contractors — even in areas that foreign hackers are known to be targeting.
In May of 2021, President Joe Biden issued a sweeping executive order that required federal agencies to maintain a software ingredients list detailing where all the code they use comes from and whether it can be trusted. The mandate aimed to plug a gaping hole in the supply chain of U.S. networks that Russian spies had exploited in a landmark hack affecting nine federal agencies and roughly 100 U.S. companies the year before.
But that rule holds no sway over the states, meaning it does not apply to the myriad systems — from electronic poll pads to vote tabulators and voter registration databases — that millions of Americans now rely on to cast their vote each November.
On their own, most states lack the expertise and resources to enforce proper supply-chain security requirements, said John Sebes, the co-director of the Open Source Election Technology Institute. Those demands “don’t fit well with existing procurement processes, nor with the skill set of procurement folks” in the states, he said.
One concern among election watchers is that the intense and sometimes violent skepticism from certain Trump supporters has led officials to downplay issues with election technology.
Many election systems “are insecure in a number of different ways,” said Brian Gallagher, a cybersecurity expert who probes election technology to help companies fix bugs. But people don’t want to talk about that, he continued, because “they don’t want to give the quote-unquote crazies anything to go off of.”
Chris Krebs, the head of the Cybersecurity and Infrastructure Security Agency during the last presidential election, said he agreed with Easterly and Nakasone that the U.S. election system is far more resilient today than it was eight years ago. But he qualified his assessment by acknowledging that elections are becoming more dependent on technology, even as the digital threats from Russia and other foreign powers are growing.
“Are we where we want to be? No, I don’t think anybody’s going to tell you that everything is absolutely perfect,” he said.
“We felt the need to just do a hard look at the software code that the offshore resources had been working on,” Secretary of State David Scanlan said in the New Hampshire Statehouse this April. | Michael M. Santiago/Getty Images
A Granite State requiem
From the moment they began looking to upgrade their voter registration database, in May of 2021, election officials in New Hampshire knew they were up against the clock.
The state’s existing system — nearly 20 years old by then — was maintained by a vendor that New Hampshire wanted to change. And, given the timing of the state’s first-in-the-nation primary, the 2024 elections were breathing down the state’s neck.
“Time is of the essence,” reads a request for proposals that then-Secretary of State William Gardner issued in May of 2021, in which the state solicited a technology expert to help pick out a new vendor.
The vendor they would ultimately choose was WSD Digital. Founded only two years prior by a senior employee at the state’s old vendor, PCC Technology, WSD was quickly snatching up contracts from PCC on the basis of experience, competence — and, it seems, bold promises about its ability to churn out projects.
In a June 2022 assessment of a proposal WSD Digital submitted for a separate IT contract in the state of Maine, multiple state evaluators questioned the speed at which it was promising to deliver on the contract.
“Timeframe may not be believable,” one of them wrote in response to WSD’s submission.
Though the contract was not for a voter registration database, the evaluators noted that WSD had no plans to use a subcontractor — news that would also prove a surprise in New Hampshire.
David Scanlan, who took over as New Hampshire Secretary of State in January 2022, and his chief of staff, Dave Lang, don’t recall how they learned WSD Digital was using overseas coders to help build the state’s new voter registration database.
But they do recall what they did as soon as they learned about it between the summer and fall of last year: They read the White House’s executive order on cybersecurity, which motivated them to hire ReversingLabs to look under the hood at the software WSD was building for them.
“We felt the need to just do a hard look at the software code that the offshore resources had been working on,” Scanlan said in the first of two interviews in the New Hampshire Statehouse this April.
The issues discovered by ReversingLabs were concerning. They could have left the state’s voter registration database vulnerable to foreign hackers had the system been brought online without first being studied, according to the individual directly familiar with the probe and those briefed on the details by POLITICO.
That did not include the hardcoded Ukrainian national anthem, which posed no technical threat, but could have given fodder to conspiracy theorists.
The first of those risks stemmed from Microsoft software that had been misconfigured — probably by accident — to connect to servers in foreign countries, including Russia. The outbound traffic could have made it easier for hackers to identify and reconnoiter the system and slip past defenses deployed to protect it.
In addition, code for the database — which was not in use yet — included popular open-source software, core-js, that is overseen by a Russian national, Denis Pushkarev.
In an op-ed published last November that did not name New Hampshire or WSD Digital, a ReversingLabs researcher argued that using core-js in election technology was dangerous. Core-js included “callbacks” to Pushkarev’s personal website that could allow Pushkarev to pinpoint specific users of core-js, the article warned.
And the op-ed, which included a quote from the ReversingLabs CEO, suggested that Pushkarev’s criminal history and publicized financial struggles could make him susceptible to blackmail.
Though a reputable coder, Pushkarev warned publicly in early 2023 that sanctions against Russia over the war in Ukraine had cut off much of the financial support he was receiving for upkeeping core-js. In the same post, he also disclosed he served prison time following a hit-and-run incident that left one of the victims, a teenager, dead.
Dan Lorenc, an open source security expert and founder of supply-chain security start-up Chainguard, said his concern is that someone could “slip malware” through core-js to its users.
While the risk is modest for some of the several million applications that use core-js, Lorenc argued that election technology should be held to a higher standard.
“When it comes to security critical software, you have to be responsible for every single component in there, including the open source, and that’s kind of a gap that not a lot of people think about,” he said.
In a text conversation, Pushkarev called ReversingLabs’ warnings “stupid and unprofessional,” arguing that any effort to “inject anything malicious” into core-js would be noticed by its users.
He also argued that his legal troubles are behind him. “There are no levers of pressure on me from any special services,” he wrote, “moreover — any compromise of the project would mean the end of my career.”
WSD Digital and ReversingLabs did not respond to repeated requests for comment over the course of several months.
Scanlan and Lang emphasized that the voter registration database was not online at the time of the probe, and that many organizations, including the U.S. government, use foreign contractors for some IT systems.
They said they opted not to cut ties with WSD because the company was transparent after they confronted it, and the scan revealed no signs that the system had been tampered with.
“There was nothing alarming that we saw that would cause any red flag for us,” Lang said.
U.S. cybersecurity officials often argue the decentralized nature of the U.S. voting system insulates it from major hacks. | Montinique Monroe/Getty Images
Lessons for other states
WSD Digital is responsible for maintaining the voter registration database in just one other state, Vermont, meaning any incident involving its technology would have been limited.
But POLITICO found that there are no uniform or widely applicable laws or practices to police the use of overseas subcontractors in many aspects of election technology — let alone to understand which individual software components make up a piece of code.
The federal Election Assistance Commission, for example, sets voluntary requirements for vendors of voting machines and electronic poll books, but it has none for voter registration databases or other election-related systems, like those used to ship ballots to overseas service members and relay unofficial election night results.
And there are no national requirements for states to maintain software bills of material, or SBOM, a list of digital ingredients that helps users vet the bits of code — many open-source — that go into modern software programs.
States can write those types of requirements into contracting documents with vendors. But few do so, let alone enforce such provisions after the fact.
“Other than some of these very large population states, most don’t have the horsepower, the experience, to do the work on their own,” said one senior U.S. election official in a large state, granted anonymity because they were not authorized to speak publicly on the issue.
In a statement, Jeff Greene, the executive assistant director for cybersecurity at CISA, which is responsible for overseeing election systems, called SBOMs a “valuable tool” to help the manufacturers of software deliver secure products and respond to novel cybersecurity risks. But he cautioned that it will take time for under-resourced industries to implement them, and that they are not a silver bullet.
“The election community employs multiple layers of technical, physical, and procedural safeguards to ensure the security and resilience of election infrastructure in use across the country,” his statement reads.
Bryan Mills, the chief of staff in Vermont’s secretary of state’s office, said Vermont was informed about the outsourcing issue in mid-April, around the time POLITICO first reached out to it. They said they have not “had any concerns“ about the work that WSD Digital is doing and are adhering to supply-chain security practices set by the National Institute of Standards and Technology.
If supply-chain issues remain a black box for many U.S. states, it is nonetheless clear they are under growing threat from malicious hackers.
In the Russian hack of the U.S. government uncovered in late 2020, Kremlin-linked hackers spent months lurking within a Texas-based federal IT contractor, SolarWinds, before slipping malware inside a software update the firm shipped to thousands of customers, including many within the federal government.
And earlier this year, unknown hackers mounted a patient, two-year-long campaign to convince the maintainer of a popular open-source software library into ceding them control over it. Then, they abused their new position to send malware to a small subset of users across the globe.
“If you look at the aircraft industry, we can get the supply chain down to the bolt,” said Marc Rogers, a longtime hacker who has held senior cybersecurity positions at leading security and telecommunications companies. “We need to get to that place in elections, so we can properly audit the parties putting everything together.”
Eddie Perez, a board member at the Open Source Election Technology Institute, put it this way: “This is absolutely an appropriate area for there to be clear policy standards and policy preferences.”
Congress has appropriated just $625 million in election security funding for the states since 2018 — though roughly two-thirds of that bankrolled changes to voting procedures made during the 2020 pandemic. | Alex Wong/Getty Images
Caught between vendors
U.S. cybersecurity officials often argue the decentralized nature of the U.S. voting system insulates it from major hacks.
But the flip side of handing so much responsibility to the states is a persistent underfunding of critical election systems — leaving some software prone to error, dependent on unproven subcontractors or vulnerable to foreign hackers.
The market for statewide voting registration database vendors is especially unpredictable, according to election officials, since each has different rules about who can vote. That means software used in one state might not be viable elsewhere.
“There’s just no money in it,” said Stuart Holmes, the director of elections for Washington state.
A dearth of reliable vendors of voter registry lists appears to have forced the hand of officials in New Hampshire.
Scanlan said the state began looking to replace its prior voter registry system because the vendor who managed it “was not working out at all” for the state.
Scanlan did not go into detail on those problems or name the vendor, PCC Technology.
An independent review commissioned by the Vermont secretary of state’s office said that Vermont’s experience with PCC Technology “began to deteriorate” after the company was bought in 2016 and later underwent a corporate reorganization, at which point it was rebranded as Civix.
A former senior IT official in Georgia, another former Civix user, said the state had a similar experience with the firm and ultimately decided to tap a new, larger vendor for their voter registration database, customer service giant Salesforce.
Officials in Vermont and New Hampshire, by contrast, would both replace PCC Technology with WSD Digital, which was founded in just 2019, according to the company’s page on LinkedIn.
WSD Digital nonetheless had one major selling point: Its founder was PCC’s former chief technology officer and a well-respected former adviser to Georgia Gov. Brian Kemp.
WSD Digital did not reply to a request for comment.
In a statement, a spokesperson for Civix said it was “factually wrong” to say its takeover of PCC Technology had a negative impact on its products. They added that Civix’s new management made a decision to limit its investment in legacy technologies for performance and security reasons.
The market for statewide voting registration database vendors is especially unpredictable, according to election officials, since each has different rules about who can vote. | Ben Gray/AP
Transparency in an age of conspiracy
Greater scrutiny of every component of election software could generate tough questions for election vendors and state officials at a time when election skepticism is rampant.
Virtually no software is flawless, and the geographic location of coders is at best an imperfect proxy for security risk.
But opacity about the software supply chain unquestionably makes systems harder to defend. And outsourcing to certain countries can give fodder to conspiracy theorists.
“Outsourcing, even indirectly, to an international source is just asking for trouble,” said David Jefferson, an expert in election security and a computer scientist at the Lawrence Livermore National Laboratory.
In New Hampshire, Scanlan and Lang said they have now made a “policy decision” to require software ingredients lists from vendors of all their office’s IT systems, like the White House executive order requires of those in the federal government.
But Lang said they continue to “wrestle with” how to use that information to build credibility with voters — instead of fueling distrust or giving a road map that adversaries can exploit.
Scanlan echoed that idea. “The biggest challenge facing election administrators in this upcoming 2024 election cycle is trying to create a climate where we can restore the trust and confidence” in the election process, he said.
That job is challenging, he continued, because many of those systems now “use computer code and are highly technical.” And, he said, “things can go wrong.”
Politico (archive.ph)
By John Sakellariadis
2024-09-01 11:00:00GMT
In a six-month investigation, POLITICO found there is little oversight of the supply chain that produces crucial election software, leaving financially strapped state and county offices to do the best they can with scant resources | Joe Raedle/Getty Images
When election officials in New Hampshire decided to replace the state’s aging voter registration database before the 2024 election, they knew that the smallest glitch in Election Day technology could become fodder for conspiracy theorists.
So they turned to one of the best — and only — choices on the market: A small, Connecticut-based IT firm that was just getting into election software.
But last fall, as the new company, WSD Digital, raced to complete the project, New Hampshire officials made an unsettling discovery: The firm had offshored part of the work. That meant unknown coders outside the U.S. had access to the software that would determine which New Hampshirites would be welcome at the polls this November.
The revelation prompted the state to take a precaution that is rare among election officials: It hired a forensic firm to scour the technology for signs that hackers had hidden malware deep inside the coding supply chain.
The probe unearthed some unwelcome surprises: software misconfigured to connect to servers in Russia and the use of open-source code — which is freely available online — overseen by a Russian computer engineer convicted of manslaughter, according to a person familiar with the examination and granted anonymity because they were not authorized to speak about it.
The company that conducted the scan, ReversingLabs, has also warned about those issues in a blog post and a talk at a hacking conference last year, though it did not specify the state and the vendor where the issues were found.
New Hampshire officials say the scan revealed another issue: A programmer had hard-coded the Ukrainian national anthem into the database, in an apparent gesture of solidarity with Kyiv.
None of the findings amounted to evidence of wrongdoing, the officials said, and the company resolved the issues before the new database came into use ahead of the presidential vote this spring.
This was “a disaster averted,” said the person familiar with the probe, citing the risk that hackers could have exploited the first two issues to surreptitiously edit the state’s voter rolls, or use them and the presence of the Ukrainian national anthem to stoke election conspiracies.
The supply-chain scare in New Hampshire — which has not been reported before — underscores a broader vulnerability in the U.S. election system, POLITICO found during a six-month-long investigation: There is little oversight of the supply chain that produces crucial election software, leaving financially strapped state and county offices to do the best they can with scant resources and expertise.
The sun sets as sign holders stand outside a polling station in Brookline, New Hampshire, on Nov. 3, 2020. The recent supply-chain scare in New Hampshire underscores a broader vulnerability in the U.S. election system. | Joseph Prezioso/AFP via Getty Images
The technology vendors who build software used on Election Day face razor-thin profit margins in a market that is unforgiving commercially and toxic politically. That provides little room for needed investments in security, POLITICO found. It also leaves states with minimal leverage over underperforming vendors, who provide them with everything from software to check in Americans at their polling stations to voting machines and election night reporting systems.
Many states lack a uniform or rigorous system to verify what goes into software used on Election Day and whether it is secure. When both state and federal officials have tried to bring greater attention to these flaws, they’ve had to contend with critics who resist “federalization” of state election processes.
After Russia’s attempts to disrupt the 2016 presidential race through hacking and disinformation, the Obama administration declared state election systems to be critical infrastructure.
But eight years later, the fragility of that system remains a cause for alarm among those familiar with the cobbled-together programs that control voting in many states.
The fear, current and former election officials said, is less a far-reaching hack that flips enough votes to swing an election than small, localized errors or attacks that undermine confidence in the ballot — or empower dishonest candidates to mount legal challenges to the results.
“I don’t think I agree that we’re 100 percent in a better place” than eight years ago, said Vanessa Le, a former special adviser to Director of National Intelligence Avril Haines.
Eight years after Russia's attempts to disrupt the 2016 race, the fragility of elections remains a cause for alarm among those familiar with the cobbled-together programs that control voting in many states. | Lynne Sladky/AP
The most vulnerable targets
If the last two elections offer any indication, voter registration databases are one of the most vulnerable targets for foreign hackers because — unlike voting machines — they must be accessible over the internet in order to operate.
In 2016, Russian hackers probed election systems in all 50 U.S. states and breached voter registration databases in at least two, according to a bipartisan Senate Intelligence Committee report. Four years later, Iranian hackers penetrated inside an unnamed state’s database, then used data stolen during the hack to mount a targeted voter intimidation campaign, the Justice Department found.
In the worst-case scenario, hackers could manipulate a state’s voter list, adding fictitious people to the rolls, changing real voters’ information or directing voters to the wrong polling places on Election Day.
That might have only a modest or indirect result on the vote. Most state and county election officials have safeguards and back-up procedures in place, ranging from interstate voter registration tools that would be harder to fool to paper-printed voter rolls, provisional ballots, audits and hand recounts.
But Le said that efforts by allies of former President Donald Trump to deny the legitimacy of the 2020 election have made voters more likely to believe exaggerated claims of an Election Day hack — especially if the outcome isn’t to their liking.
“You don’t have to actually compromise something to drive hysteria,” said Le, who was the lead investigative counsel for the majority staff on the Senate Intelligence Committee report.
Many state and federal election officials insist there has been significant progress in securing elections since 2016.
State and federal officials are in more regular communication than they were eight years ago, when many states balked at the Obama administration’s initial efforts to warn them about the threat from Russia. The Cybersecurity and Infrastructure Security Agency, now the lead federal agency on election security, didn’t even exist back then.
Perhaps most importantly, more than 95% of U.S. voters now vote by hand or on machines that leave some type of paper trail, which officials can audit after Election Day.
That has led many U.S. officials to make bold claims about the security of the 2024 elections.
In January, Paul Nakasone, the then-head of the National Security Agency and U.S. Cyber Command, testified that this year’s election will be “the most secure to date.” And in May, the director of CISA, Jen Easterly, told the Senate Intelligence Committee that U.S. election infrastructure is the “most secure in history.”
But even as federal officials moved to fill the gaps of the past, new threats have emerged. Declassified intelligence community reports from the 2020 and 2022 elections found that a broader array of foreign adversaries — among them cybercriminal hackers and influence peddlers from China, Russia, Iran, the Lebanese militant group Hezbollah and Cuba — were seeking to sway the American electorate than in previous cycles.
Congress has appropriated just $625 million in election security funding for the states since 2018 — though roughly two-thirds of that bankrolled changes to voting procedures made during the 2020 pandemic.
“Congress has a terrible record here,” said Adrian Fontes, Arizona’s top election official, during an event on election security in Washington, D.C., in June. “The states hold elections for federal offices under federal rules on federal ballots and federal case law and federal everything — except federal dollars.”
More than 95% of U.S. voters now vote by hand or on machines that leave some type of paper trail, which officials can audit after Election Day. | Michael Ciaglo/Getty Images
A need for more oversight
Congress and the states have also failed to set rules of the road for election contractors — even in areas that foreign hackers are known to be targeting.
In May of 2021, President Joe Biden issued a sweeping executive order that required federal agencies to maintain a software ingredients list detailing where all the code they use comes from and whether it can be trusted. The mandate aimed to plug a gaping hole in the supply chain of U.S. networks that Russian spies had exploited in a landmark hack affecting nine federal agencies and roughly 100 U.S. companies the year before.
But that rule holds no sway over the states, meaning it does not apply to the myriad systems — from electronic poll pads to vote tabulators and voter registration databases — that millions of Americans now rely on to cast their vote each November.
On their own, most states lack the expertise and resources to enforce proper supply-chain security requirements, said John Sebes, the co-director of the Open Source Election Technology Institute. Those demands “don’t fit well with existing procurement processes, nor with the skill set of procurement folks” in the states, he said.
One concern among election watchers is that the intense and sometimes violent skepticism from certain Trump supporters has led officials to downplay issues with election technology.
Many election systems “are insecure in a number of different ways,” said Brian Gallagher, a cybersecurity expert who probes election technology to help companies fix bugs. But people don’t want to talk about that, he continued, because “they don’t want to give the quote-unquote crazies anything to go off of.”
Chris Krebs, the head of the Cybersecurity and Infrastructure Security Agency during the last presidential election, said he agreed with Easterly and Nakasone that the U.S. election system is far more resilient today than it was eight years ago. But he qualified his assessment by acknowledging that elections are becoming more dependent on technology, even as the digital threats from Russia and other foreign powers are growing.
“Are we where we want to be? No, I don’t think anybody’s going to tell you that everything is absolutely perfect,” he said.
“We felt the need to just do a hard look at the software code that the offshore resources had been working on,” Secretary of State David Scanlan said in the New Hampshire Statehouse this April. | Michael M. Santiago/Getty Images
A Granite State requiem
From the moment they began looking to upgrade their voter registration database, in May of 2021, election officials in New Hampshire knew they were up against the clock.
The state’s existing system — nearly 20 years old by then — was maintained by a vendor that New Hampshire wanted to change. And, given the timing of the state’s first-in-the-nation primary, the 2024 elections were breathing down the state’s neck.
“Time is of the essence,” reads a request for proposals that then-Secretary of State William Gardner issued in May of 2021, in which the state solicited a technology expert to help pick out a new vendor.
The vendor they would ultimately choose was WSD Digital. Founded only two years prior by a senior employee at the state’s old vendor, PCC Technology, WSD was quickly snatching up contracts from PCC on the basis of experience, competence — and, it seems, bold promises about its ability to churn out projects.
In a June 2022 assessment of a proposal WSD Digital submitted for a separate IT contract in the state of Maine, multiple state evaluators questioned the speed at which it was promising to deliver on the contract.
“Timeframe may not be believable,” one of them wrote in response to WSD’s submission.
Though the contract was not for a voter registration database, the evaluators noted that WSD had no plans to use a subcontractor — news that would also prove a surprise in New Hampshire.
David Scanlan, who took over as New Hampshire Secretary of State in January 2022, and his chief of staff, Dave Lang, don’t recall how they learned WSD Digital was using overseas coders to help build the state’s new voter registration database.
But they do recall what they did as soon as they learned about it between the summer and fall of last year: They read the White House’s executive order on cybersecurity, which motivated them to hire ReversingLabs to look under the hood at the software WSD was building for them.
“We felt the need to just do a hard look at the software code that the offshore resources had been working on,” Scanlan said in the first of two interviews in the New Hampshire Statehouse this April.
The issues discovered by ReversingLabs were concerning. They could have left the state’s voter registration database vulnerable to foreign hackers had the system been brought online without first being studied, according to the individual directly familiar with the probe and those briefed on the details by POLITICO.
That did not include the hardcoded Ukrainian national anthem, which posed no technical threat, but could have given fodder to conspiracy theorists.
The first of those risks stemmed from Microsoft software that had been misconfigured — probably by accident — to connect to servers in foreign countries, including Russia. The outbound traffic could have made it easier for hackers to identify and reconnoiter the system and slip past defenses deployed to protect it.
In addition, code for the database — which was not in use yet — included popular open-source software, core-js, that is overseen by a Russian national, Denis Pushkarev.
In an op-ed published last November that did not name New Hampshire or WSD Digital, a ReversingLabs researcher argued that using core-js in election technology was dangerous. Core-js included “callbacks” to Pushkarev’s personal website that could allow Pushkarev to pinpoint specific users of core-js, the article warned.
And the op-ed, which included a quote from the ReversingLabs CEO, suggested that Pushkarev’s criminal history and publicized financial struggles could make him susceptible to blackmail.
Though a reputable coder, Pushkarev warned publicly in early 2023 that sanctions against Russia over the war in Ukraine had cut off much of the financial support he was receiving for upkeeping core-js. In the same post, he also disclosed he served prison time following a hit-and-run incident that left one of the victims, a teenager, dead.
Dan Lorenc, an open source security expert and founder of supply-chain security start-up Chainguard, said his concern is that someone could “slip malware” through core-js to its users.
While the risk is modest for some of the several million applications that use core-js, Lorenc argued that election technology should be held to a higher standard.
“When it comes to security critical software, you have to be responsible for every single component in there, including the open source, and that’s kind of a gap that not a lot of people think about,” he said.
In a text conversation, Pushkarev called ReversingLabs’ warnings “stupid and unprofessional,” arguing that any effort to “inject anything malicious” into core-js would be noticed by its users.
He also argued that his legal troubles are behind him. “There are no levers of pressure on me from any special services,” he wrote, “moreover — any compromise of the project would mean the end of my career.”
WSD Digital and ReversingLabs did not respond to repeated requests for comment over the course of several months.
Scanlan and Lang emphasized that the voter registration database was not online at the time of the probe, and that many organizations, including the U.S. government, use foreign contractors for some IT systems.
They said they opted not to cut ties with WSD because the company was transparent after they confronted it, and the scan revealed no signs that the system had been tampered with.
“There was nothing alarming that we saw that would cause any red flag for us,” Lang said.
U.S. cybersecurity officials often argue the decentralized nature of the U.S. voting system insulates it from major hacks. | Montinique Monroe/Getty Images
Lessons for other states
WSD Digital is responsible for maintaining the voter registration database in just one other state, Vermont, meaning any incident involving its technology would have been limited.
But POLITICO found that there are no uniform or widely applicable laws or practices to police the use of overseas subcontractors in many aspects of election technology — let alone to understand which individual software components make up a piece of code.
The federal Election Assistance Commission, for example, sets voluntary requirements for vendors of voting machines and electronic poll books, but it has none for voter registration databases or other election-related systems, like those used to ship ballots to overseas service members and relay unofficial election night results.
And there are no national requirements for states to maintain software bills of material, or SBOM, a list of digital ingredients that helps users vet the bits of code — many open-source — that go into modern software programs.
States can write those types of requirements into contracting documents with vendors. But few do so, let alone enforce such provisions after the fact.
“Other than some of these very large population states, most don’t have the horsepower, the experience, to do the work on their own,” said one senior U.S. election official in a large state, granted anonymity because they were not authorized to speak publicly on the issue.
In a statement, Jeff Greene, the executive assistant director for cybersecurity at CISA, which is responsible for overseeing election systems, called SBOMs a “valuable tool” to help the manufacturers of software deliver secure products and respond to novel cybersecurity risks. But he cautioned that it will take time for under-resourced industries to implement them, and that they are not a silver bullet.
“The election community employs multiple layers of technical, physical, and procedural safeguards to ensure the security and resilience of election infrastructure in use across the country,” his statement reads.
Bryan Mills, the chief of staff in Vermont’s secretary of state’s office, said Vermont was informed about the outsourcing issue in mid-April, around the time POLITICO first reached out to it. They said they have not “had any concerns“ about the work that WSD Digital is doing and are adhering to supply-chain security practices set by the National Institute of Standards and Technology.
If supply-chain issues remain a black box for many U.S. states, it is nonetheless clear they are under growing threat from malicious hackers.
In the Russian hack of the U.S. government uncovered in late 2020, Kremlin-linked hackers spent months lurking within a Texas-based federal IT contractor, SolarWinds, before slipping malware inside a software update the firm shipped to thousands of customers, including many within the federal government.
And earlier this year, unknown hackers mounted a patient, two-year-long campaign to convince the maintainer of a popular open-source software library into ceding them control over it. Then, they abused their new position to send malware to a small subset of users across the globe.
“If you look at the aircraft industry, we can get the supply chain down to the bolt,” said Marc Rogers, a longtime hacker who has held senior cybersecurity positions at leading security and telecommunications companies. “We need to get to that place in elections, so we can properly audit the parties putting everything together.”
Eddie Perez, a board member at the Open Source Election Technology Institute, put it this way: “This is absolutely an appropriate area for there to be clear policy standards and policy preferences.”
Congress has appropriated just $625 million in election security funding for the states since 2018 — though roughly two-thirds of that bankrolled changes to voting procedures made during the 2020 pandemic. | Alex Wong/Getty Images
Caught between vendors
U.S. cybersecurity officials often argue the decentralized nature of the U.S. voting system insulates it from major hacks.
But the flip side of handing so much responsibility to the states is a persistent underfunding of critical election systems — leaving some software prone to error, dependent on unproven subcontractors or vulnerable to foreign hackers.
The market for statewide voting registration database vendors is especially unpredictable, according to election officials, since each has different rules about who can vote. That means software used in one state might not be viable elsewhere.
“There’s just no money in it,” said Stuart Holmes, the director of elections for Washington state.
A dearth of reliable vendors of voter registry lists appears to have forced the hand of officials in New Hampshire.
Scanlan said the state began looking to replace its prior voter registry system because the vendor who managed it “was not working out at all” for the state.
Scanlan did not go into detail on those problems or name the vendor, PCC Technology.
An independent review commissioned by the Vermont secretary of state’s office said that Vermont’s experience with PCC Technology “began to deteriorate” after the company was bought in 2016 and later underwent a corporate reorganization, at which point it was rebranded as Civix.
A former senior IT official in Georgia, another former Civix user, said the state had a similar experience with the firm and ultimately decided to tap a new, larger vendor for their voter registration database, customer service giant Salesforce.
Officials in Vermont and New Hampshire, by contrast, would both replace PCC Technology with WSD Digital, which was founded in just 2019, according to the company’s page on LinkedIn.
WSD Digital nonetheless had one major selling point: Its founder was PCC’s former chief technology officer and a well-respected former adviser to Georgia Gov. Brian Kemp.
WSD Digital did not reply to a request for comment.
In a statement, a spokesperson for Civix said it was “factually wrong” to say its takeover of PCC Technology had a negative impact on its products. They added that Civix’s new management made a decision to limit its investment in legacy technologies for performance and security reasons.
The market for statewide voting registration database vendors is especially unpredictable, according to election officials, since each has different rules about who can vote. | Ben Gray/AP
Transparency in an age of conspiracy
Greater scrutiny of every component of election software could generate tough questions for election vendors and state officials at a time when election skepticism is rampant.
Virtually no software is flawless, and the geographic location of coders is at best an imperfect proxy for security risk.
But opacity about the software supply chain unquestionably makes systems harder to defend. And outsourcing to certain countries can give fodder to conspiracy theorists.
“Outsourcing, even indirectly, to an international source is just asking for trouble,” said David Jefferson, an expert in election security and a computer scientist at the Lawrence Livermore National Laboratory.
In New Hampshire, Scanlan and Lang said they have now made a “policy decision” to require software ingredients lists from vendors of all their office’s IT systems, like the White House executive order requires of those in the federal government.
But Lang said they continue to “wrestle with” how to use that information to build credibility with voters — instead of fueling distrust or giving a road map that adversaries can exploit.
Scanlan echoed that idea. “The biggest challenge facing election administrators in this upcoming 2024 election cycle is trying to create a climate where we can restore the trust and confidence” in the election process, he said.
That job is challenging, he continued, because many of those systems now “use computer code and are highly technical.” And, he said, “things can go wrong.”
