Google Volunteer Team Patches Thousands Of Open Source Projects

A Google security researcher this week offered the first details on an effort by a 50-member volunteer team at the company last year to help patch more than 2,600 open-source projects against a critical vulnerability in a widely used Java process.

A consultant at FoxGlove Security first drew attention to the so-called Java deserialization issue in November 2015 when he demonstrated attacks exploiting the vulnerability in WebLogic, Websphere, JBoss and other middleware products.

The demonstration prompted vulnerability disclosures from multiple vendors, including Oracle, IBM and Cisco, and led many security researchers to estimate that millions of applications—commercially developed, custom-built and open source—were susceptible to the issue.

Google open source security

About five months after the FoxGlove security consultant’s demonstration, a Google engineer noticed that several open-source projects were still using versions of the Apache Commons Collection Java library that was the source of the problem.

Since fixing the issue typically required only a single line of code, the engineer began updating codebases of vulnerable open-source projects via the GitHub user interface, initially on her own and then with help from others. “As more work was completed, it was apparent that the problem was bigger than we had initially realized,” Google software engineer Justine Tunney said in a blog this week.

For instance, in the case of many open-source projects, it wasn’t enough to patch just the one project, but every other project that depended on it, and those downstream of those dependencies as well. “We came to the conclusion that, in order to improve the health of the global software ecosystem, the old version of Collections should be removed from as many codebases as possible,” Tunney said.

Tunney and other volunteers used Google’s BigQuery data analytics engine to write a SQL query that searched all of the public code on GitHub in a couple of minutes and showed some 2,600 projects that used the vulnerable libraries. 

Instead of simply alerting the project owners about the issue, the Google engineers decided to update the code on their own considering the severity of the flaw and how widespread it was.

Volunteer programme

Typically, when Google engineers want to make changes to code bases owned by hundreds of developer teams at the company, it uses an internally developed tool called Rosie to accomplish the task, Tunney noted. But since no equivalent tool was available to automatically patch the vulnerable open-source projects, Tunney and others formed a task force comprising about 50 volunteers to patch the projects the hard way.

The effort was christened Operation Rosehub and was organized mainly through company mailing lists, Tunney said. Working together, members of Operation Rosehub sent out patches to the open-source projects in a few weeks.

“There was no mandate from management to do this—yet management was supportive. They were happy to see employees spontaneously self-organizing to put their 20 percent time to good use,” she noted.

Only open-source projects on GitHub that directly referenced the vulnerable libraries have been patched. Many other projects and applications remain vulnerable to the threat. As one example, Tunney pointed to a recent ransomware attack against the San Francisco Municipal Transportation

Agency where attackers took advantage of the Java deserialization bug to gain access to critical systems and infrastructure.

“We want to draw attention to the fact that the tools now exist for fixing software on a massive scale, and that it works best when that software is open,” she said.

Originally published on eWeek

Quiz: What do you know about Google and Alphabet?