Originally published July 1, 2016, DevOps Buyer’s Guide, SD Times
No matter how good your perimeter security is, experts agree: Your system has been breached, whether you know it or not. The costs of security flaws—cybersecurity expert Joe Franscella calls them “The Five Horsemen of the Internet Apocalypse: Scam, Extortion, Embarrassment, Theft and Death”—are enormous. So why don’t we consider security a first-class citizen in DevOps?
“Security is still one of the last places where that archaic approach of development handing off the software to a different team and walking away still reigns,” said Tim Buntel, vice president of products for XebiaLabs. “Secure software is just good software, and good software is secure software. Everything that we’re doing in DevOps is allowing us to build better software at scale and release it faster.”
But building security in from the start rather than tacking it on at the end “takes a lot more than getting the security guys to attend standup meetings,” Buntel continued. All too often, he said, even large regulated industries have a tiny cadre of security experts vetting a fraction of a huge portfolio. And these enterprise static analysis runs can take days.
(Related: Putting the test back in DevOps)
For this DevOps Buyer’s Guide, XebiaLabs, along with Microsoft, Dynatrace, CollabNet, Appvance and CloudBees, spoke with SD Times about best practices for Rugged DevOps (a term coined by DevOps author Gene Kim) or DevSecOps. All agree that the time is ripe for adding security scans and stack analysis earlier in the DevOps workflow and mitigating malicious activity. To paraphrase Bruce Schneier, software security may be getting better—but it’s getting worse faster.
Is there a move toward Rugged DevOps?
The open-source Jenkins Continuous Integration (CI) platform has had a pivotal role in the DevOps tool chain and even the cultural lore. CI today, however, is just one piece in the DevOps Continuous Delivery pipeline. Sacha Labourey, CEO and founder of CloudBees, which commercializes Jenkins, has seen his own company paralleling the evolution of DevOps.
“We saw it through the fast adoption of Continuous Delivery, which led to increasingly sophisticated ‘flows’ being implemented on top of Jenkins, he said. “Consequently, about two years ago, we initiated the development of what’s now known as Jenkins Pipeline, a core feature of the newly released Jenkins 2.0. We also see an increased use of Docker, since it makes it very easy to have the exact same container used in development, testing and production. To that end, we also contributed a lot of features back to the Jenkins community.”
With a large target on its back, Microsoft has focused on security for years. Today, CEO Satya Nadella encourages a “live site culture,” or production-first mindset.
“Part of that mindset is saying, ‘Anytime I see something go wrong, it’s an opportunity for learning. Anytime I see a breach in security, I need to ask what can I do so this doesn’t happen again.’ How can we shorten our detection time, improve mitigation, and limit the radius of users affected?” said Sam Guckenheimer, product owner for Visual Studio Cloud Services at Microsoft.
Those questions are more common today in part thanks to the movement that began 15 years ago, said Guckenheimer. “You had in 2001 the Agile Manifesto: Build software in potentially shippable increments. In 2007, you had 10 deploys a day at Flickr. I think DevSecOps is next,” he said.
What’s holding us back is cultural, but it’s also technical. “Part of the problem is that most security tools are too slow to work in a Continuous Integration model,” said Guckenheimer. “Checkmarx is probably the tool that’s cracked that first. Ideally, you want to be able to have your code scanned as part of the pull request in the Continuous Integration flow, and that’s just not practical with most tools that exist.”
Increasingly automated software delivery tool chains and pipelines can become critical assets similar to the “infrastructure as code” concept. But all the vendors interviewed agreed that Rugged DevOps is primarily a cultural effort. “Tooling needs to help make that happen, but won’t lead it,” said Labourey.
Combatting apathy, enforcing empathy
At Microsoft, one method of instilling application-level security in team culture is via war games waged on software in production. Red teams are attackers, blue teams are defenders, and a referee verifies findings and lets the blue team know if they have thwarted a red team attack or a discovered a genuine external threat. “There are rules of engagement: You can’t compromise the customer SLA (service level agreement), you can’t exfiltrate data, you can’t damage the database or bring down the service, but as the red team, you prove that you can get right to that point,” said Guckenheimer.
While some Microsoft teams, such as Azure public cloud, do war games continuously, “For us it’s more like quarterly. We do not have a permanent red team; we rotate them,” said Guckenheimer. “We do have a permanent blue team who are real defenders. The goal is to make them better. When you do a retrospective on these things, everyone comes and listens.”
As a result of war games, Guckenheimer lives by basic security rules:
• Use just-in-time administration
• Use multifactor authentication
• Manage and rotate secrets via key vaults
• Use a DevOps release pipeline
• Destroy compromised instances
• Don’t tip your hand to attackers
• Segregate domains; don’t dual-home servers
• Use different passwords
• Don’t use open file share
• Assign only one admin per workstation
• Think before clicking links (to stop phishing)
“Shift left” is the mantra for DevOps, and security is no exception, according to Appvance CEO Kevin Surace. “DevOps means shifting everything left, including app penetration and DDoS (distributed denial of service) testing,” he said.
“It’s great to do once-a-year tests outside or have a security center of excellence. But any build can and does add security risks, which need to be found and evaluated. Source-code scanning should always be run, but you won’t be able to find everything until you execute use-case-driven [app penetration testing] at every build or at a minimum for each release candidate.”
War games and penetration tests are fun, but how do you create that empathetic connection between development and security? One controversial technique used to create empathy is giving pagers to developers so that they feel the pain of late-night operations snafus. Is there a similar approach that could happen with security?
“I don’t like the pager idea,” said Andreas Grabner, a developer advocate for Dynatrace. “What I like is that the team itself, we don’t deploy after 1 p.m. Why? Because we monitor. We still have three to four hours before we go home to figure out if that was a good or bad deployment. If at 1:30 p.m. we see an impact on end users, then we can say we introduced a bad deployment, or we roll back to a previous state.”
A pipe dream for low-tech companies
“These days, every business in the world relies on software to do business, but only a small percentage are actually software companies,” said Grabner. “They have to become software-defined businesses, but there’s not enough talent in the world to go around. That’s why the only way out of this is with solid automation and detecting all these problems in your pipeline.”
But is a Continuous Delivery pipeline with security gates even possible for many organizations? “That’s what everyone wants, but it’s very far away,” Grabner admits. Test automation is still in its infancy. “But the awareness that quality needs to be a core part of development is extremely increased.”
The concept of quality gates comes from Toyota’s iconic production line innovations, where any worker can stop the line if a quality check fails. In the case of software pipelines, according to Grabner, the automated quality gate can track architectural metrics such as the number of database queries executed, the number of web services calls, memory usage and more. “What we do with these quality gates is, we are detecting regressions caused by changes pushed on the pipeline: Something has changed from the way it used to be,” said Grabner.
“This feature consumed x amount of memory, and now it consumes y. If it has a negative impact, we need to stop the pipeline. This is what we call metrics-driven Continuous Development.” Teams can also aim to improve the mean time from finding the issue to fixing it.
Monitoring deployed software is key. “We always also combine it with synthetic monitoring. That means if I deploy a new feature, I can monitor how real users use that feature, while synthetic monitoring checks the feature every 10 minutes,” said Grabner.
Securing components not your own
What about the code you didn’t write? How do you add security to 90% of code that is third-party components or open source? “Never just assume security and do use a governed adoption process and DevOps tools that support that,” said Ward Osborne, information security officer at Collabnet. “Limit your use of third-party to what you need. Test it. Disable all the stuff you don’t need at the start. Go through security testing.
“Back to the empathy question, if open source is important to your work, then it is good to establish relationships with the creators of the code and help them make it more secure—that is always a plus. Going forward, what we will see, as security becomes more integrated into development processes, is that open-source code will become more secure as well.”
There’s no excuse for playing fast and loose with frameworks, components and libraries, according to Guckenheimer: “Anyone worth their salt these days will only use trusted libraries. There are companies that specialize in that: WhiteSource and Black Duck and Sonatype, who will try to ensure that you are using trusted versions.”
Further, the pipeline also helps enforce policies around trusted components. “Presumably, you don’t consume anything, by policy, that isn’t acceptable because of known vulnerabilities and unsuitable maintenance on its side. These policies are reasonably easy to enforce with tooling,” said Guckenheimer.
Automation: If it hurts, do it more
The vendors surveyed agree that one baby step must happen before achieving the nirvana of a perfectly built app: test automation. “You need to aim for total automation, and if it hurts, it probably means you need to do it more, not less,” said Labourey. “That’s the only way to reliably and deterministically build products and make sure nobody can intrude through that process.
“Some initial reactions lead to thinking that automation will reduce security and just accelerate the deployment of buggy and insecure applications to production. Au contraire: If the right process is applied, automation behaves exactly like a boa constrictor, increasingly constricting any space left for human error, making it possible to reliably inject quality and security improvements through the process.”
But it will take a cultural change to get those who talk about DevOps and those who talk about security to do the unthinkable: eat lunch together at the next RSA conference. Creating virtuous loops, training consistently around phishing and other exploits, employing quality gates, scanning code and searching for anomalies is never-ending, but you’d better get good at it: It’s no longer optional.