Achieving Code Compliance in an Agile Environment

Thursday, July 18, 2013

Phil Cox


IT shops today implement speedy and dynamic application development techniques to meet business demands. You can’t visit an IT-related website these days without being peppered with such terms as cloud, agile, DevOps, and continuous deployment.

Being agile to meet business needs is the right idea, but when you try to take that agile environment and put it in front of auditors who are used to concepts like waterfall, security architecture reviews, and change management boards, you get confusion and misunderstanding which complicate the audit process.

How can you maintain the benefits of agile software development yet implement controls around the process to meet audit requirements?   

To meet compliance requirements for code development, you generally need the following foundational components:  

  • Software development lifecycle (SDLC): A documented version of how you will develop software in a manner that ensures a certain level of quality in that code.
  • Ticketing system for evidence collection: All compliance regimens require documentation and tracking, and a ticketing system provides a way to meet this requirement while automating the process.
  • Source code control system: A system to provide appropriate tracking of changes as well as rollback functions.
  • Mechanism to perform independent code review and testing: Automated or manual tools and processes that can be used to ensure a level of code quality that is consistent with the service you are delivering.
  • Process for changes before deploying code to production: Formal approval process for all changes that will end up being deployed to the production product.

An Agile Process Under DevOps In a DevOps-oriented environment — one with no separate operations team — you still need to code review and test and get approval before pushing code to production. This is where it gets tricky. In such a shop, you could specify that all developers have authority to promote to production after an independent code review and test (you can’t get away without this). It would be a business and policy decision to allow any or all developers to promote to production, and one that you would need to justify to your auditors.   To meet almost any widely applicable compliance requirement (such as PCI, SSAE16, and so on), all code must be both reviewed and tested independently — that is, not by the developer who wrote the code. A single person can be both the reviewer and tester. So in a DevOps environment that uses Scrum, the process to meet this requirement would look like this:

  1. The sprint team creates a story about what to do in the agile tool (a ticketing system, for instance).
  2. A story/task is allocated to a team member (developer).
  3. The developer tags code changes with the ticket number related to the story.
  4. Once the changes are done, the developer puts the story/task into review/test state, and another developer picks it up, or, ideally, the ticketing system uses a workflow associated with it to route the task to the appropriate developer.
  5. Once the testing succeeds, the task is marked complete.
  6. The developer then promotes the story/task to production.

If you have implemented adequate automation and tracking tools, and all of your developers have the requisite skills to deploy production code, then you should be able to make a business case that they be able to do so.

Don't overlook the requisite skills part of that last statement. In a true DevOps environment, you will need to ensure that the developers deploying to production are indeed qualified and authorized to approve the changes they are deploying. You may be able to justify having a junior developer promote to production if you have an adequate level of expertise in the review and test stage. How We Do It at RightScale It’s not trivial to create a bulletproof, robust, and consistent process that will pass an audit. Saying that you do it is easy, but evidencing it is the hard part. In particular, implementing the code review and testing process is the most difficult part of development and compliance in agile environments.

At RightScale, we embrace the agile methodology through Scrum, but we have not taken agile all the way to a DevOps delivery model. Thus we have a bit more “tradition” in our development practices. Software Development Lifecycle Our SLDC is defined in our RightScale Agile Manifesto (RAM) — think Agile Manifesto++. The RAM covers the following:

  • Anatomy of a release
  • Backlogs
  • Planning
  • Commit and branching standards
  • Sprint development capacity
  • Sprint QA engineer concerns
  • Scrum lifecycle
  • Sprint metrics
  • Retrospective
  • Regression
  • Integration, release candidate, and release branches
  • Release sequencing
  • Documenting story requirements
  • Sprint end cleanup
  • Feature freeze/branching date
  • Engineering escalation process
  • Demos
  • Development standards

Stories and Ticketing For each story that is to result in a code change, a developer creates code review (CR) and QA subtasks in the agile tracking tool. The code cannot be merged into the master source code branch and ultimately deployed to production unless the CR and QA subtasks have been completed. Development The development flow is as follows:

  1. When code is submitted to our version control system, the event is analyzed in real time. We use an automated task to ensure that each commit is associated with a legitimate story/task, and generate an alert if it is not.
  2. The commit message is verified to ensure that it contains the prerequisite ticket reference in the desired state. Our agile tracking tool creates a unique ID for each story.
  3. When a story is complete, we perform a merge into the master source tree.
  4. The merge is verified that it is complete and done by an individual who is authorized to perform the merge. In the event of an unauthorized merge, the engineering director or team lead determines the cause of the error and makes appropriate corrections.
  5. During a release, developers cut a release branch and perform final testing, and our operations team deploys the release into production.
  6. We have a scheduled audit that covers any abnormalities in the real-time process.

Note that the checks are not all devoted to compliance, but also to improving engineering visibility and ultimately efficiency. We can use them to see if someone had no time allocated to work they did, or worked on a deleted task. We also do more traditional sweeping validation because real time is subject to blips beyond our control. Summary By using our tools and implementing processes, we ensure that every change:

  • Is documented via a story or task ticket
  • Has undergone independent code review
  • Has undergone independent testing
  • Is approved for deployment to production

Further, we have evidence of each of the steps for proof audits.   In some agile environments, managers may get pushback from developers who feel they are being constrained by the rigorous compliance processes, but these controls help improve the quality of code, and that is hard to argue against. Developers who think their code does not need review or test are fooling themselves, and will likely be the cause of a company’s major breach.  

None of this effort to ensure compliance is free — it costs time and effort. However, using automation in the validation process can significantly minimize the impact. It costs, but it is worth the cost, and the benefits you gain more than outweigh the drawbacks in the long run. How much would a code review of the LinkedIn password storage have saved that company in the long run?  

The bottom line is that you can have compliance in agile environments. Take the time to do it right and you’ll reap the benefits of both compliance and agile development.   

This article was co-authored by George Goodyer, senior director of software development at RightScale. Cross Posted from the RightScale Cloud Management Blog 

Possibly Related Articles:
Cloud Security
Development SLDC Code Compliance Agile Environment
Post Rating I Like this!
Kylie Wilson I think agile is key to successfully implement projects. I got my Scrum Master certification done and now working in a project using Scrum. and I am loving it :)
Amani Phoenix Agile certifications are a very good way to gauge an individual’s knowledge of what Agile really is and how Agile Principles are implemented across different scenarios. It is very important for an individual to participate in a good training session ahead of any Agile certification e.g. PMI-ACP™ by PMI, Scrum Master Certification (SMC™)by SCRUMstudy, The Agile Project Management™ (AgilePM®) certification by APMG etc.
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.