Quality Coding Takes A Break For The Holidays. But Why?

by Fergal Glynn, Director of Marketing, VeracodeI recently read a blog post by CloudFlare and Shawn Graham that asked a fantastic (and timely) question: “Do Hackers Take The Holidays Off?” CloudFlare sees traffic for hundreds of thousands of websites and was able to answer the question. They looked at the average percentage of requests that constitute threats, graphed the deviation, and then overlaid any events happening on those days. Their conclusion: it depends on the holiday.

Fergal Glynn

I recently read a blog post by CloudFlare and Shawn Graham that asked a fantastic (and timely) question: “Do Hackers Take The Holidays Off?” CloudFlare sees traffic for hundreds of thousands of websites and was able to answer the question. They looked at the average percentage of requests that constitute threats, graphed the deviation, and then overlaid any events happening on those days. Their conclusion: it depends on the holiday.

This got me thinking. “Wouldn’t it be interesting to see if holidays or the time of year affects code security quality?” At what time of year is the most insecure software written? During the summer? over the holiday period? during March Madness? This summer I saw a few people at the beach with their laptops, I wonder if any of them were coding?

What I looked at
My company, Veracode, sees the security quality of the thousands of applications we test. I wanted to see if there is any seasonality in code security quality. I decided to focus my analysis on applications in early stages of the development life cycle, in other words, those that were actively in development, or undergoing Alpha and Beta Testing. Early in the development process, I reasoned, there would be a relatively quick turnaround from when the code was written to when it was scanned. In other words, I’m assuming that during the early stages of a development project, code is written and scanned within the same calendar month. I looked at application size and a roll-up of the total quantity of flaws per application.

What’s Normal?
To do a comparison, you first need to know what normal looks like. Therefore I looked at the thousands of alpha and beta-stage applications Veracode scanned over the past couple of years. I saw an average flaw density of 24 flaws per megabyte of executable code and a median flaw density of 3 flaws per megabyte of executable code. If you are interested, we talk more about flaw density observations in State of Software Security Version 3.

The Results

For the time period I looked at (the last 24 months), January through September is relatively flat and in line with the average flaw density. Then, there is a big bump in flaw density in October and November. Things begin to settle down once we go into December. The jump in application flaws is easy enough to spot. But what could cause this? Some of it could be seasonal. Maybe the build up to Thanksgiving has developers distracted? Are developers adjusting after the Summer break when “the living is easy” and the roads are quiet? Fall brings the extra pressure of dropping kids at school and rushing in the evenings to pick them up after sports. There is also the added pressure to produce a high volume of code to meet end of year deadlines and releases. Scientific literature is full of studies on the effects of stress on higher cognitive activities, and its reasonable to assume that application developers, like most of us, may respond to added pressure by making more mistakes in the code they write.

What do you think? Is the quality of code development seasonal?

For more security intelligence, read the results of our latest State of Software Security report. (You can download the report here.)The report provides security intelligence derived from multiple testing methodologies (static, dynamic, and manual) on the full spectrum of application types (components, shared libraries, web, and non-web applications) and programming languages (including Java, C/C++, and .NET) from every part of the software supply chain on which organizations depend.

Suggested articles

Discussion

  • Elmo on

    Add another variable -- proximity of release date. In my experience there is often a push to have a release ready to go by the 1st of the year and there seems to be a concomitant decline in testing standards which allows more defects to leak into production code.
  • Anonymous on

    Bar graph would have been a better choice.
  • Greg Fortune on

    Why is there some a large variance in data and why is that not accounted for?  With a mean of 24 and a median of 3, that means you must have a large number of outliers or a couple of huge outliers.  If those outliers happen to submit code during the months in question but not much code during the rest of the year, that would heavily skew the results during those months.

    Does the general result hold true regardless of the client?  What accounts for the variance?  Consider breaking the results out by client to see if any of the large flaw density clients match the current results.  It would also be fascinating to see how flaw density varies by client demographic (company size, programming language, product market, etc).

  • Anonymous on

    ning.spruz has a totally unoffensive new quality control code that debugs code with no compiling.

     

  • Chris McKay on

    A comparison with southern hemisphere development would be interesting (if an equivalent testing facility could be found). While the southern hemisphere (mostly) has a build up to Xmas in December, January/Febrary is the summer break.

    C

  • Anonymous on

    Perhaps it's that companies often do new hires in the summer months?

  • Rdoetjes on

    I think statistically this could be a bit out of whack.

    You have on average 3 errors (a relatively small number) but most companies only have 1 major release a year planned at the end of the year, so the more submissions could easilt deviate your normal median. You would need to also include the number of Megabytes of code that is tested in your analysis. Only then you can have a more representable analysis.

  • Anonymous on

    Could the decline in daylight hours - especially for developers in northern latitudes - have some bearing too? I know that Seasonal Affective Disorder (SAD) can take its toll here in Canada...

     

  • Fergal on

    Greg, on your final thought, the mean and median I show was calculated using the entire 24 month data set. I did not break it down to a month to month figure. I agree that I should show the changes in mean and median on a month to month basis. I will add that to the next iteration. 

Subscribe to our newsletter, Threatpost Today!

Get the latest breaking news delivered daily to your inbox.