Jump to content

Hornbill testing before releasing updates?

Recommended Posts

Hi all.

I work in IT for too much long. Yes, too much really (punch cards still existed "live" when I began).

I remember the first times MS Windows released patches or new Windows versions, I was enthusiastic and applied those patches or updated my OS right the way. Then, after applying those patches or installing a new OS for a few times, I've learned that it would be better to wait until all the bugs were solved and decided to install all the stuff after all the issues were solved after being handled by other people.

In Hornbill it happened the same while we could apply the patches and updates when we wanted to do so. I always waited some days before applying a new patch or release.

Now, all the patches and updates are mandatory, automatic and unattended and you know what? I came back to those times when I installed a new MS Windows patch or version right the way.

Just to say that, whenever I receive an email from Hornbill notifying me that a new patch has been applied, my immediate thought is: now what? Which new issues will appear with this new release?

Please, Hornbill: make more testing in new releases and patches before making them live. Create a beta testing environment for your main customers to test the new releases and patches before applying them live.

Your customer base will thank you.



Link to comment
Share on other sites

@Alberto M

Thanks for your comments and thoughts.  I wanted to reply because I think its worth keeping things in context.  We at Hornbill more than anyone feel the pain when things go wrong, today and yesterday have been particularly difficult days to manage because a couple of impactful things have gone wrong, and mostly when things go wrong these are due to change, the big thing for most IT people to face.  Its a balance, change and risk breaking things, or leave a long and become outdated.  

With regards to testing, you said "make more testing in new releases and patches before making them live" and what I would really like to point out is we do, in fact we put more combined effort into testing and test automation than any other single development activity, testing is our number one priority.  However, Hornbill has many millions of lines of code, there is a lot of moving parts and sadly despite our best efforts and often immense levels of test coverage and daily effort, things still do get missed, go wrong or sometimes we experience unexpected external factors. 

In almost every occasion we have a problem, we count our recovery times in minutes. Sometimes it takes a little more time to understand the nature of the problem we are seeing, and it definitely takes a little time to put in place a good an reliably recovery strategy, we always have to balance the quality of the resolution with the speed of resolution and we need to do that effectively and objectively in an environment that suddenly becomes pressured and hostile as customers rack up the calls for help and resolution. 

I would ask you and our customers generally to judge us on our overall performance and the overall quality of our service over a reasonable period of time, rather than on the bad day, so to speak, I would hope most customers wad judge us favourably but I will happily admit we are not perfect, this do go wrong from time to time. 

In terms of todays problem for example, we took the decision to roll back to a previous binary version, that is not something we have done very often at all. Today was different, there were actually four completely independent issues we identified, two of which we had never seen before and did not have test cases in place for, one we had never seen before what we have never even anticipated and another was of our own making, that we should have caught in testing but did not. 

Every time we see such errors, the very first discussions we have internally is "what tests do we need to add to ensure this case is never encountered in production" and every time we add these cases as a priority over fixing the code, then we fix the code knowing there is a test to verify the fix, and to guarantee no re-introduction.  The point is, we do take testing very seriously. 

I understand that none of this helps any customer who is depending on Hornbill to do their day job, or where it is mission critical, and I can only apologies to each and every customer where there is impact, both personally and on behalf of Team Hornbill, but I would hope our customers would generally understand that when we make changes, and when that leads to such problems, firstly we don't do that on purpose, and b) we will do whatever it takes to resolve that issue.  

The alternaive to our current approach is to move away from CD and instead, say do bi-annual or annual releases, forcing our customers to do upgrades like most of our competitors do in this spec.  I personally hate that idea because that is essentially making our customers responsible for the upgrades.  I believe strongly that most customers, on-balance, would rather see positive change and evolution of the tool they rely on daily, and to avoid such big upgrades. 

I hope that makes some sense to you





  • Like 3
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...