Release and testing procedures (in higher education)
Illya posted some thoughts on Agile Release & Testing Procedures and instead of writing a big long comment I figured it was worth a blog post. At the University of Waterloo I have had experience deploying a number of different applications for a variety of audiences… it is next to impossible to get all the details in a post but here is the general truth: there are no enforced institutional wide procedures for web applications. You might think the lack of procedures is bad but it is a result of the relatively low risk environment (even though the campus community has a low tolerance for bugs and changes). There are rarely formal teams of developers, it is mostly the loan coder building a specialty application – enforced procedures would frustrate them.
When you are dealing with a simple web page, say the uni home page, I have essentially covered the typical user acceptance, performance, and stress tests when the page goes live. I go through the gamut of web browser testing, try some OS variations out, and then get it out there. There is a relatively low risk here as the users don’t interact with a database or a whole heck of a lot client side. Once rendering issues are dealt with, it is pretty much unlikely to have other issues. This is with 30 000 or more people seeing it within a short period of time too. I had relative success but I think it was more luck and the fact we kept web pages simple.
Stepping up the development a bit, throw in a Ruby on Rails or PHP application. My testing procedures involved pretty much the same as the web page testing: poke away at it, fix bugs as they appear, and get it ready to go off of the development server to production. We (co-op student and I) never really sat on changes very long. The thinking was that if it went bad on the production end we just roll back the version, fast. When I made the jump to Ruby on Rails development with Capistrano and SVN that became so easy it was scary. On many occasions we had new versions going up two or three times a day. Minor changes, but they add up. This meant a lot of bugs made it out to the community version but as a whole the community appreciated seeing the progress. Our harshest critics were few and usually the type of people that would sit on things until they are perfect, the web is never perfect.
Now I find myself in the .NET/C# development world. I am happily hacking away at the JavaScript on the front end but I still live in the development environment. Here we have a solid team, a lot of developers, some serious tools, and totally different requirements from the client relationship/expectations end. At the moment we are doing limited testing that makes sure it works and then pushing it to an environment that a group has a ‘sanity check’ and gives us feedback. Releases are going out on a weekly build routine with a daily routine for an internal release. The whole process is evolving as we go but in a very general sense we are aiming to maintain a weekly build schedule for one set of users, daily internally. Our goal is to not leave the application in a non-working state and at any time the build could go live. This habit takes time to develop though… I don’t expect us to be in the groove until over the summer.
That is the nutshell version of what I have had experience with, I suppose it is Agile without the buzz terms. Personally I don’t see a reason why any web application couldn’t work on a daily build process. If you break the big change down to a lot of little changes you reduce the risk of breaking it and you ensure stability (so the theory goes). The problem is that in order to break a big thing down to a bunch of little things you need to take the time to talk it out, plan it out, and scope out what goes into a big thing. It is a way of thinking and it doesn’t happen overnight, most people need experience thinking that way.
I am interested to know what other higher education folks are doing with release and testing procedures.