[NOTE: This post is not for developers, although I would encourage developers to read it. This post is for my friends who are business owners.]
Last night I wrote the info@ email address on a website and asked a questions about their product. They politely responded with two links to their website. This morning however, when I went to visit them, I was greeted with the message above. It’s now three hours later and the site is still “experiencing difficulties”. (that is developer speak for DOWN) I started poking around to see who built the site. They are a local company and I assumed their vendor was also. A quick whois showed that their site was built/hosted, not by a local development shop where they had a personal relationship with the development team, but by a company that specialized in their particular “vertical”. Checking the vendor’s website, it was up but all of their “featured sites” were throwing the same error. Obviously they had rolled an update of their system into production and it was broken. This problem can be prevented by business owners insisting on – and paying for – “Best Practices” in their software vendors.
This is not the first time I have run across this so I wanted to point it out to all my friends who are business owners. If you are considering turning your website over to a company that specialized in a vertical like this, investigate them thoroughly first. Ask them questions like:
- How and when is the system updated?
Making an update in the middle of a work day is a no-no. If your business depends on weekend traffic, updates “during the weekend” are a no-no. For most large scale systems, updates should be rolled out over-night after all tests have passed.
- Do they have a plan or just “roll code”?
All updates – regardless of severity – should be planned out well in advance. Most professional development shops have a “Staging Server” where the update is practiced to make sure everything goes smoothly. If your vendor does not practice their updates before hand then you can rest assured that you will have a problem at some point.
- Can they roll back if things don’t go as expected.
In the case above, I know the site has been down for over two hours. If your company can measure downtime in dollars per minute, this is unacceptable. Your vendor should have a “Plan B” so that if things don’t go smoothly, they immediately revert to the previous production system, figure out where they went wrong and schedule another maintenance window in the future.
- Do they run a suite of unit tests on their code before it goes into production?
“Unit Testing” is a software development terms that simply means that the developers write programs that test the system. When developers talk about “test coverage” it is the amount of the system that they have written tests for. Test coverage is expressed as a percentage. Most vendors won’t have 100% test coverage but it is reasonable to expect 65%-75% test coverage. Ask your vendor how much of the system is covered by unit tests. If the answer sounds suspicious, ask to see the report form the last run. When unit tests are run against a system, a report is generated telling the developers which tests passed and which failed. Even though you may not be able to read the report, not being able to produce it is a bad sign.
- Do they do “regression testing” on their code?
Good software development shops employ a technique called “regression testing”. Regression testing makes sure that before an update goes out, that the changes don’t break something else along the way. Regression testing is usually automated but it takes quite a bit of time to write and implement the scripts. Make sure your vendor tests updates before they go out to ensure the quality is what you expect form your web site.
Downtime is inevitable in any computer system. However, you, working with your vendor, can work to minimize it. Talk with your vendor about their upgrade and testing policies and procedures. Be prepared to be told that your bid did not include them because, like documentation, testing is the first thing to go when a client starts pushing back on the price of a system. Testing is not free and you should expect to pay for it to insure the quality of your system. On the other hand, if you pay for it, you should insist on seeing regular test reports. When it comes to major system upgrades – regardless of whether you are one system out of many or you are the only client – insist on discussing the roll-out plan before it is implemented. Ask if the rollout plan had been tested in staging. Ask to see a copy of the latest test results to make sure they have been done.
If your vendor can’t discuss these items with you, it may be time to look for a new vendor.
Until next time
I <3 |<