Working under stress

A couple of years ago, I received a panicky 'phone call from a friend. The company he worked for had been mentioned, along with their website address, on the front page of The Telegraph. The site had started to load more and more slowly, and now it wouldn't load at all. I asked him who'd built the site and where it was hosted. It transpired the site had been built by a man who was away and, consequently, couldn't be contacted. After a little investigation we located the hosting on one of those £50 a year servers. Of course, there was nothing we could do to help and the exposure in The Telegraph was largely wasted.

I was reminded of this by three occurrences in the last month, concerning Twitter, the BBC and Premier Inn. Twitter users will be well accustomed to the application's occasionally flaky service. We don't pay for it, we use it frequently, and we get all sarky when it can't take the very high strain.

On March 29th the BBC website and related services (such as iPlayer) were unavailable. The final explanation given was that there had been a major network problem.

Finally, Premier Inns recently broadcast an email campaign offering their popular rooms for £19 offer. (Popular but elusive; I've never been able to find one.) This was followed days later by another email apologising that the website hadn't been able to cope with all the resulting traffic.

Superficially, these all look like the same issue but, in fact, there are a couple of factors to consider. Firstly, there is normal load. Whether you are talking about a website or an application or any other aspect of your online infrastructure, you need to think about how many users you will have, how often they will use your service and what resources they will use. For example, if you have a popular website that is largely text and pictures, then you simply need a server that is good at spitting out web pages. However, if you are Premier Inn, where your users are accessing a database and using up processing power, then you have more factors to take into consideration.

Secondly, there is the question of 'spikes', i.e. sudden load on your server and infrastructure. These spikes can be quite dramatic compared with normal usage. You might be an online clothes retailer, advertising that your 30% off sale starts at 9am on Monday. Your set up is going to have to cope with something quite different to the normal steady usage of people browsing and purchasing.

Of course, you shouldn't wait until your site is live to consider these issues and you certainly don't want to find out about them on the day of the sale you've spent so much time and effort promoting. But you'd be surprised by how many people don't consider them. I can count on the fingers of one hand the clients who have raised this as a concern with me before I have had a chance to discuss it with them. And that's fair enough. Clients assume that their suppliers are thinking about these things on their behalf.

However, as with so many other topics - like security, DDA, cross-browser testing - the IT industry repeatedly lets its clients down. There are few professional qualifications and anyone with a PC can set themselves up as a web designer or developer. Consequently, it does fall to the client to ask the questions, not to assume that, having paid for the development of their site or application, it will run on machines that are adequate to support its usage.

Lush and PCI Compliancy

"I can't believe that a company of this size can be so naive about website security".

The above quote (from a post by 'symball') is taken from an article in today's Guardian about the hacking of the website belonging to Lush Cosmetics. The company have known since at least Christmas Day that they were being hacked and it has now admitted that the hacking dates back to October last year. Customers are reporting that their cards have been used fraudulently.

The sad truth, though, is that that online security is poorly understood and badly enforced. Whilst any company trading online should be PCI compliant the truth is that many online traders are simply unaware of this requirement and many web development companies, particularly those with a strong design or marketing bias, don't have the technical skills to set up a site that is compliant. Certainly, the company working for Lush should have known better to hold onto card details.

However, the problem here is not just about PCI Compliancy. We have had numerous hacking attempts on our webservers over the years and we have a full time systems administrator who keeps our boxes up to date with the latest security patches precisely to keep hackers out. All too often, though, less technical web development companies rely on their hosting company for their security and this simply isn't good enough.

We have 'inherited' websites in the past and had the difficult job of explaining to the client just how much work needs to be done to their site before we can put it onto one of our live boxes. Similarly, we have in the past, (reluctantly) lost clients who were not interested in the ongoing costs of maintaining the security, both through the necessary hosting charges (to constantly review and maintain server security) and the cost of keeping up with the constant changes to PCI Compliance.

If your business sells online, you need to check with your web developers about your PCI compliance and your server security. If you are unsure, then contact a company such as Security Metrics who can do both PCI checks and 'penetration testing'.

And if you are storing your customers' credit card numbers, I would start worrying about this RIGHT NOW.

Why I walked out.

It's been a busy couple of months. A new project with a client that I can't yet name, which is the hugely satisfying culmination of six years' hard work, not to mention new lessons learned about the need to project manage clients during large scale developments with staggered code deployments. I know I've neglected the blog - just the sort of scenario I warn blogging clients about - but I've had two or three topics I want to blog about knocking around my head and I've been looking forward to having the time to put them into writing.

However, this evening I want to use the blog for a different reason, which is partly by way of explanation of my sudden exit from an event I attended, this evening. The event was called 'Beyond Websites - Using Web Technology Creatively'. I must admit I was slightly dubious about the title, which seemed to offer two different topics but, with the advent of HTML 5, I had little doubt this would be an interesting couple of hours, with a presentation from Keith Mitchell (@specialized), a Research Fellow at Lancaster University, and then a panel discussion.

The presentation was entertaining but a little disappointing, mostly concerned with watching television over the web. There was some talk of community clouds - effectively caching programmes - which I believe will be rapidly superseded by the rollout of more powerful comms and - as the lady from Business Link pointed out - better compression algorithms, anyway, and there was also a demo of some software that would enable the user to watch television whilst viewing a clickable television guide and relevant feeds from Twitter and Facebook, which struck me as reactive rather than innovative development. Certainly I wouldn't have described it as a creative use of web technology.

The panel discussion was a further let down. There was some talk about existing websites, particularly Vimeo, some paranoia about Google claiming copyright to any documents placed on Google Docs (take a *closer* look at the T&Cs) and then some discussion about targeting content, with specific mention of 'Googlezon' from "Epic 2015" (2014, in fact).

So, why the hissy fit, albeit in the relatively middle-class form of walking out?

Well, I won't pretend it was because the session didn't live up to its name (although more on this in a bit). I'm as happy as the next (nerdy) guy to spend an evening talking about the web/Internet and guessing at where it's heading. I would have had no problem with that. There are two overlapping elements to evenings such as this one that really get my goat. The first is a rather smug assumption that we're at the forefront of a cultural movement, i.e. that what we're doing today is what everyone else will be experiencing tomorrow. The second is the weak recycling of common 'wisdom' regarding the web, what's cool and where it's "definitely" going to go wrong.

So, just because we use Twitter and blog, doesn't mean everyone is going to. Indeed, I'd argue that the very fact that we are entrepreneurs working in design, marketing and new media means we are exactly the kind of self-aggrandising/outgoing folk who will engage in these activities. Does it mean other people won't? Of course not. Be just because we *all* do, doesn't mean *everyone* else will.

Similarly, it wasn't true to say that everyone spends more time online than they do watching television. That may be true of teenagers but when I was a teenager I spent more time in my room reading and listening to music than watching TV. Let's not confuse human behaviour with cultural trends. And as for the ridiculous anecdote about the literature professor who can't read War and Peace since he started using the web... That man is in the minority.

I'd recommend these people read Tim Berners-Lee, Clay Shirky and maybe Brian Eno's insightful 1995 essay on targeted marketing.

So, yes, I got impatient and I have a low boredom threshold and I walked out. But what would have made me stay? Two things, I think.

Firstly, we could have enjoyed some talk about how the way in which we use the Internet has changed. The Internet is a load of computers/servers connected together. Over that, the web was laid, pages of content that joined together and, initially, that was how we used the Internet (and for email and IRC, of course). Now we use the Internet to deliver content to the apps on our smartphones or to 'narrowcast' films to our TVs, and the web part of the general connectivity has become a little less important (although it won't go anywhere in the short-term). We could have usefully talked about how users are accessing data and how we adapt what we do to satisfy their demands. And how we keep that fresh, interesting and, yes, creative.

The second thing that could and should have been different this evening was that we should have talked about the Internet as a huge, interconnected resource, to which people contribute. Web 2.0 as it is now called (and was so referenced as at the start of the evening). Instead we talked about the web - but, really, the Internet - as a mechanism that delivered content to us. We talked about how the content presented to us could be better refined, more targeted. But that isn't the point. The original Web 2.0 - as defined by Tim Berners-Lee - sometimes called the semantic web, was about joining up data, in all its forms (including video). One of the panellists did make a good point this evening (and I must apologise that I can't remember who it was), which was that often we start out looking at one thing on the web and, by following links (and, I'd add, our interests), we find our way to all sorts of different sites, pages and information. This is what excites me about the web; the interconnectivity, the wealth of data, the user content, the real-time commentary on life.

The web, the Internet, whatever you want to call it, is - at the risk of sounding clichéd - a fresh, innovative, exciting space. What excites me is not how I might watch TV over the web or how content might be targeted for my consumption. No. I love being in the middle of a strange town with my smartphone telling me where I am and what's around me. I love being able to map the runs I do and share them with my long-suffering friends. I love the fact I can IM my daughter in Berlin *right now* and chat to her. I love the fact that via apps and pages and any and every other means, we can all access and share films and music and data and opinions and all the other things that constitute the culture that we enjoy. That, I think, is worth talking about and, indeed, celebrating.

What yesterday's problems tell us about Twitter's testing

Those of you who keep an eye on the news - and who don't switch off at the mere mention of Twitter - will probably be aware that Twitter had problems yesterday as users (with development skills) were able to include code in their posts leading to the problems described in The Guardian (here) and The Telegraph (here).

Unsurprisingly, the language used to describe the incident is full of the kind of words that IT people use to keep everyone else at a distance and to spray a little nerd glamour on themselves. But for all the talk of malicious code, worms, onmouseover, hacks and loopholes, the truth of the matter is remarkably straightforward.

So, first, a quick description of how browsers work. When you load a web page, your browser requests HTML (the language in which web pages are written) from the web server and as it receives the code, it uses it to build the page, top down. It's a fundamentally simple process and the browser simply processes each line in turn. So, when a browser displays a page of 'Tweets', the messages are simply part of the HTML. If some other code is included in the HTML for that message, then the browser simply interprets it.

I first came across this as an issue ten years ago when I was leading the testing for RBS's Digital Banking software, their first purely web-based Internet Banking service. I discovered during our early testing that on the page where a user would be able to name their accounts, I could enter basic HTML, which would then affect the way the page was subsequently displayed, once it had been saved.

From then on, every entry field in the website had to be coded in such a way that any characters that might be used to insert code were not permitted. And every test script included tests to ensure that if those characters were used, the data would not be saved to the database. We had a few data input boxes and the testing was time consuming but, of course, it was vitally important that no one could introduce code and make the site work in a way that wasn't intended. These scripts were used to test every release of Digital Banking, even if the changes were in a different part of the system from the data entry.

Twitter has one input box. That's it: one. It might be deployed on one or two different pages but it is the same code, the same function.

So, what does this tell us about Twitter's testing. If it tells us one thing, it tells us it isn't as robust as it should be. It doesn't really matter whether the issue is down to a tester who ticked a box without actually doing a test, an automated script that wasn't run, poorly documented test scripts or a missing process that should confirm that all scripts are complete. This was a bad drop by an organisation that has tens of millions of users and a burgeoning usage by business.

For as long as I have been in IT, testing has been the poor cousin to development, and regarded as an unnecessary headache by developers. IT and project managers must never lose sight of the importance of this stage of IT development: customer and client confidence is easily lost and difficult to regain.

Keeping content accessible and other things *you* should do for your site

There's a lot of things you might ask or even demand from the company that builds your web site. Certainly one of those things is that it should comply with the Disability Discrimination Act but you might also ask for, say a news page and a calendar of events.

But whilst it's very easy to have ideas at that point in proceedings, you should think about your ongoing commitment. There's nothing that makes a site look sadder and neglected than a news box on the front page that hasn't been updated for months or a calendar with nothing on it. It's important to ensure that somebody in your company - or perhaps someone outside it: your copywriter or marketing people - takes responsibility for that content and ensures it is updated regularly and with some care.

However, the main reason for this quick blog is to do with your responsibility for accessibility. Today we are sending out our latest newsletter, which is about accessibility and the DDA. In the newsletter we provide a link to a tool for checking accessibility and, of course, it occurred to us that a sharp client or two might use it to check our sites: our own and the ones we've built.

We were more than a little surprised at first to find some of them failing because we always check our sites for both W3C and DDA compliancy once they're finished. However, on closer inspection, we found that it was the user generated content that was causing the problem and not the code we'd written. That was, of course, a relief but then Louise asked whether we had even spoken to our clients about how to keep their content accessible. Well, that did take the smiles off our faces.

So, from next week, we will be briefing our existing clients on how to make sure that the content they put up is accessible and making sure that it's part of the training for our new ones.

Why the government needn't stick with IE6

One of the features of the web that I initially found exciting was the concept of platform independence. One could build a page in HTML and anyone using any browser on any machine running any operating system would be able to see that page and use it just the way its author intended.

I will pause here to allow the hollow laughter of web developers everywhere to fade to an echo and so not interrupt my thoughts.

The truth of the matter is that over the last fifteen or so years we have had a multitude of browsers that work differently on various platforms and that has made both the development and testing of web pages and applications far more time consuming and, therefore, expensive than necessary. At Meantime we develop in whatever we consider to be the most standards compliant browser at the time and test primarily in whatever is the most popular, and both of these are moving targets.

As you would expect, Microsoft have a major role in the history of browsers but it is a surprisingly chequered history. Bill Gates was famously dismissive of the Internet at the outset and, if memory serves me correctly, it was not until IE4 was released that Microsoft really found their feet in the marketplace. Their success lasted 18 months until early 1999 when IE5 was released.

The extent of IE5's problems can be inferred from the fact that, on this one occasion, Microsoft released an interim version of the browser, IE5.5. And when IE6 came out, it was widely agreed that unhappy IE5 users would have been better waiting for that than 'upgrading' to IE5.5. (I remember all this vividly; I was responsible for the team testing the Royal Bank of Scotland's Internet banking software at the time.)

Of course, all of this made life incredibly torrid for those people in organisations who were responsible for the software and applications architecture. It was a period beset with problems and costs, and this was during the period immediately following the nasty surprise cost of the Millennium Bug. But IE6 was stable and remained Microsoft's browser offering for five years.

Despite that long period of stability - or, arguably, because of it - there did not appear to be a strong appetite for change when IE7 came along or, a couple of years further down the line, IE8. And so we find ourselves in 2010 with many large organisations and particularly government departments still using IE6, which is now nine years old.

The problem is that an awful lot has happened in those nine years and right now is a particularly exciting time with the new HTML5 and CSS3 support in the latest browsers: Chrome already has it, as does Firefox 4, which is out in a mature beta, and so will IE9's public beta, which is released in just a couple of weeks' time.

And in the middle of all this excitement, the government has announced that they won't be upgrading from IE6. Such a move, they say, would be "a very large operation" potentially at "significant potential cost to the taxpayer". They say that it will be "more cost-effective in many cases to continue to use IE6 and rely on other measures, such as firewalls and malware-scanning software, to further protect public sector internet users."

This is such a short-sighted option that I don't feel I need spell out the various reasons why it is a bad idea. Indeed, the quote above makes it clear there is an attendant security risk which is a strong argument in itself.

What is perhaps less obvious is that as software is increasingly web-based, software that is used by local authorities, it is not a straightforward transition to move between browsers: there is a lot of regression testing and consequent redevelopment to be done.

But it's not satisfactory for government IT strategists to simply throw their hands up and say it's too difficult or too expensive to change. Yes, I understand this is supposed to be a time of austerity, but that doesn't mean that an economical solution can't be found. In six months' time we will have a clear view of which of the current crop of browsers is the best and, as a first step, this can be deployed across government departments. From there, a two year plan to migrate applications from IE6 to the new browser will get departments to the point where they can look at their next upgrade. Because this is a rolling process and not a one off. Government needs to recognise that fact and start to make the cultural change that will enable them to leverage the benefits of what is still a fast moving (and exciting) technology.

Bespoke software is alive and well.

In a recently posted article on the Business Computing World website, Haseet Sanghrajka of ST Consulting writes under the byline "How Application Platforms Are Killing Bespoke Software'.

Now, I can only assume that the title comes from a bit of over-enthusiasm for his offering because this is certainly not what Mr Sanghrajka succeeds in demonstrating. What he does argue is that application platforms are the best way forward for companies needing software solutions, as opposed to either package or bespoke solutions.

Let's tackle the package argument first, as it is the easiest. I would certainly agree that more complex business software, when taken in package form, is limiting for any business. However, there should be allowance made for packages such as QuickBooks or MS Office: there are times when a package solution is entirely appropriate.

His argument against bespoke software seems to be related to development time, cost and maintenance. So, let's return to those arguments after we've had a look at what Mr Sanghrajka is proposing, which is an 'application platform'. Although this is not particularly well explained in the article, in his terms it consists of some packages - that is, Microsoft Dynamics, Office and Outlook - plus a bespoke development language, .NET. (Packages and bespoke development are, of course, precisely what he is campaigning against.)

Those of you who have any experience of Dynamics will be aware of its eye-watering price tag (although MS have now introduced a cheaper version to try and encourage take up in smaller businesses). Microsoft themselves are pleased to tell their resellers that they can anticipate many times the cost of the package in terms of consultancy fees. Readers of Mr Sanghrajka's article won't be surprised when they reach the end to see that his company sells precisely this type of consultancy.

But back to our argument. Dynamics is more commonly known as Dynamics CRM and it is for this purpose - customer relationship management - that the product is most commonly sold. Extending the product out to cover other functions takes us into the territory of square pegs and round holes, and it is for this reason alone that I struggle to see how this particular application platform can compete with bespoke software.

So, to pursue the argument, perhaps the article is only really supposed to be about CRM. One of the touted benefits of Dynamics is just how powerful, configurable and flexible it is. I don't disagree with that. But you can deduce from those features that it is a complex piece of software, which is why companies such as ST Consulting can make a business out of configuring it.

It is this complexity and the cost of the consultants needed to maintain it that is seeing organisations move away from Dynamics (and SAP, as well) to simpler, bespoke solutions that do exactly what their companies need and nothing more. Certainly any bespoke provider worth their salt can provide a CRM for less than the cost of a configured Dynamics solution.

So, the argument was that bespoke software is too expensive, takes too long to develop and needs maintenance. But Dynamics will frequently be more expensive as a whole, takes time to configure and is too complex for organisations to manage themselves, resulting in ongoing consultancy costs. What's more, working with a good software house - with business and systems analysts, experienced developers, and a strong testing and change management process - will lead to cost-effective software being delivered to spec, on time and on budget.

I'd like you to indulge me in a couple of very specific ripostes, too. Quite apart from the fact that the argument has not been raging for decades (two perhaps, which is technically plural, I suppose), Mr Sanghrajka has it plain wrong when he says "Unlike traditional bespoke development that relies on software coders, application platform technology allows organisations to build on the underlying relational database using point and click development tools. Key concepts such as database fields, data relationships and workflow are automatically handled by the underlying application platform layer." This is disingenuous, at best. Building and maintaining relational databases requires good understanding and experience. Whilst a casual user may be able to add an attribute here and there, by the time they are creating entities and connecting them, they are only a short step from having to call those Dynamics consultants back in to sort out the ensuing problems with performance, reporting and data integrity.

Secondly, Mr Sanghrajka states that "it is estimated that an organisation can create a new solution from scratch in the same time frame it takes to deploy a traditional packaged application." Does this need much more than common sense to disprove it? Unless the new solution is, perhaps, a VAT calculation and the package is, say, Microsoft Dynamics.

My argument here is not with Mr Sanghrajka, who has a business to run and who no doubt has a lot of belief in Dynamics. What I dislike is the false argument. By Mr Sanghrajka's own admission, bespoke is the best solution. However, his arguments against it are flawed and actually apply more acutely to Microsoft Dynamics. Or the 'application platform' if you prefer.