Barnet Libraries' Capita IT crash: the unanswered questions - and now a devastating audit report

Comms Team, North London Business Park

In the topsy-turvey, inside out world of Broken Barnet, there is a great deal of amusement to be found in the study of our Tory Council's management of 'communications'. 

The manufacture of corporate spin is undertaken by a team of obedient drones at North London Business Park, who for most of the time, it seems - when they are not reading Private Eye, or, with gritted teeth, making unguarded remarks about Mrs Angry, within earshot of passing spies - sit forlornly at their desks, obliged by their senior managers to portray a rosy, heart-warming picture of life as it is not now, has never been, and never will be, in this most rotten of Rotten Boroughs. 

One would imagine that even Winston Smith would be giving in his notice before lunchtime on his first day at work, should he be unfortunate enough to be sent to work in Broken Barnet, on secondment from the Ministry of Truth.

The issue of the iniquitous library proposals approved at last week's Full Council meeting presented a real challenge from the point of PR, of course. How to pretend this full frontal assault on a much loved service is anything but that? That it is indeed a wonderful outcome for the residents and taxpayers and library users of this borough?

Read on, but try not to be distracted by the fact that the press release had a date written in Russian:  "Published: 05 Апрель 2016" (the spoilsports have changed it now) ... Tory leader Richard Cornelius still lives in the Cold War era, of course, and is awfully concerned about the Red Peril. Proof, see, that he is right to be worried: and yes - we have our people everywhere, you know. 

But look: wonderful news:

Plans to maintain 14 library sites approved

Hoorah! Your Tory councillors have saved 14 libraries! 

Oh: well, no, not 14 libraries, exactly ... 14 library sites. 

Ah. Or, if you will allow Mrs Angry to translate:

Devastating changes to your local libraries have just been approved by your Tory councillors, which represent the complete destruction of your library service, and its replacement by DIY, self entry, self service book processing units, but: we don't have the courage to tell you the truth. 

It is not the idea of a library service that is of any worth, of course, in Broken Barnet, where everything must be measured by its material value. The sites have been maintained - for the time being, handed over to Crapita to manage, and screw revenue from, they boast, from the extra space accrued by shrinking the amount of space for the library part of a library building. 

And any sites that have been 'saved' are only theoretically safe: if for example, as seems highly likely, no voluntary bodies come forward capable of running the 'partnership libraries', you can bet your bottom dollar that the council will then turn around, shrug its corporate shoulders and blame residents for the fact they will have to shut that branch, and sell the property. Kerrching!

Apart from this piece of corporate bunkum, we have another example of creative writing on the council's website, in regard to their statement about the library system crash: 

Libraries

We continue to experience problems with some elements of our Library IT system, which means that some library services remain unavailable to residents. 

Problems with some elements of the IT system. 

A masterly piece of understatement.

As reported in the previous post, at last week's Full Council meeting, at which the library cuts were approved, there was a great deal of confusion, both in the public gallery, and amongst the Labour councillors about the present state of the library service. 

The Labour group had submitted an amendment that did not mention the continuing crisis, because, they claim, they had been informed it was over. 

Let us be very clear about this: it was not. Today it was announced that what had been only a 'glitch' had been 'fixed'. Hmm. Read on.

As a library user reported shortly afterwards in a comment on the previous post:

I agree that the IT problems are ongoing, as any regular visitor to the libraries would know. There is presumably no credible rescue plan in place, or staff are not briefed about it, as they have been saying the problems will be resolved in a "few" or "two" weeks for some time. This makes it impossible to renew books from home, impossible to check from home or in the library what you, or more importantly your children, have on loan, means staff have to ring round to get reservations and means no collection of overdue fines.

Of course now that the library proposals have been approved, the borough's children will have greatly diminished right of access to any libraries, or any books, pcs, or study space.

As things stand now, before the hatchet men move in to install the robot libraries, the human beings they are intended to replace are still struggling to explain to readers why they cannot access the normal facilities offered by their library. 

And the staff members who have just been told half of them are about to lose their jobs have been given instructions as to deal with the continuing failure, while attempts are meant to slowly test the replacement system, which of course is a separate issue to the serious problems caused by the crash, some of which are insoluble. 

Data relating to stock and membership appears to be irretrievable, and if so, this means the service - if and when a system is up and running again - has effectively lost details of many of the books it holds. Mrs Angry understands:

1. It has become necessary to 're-establish' a library catalogue by means of a physical stock check.

2. Staff are being told to make sure readers are informed that their personal data has been 'corrupted' rather than lost. 

Since drafting this post, we see this morning a claim by Barnet, reported in the local Times online, that all is well, once again, in Barnet Libraries. Well, goodness me:

"Barnet Conservatives claim its library service, Vubis, which crashed at the start of March is mainly back up and running".

Mainly? 

Any difference, as regards users, since last week? Ah, hang on: here is Cllr Thompstone, to update us:

"The central library IT system has now been restored and most services are available to residents.

Residents are able to borrow and return books, as well as renew them in person. Users can access e-books and audio-books and adults can access library computers and the free wifi.

Throughout the recovery phase all libraries have remained open and the extended opening hours at Edgware are again available. 

Some data does still need to be verified or manually re-entered. This will continue to affect users".

Ah. 

Most services are available. Most. Some data does still need to be verified or manually re-entered. 

That would be book stock, catalogue, and membership details, then, would it? 

Meh. Hardly central to the function of a library service, any of these things, of course, in Broken Barnet.

You may disagree, and support Mrs Angry's view that the system, as regards users, hardly reaches the definition of 'fixed'. In fact, it is hard to see much difference to the status quo last week. What do you think, readers?

This cheery news will come as something of a surprise to the staff, struggling to deal with the impact of the lost data which is not lost data, and only corrupted - and irretrievable, and the backlog of work that has ensued from this mess. 

(In fact - updated Thursday -  it would appear that although a version of Vubis is back up, after six long weeks, there is still no access to the catalogue, and the data that is available is limited, unpredictable, and fragmented. There is no two ways about it: things are NOT back to normal, and it will be a long time before these problems are fully resolved).

But will an admission of corrupted data, which was not lost, but has not been found, reassure anyone who might be worrying about a potential breach of the Data Protection Act, and a risk to the security of their personal information? If you are concerned, you may wish to make enquiries with the Information Commissioner. 

Mrs Angry has always found the ICO to be very helpful, dating back to the time when Barnet Council's illegally operating, unlicensed, jackbooted security firm 'MetPro' secretly filmed her, with hidden lapel cameras, at a council meeting and no one would hand over the footage (never did get hold of it); not to mention in regard to various problems over the years with FOI responses, previous data issues etc.

But we digress.

All library members will now have to be issued with new application forms, which will be uploaded by the soon to be booted out staff, along with the massive backlog of other work caused by the crash. 

In theory, the extra workload is meant to be done by agency staff paid for by Crapita, but it seems there is some confusion over this and regular staff are already undertaking such tasks, which if true, suggests careful audit should be made of exactly how much financial redress from the contractors is being made.

There is one happy outcome from all this disaster management activity: because all readers must re-register, Mrs Angry understands that staff are expected to ask them all if by chance they wish to take part in the wonderful new 'extended hours' scheme - ie the unstaffed, unattended, robot library system, that dare not speak its name. 

Mrs Angry is uncertain of the exact wording suggested to staff: presumably something on the lines of: 

I've just been sacked, and won't be working here much longer, and if you want any help in the library, there won't be any, and your children won't be allowed in on their own either, but while I am still around, would you like to sign up for the new shrunken, hollowed out, unstaffed DIY libraries instead?

At last week's Full Council meeting, where the awful new proposals were approved, by a bunch of Tory councillors squirming in their seats, and keeping their heads down, calculating the impact on their electoral chances next time round, Labour members attempted, unsuccessfully, to plead for a delay in the vote, in the light of the news - and apparently it was news to them - that the IT system was still not restored, and problems continuing.

Some library services remain unavailable to residents, says Barnet Council

That the Tories were determined to force through the plans in these circumstances is predictable, but they must know the huge risk such an action bears. 

And if they are unaware of the risk, that is partly because the full impact of the crash are not known, or at least not known to members - how can they be, when the crisis continues, and the system is not yet operational, and available to the public?

Mrs Angry has been calling all along, in vain, of course, for an independent investigation into the library system crash, because it is clear there are so many unanswered questions surrounding this catastrophic failure, and with massive implications for the wider contractual responsibilities that rest with Capita.

The only written explanation for the cause of the library IT failure was contained in the delayed publication of 'Appendix L', as seen here.

Mrs Angry has had several people contacting her in regard to the statements made in this Appendix, and commenting on some of the issues raised. 

The latest contribution, from an IT specialist we shall call ... 'Mr B', raises some very, very interesting points, as you can read in the following extracts - Appendix statements in blue:

2. The incident occurred due to a combination of server and system errors. On 2nd March, Infor (the third party support provider for the Vubis application), reported to LBB Libraries that the library system was running out of space on the server. Customer Support Group (CSG) (Capita) responded to provide additional physical storage. At this time, it was unknown that back-ups for the system had been failing since the end of December 2015 (unrelated to the storage issue). The automated messages from Vubis alerting a nominated user of back-up failures were not being received. Investigations to understand why these were not received are hampered due to the IT crash.

Mr B points out that it appears the alerts were sent to a nominated user who had left the council when Capita took over - this he says suggests there was not a proper handover of the IT systems. 

As to the 'hampered' investigations into why the alerts were not received: it would seem Capita knows this is because they were sent to the wrong person.

Crapita's nominated receiver of alerts for back up failures

Mr B finds this rather troubling: 

This is pretty worrying, since it may indicate problems waiting to pop up elsewhere.

He notes that it would also seem that during the two years and more of Capita running things, there was no proper overview of the system, no on checking it for at least four months, if not longer. This would be strange, he suggests, as Capita has a very large IT section, with plenty of experienced engineers. Was Capita perhaps intending to install its own system? Mrs Angry asked a question on those lines at a recent meeting, and was told quite categorically by deputy CEO John Hooton - No.

Consequently, the back up failed again, causing the system to crash, and corrupt.

Mr B: Why failed again? Was there another instance of failure? Why did a back up failure cause the system to crash: it does not make sense ...

3. When the server was rebooted, it began to corrupt the data on the system. Whilst local backup processes were put in place these were backups to the local machine which also corrupted. The root cause analysis (RCA) has been concluded to be as follows:

Mr B: It would appear there is a confusion here between 'backup' and 'mirroring'. In short, backup is an image of the system taken at a point of time, and removed from the system. Mirroring is an availability solution, where every transaction is written simultaneously to separate storage devices, so if one fails, the system can continue running uninterrupted with the other mirror image. What we learn from this confused statement is the mirrored system disks (where the operating system is installed), had one or more of its disks fail. We don't know, and neither does Capita, when it happened, since the activity logs were stored on those disks.

4. A number of disk drives on the server displayed hardware failures. These

were replaced and the system was left overnight to rebuild. This is a standard system administrative function to resolve a failed disk. 

Mr B disagrees:

The correct procedure before doing such modification is to first:

a. check there is a valid backup

b. to check that the operating system logs are clear of errors

c. to check that the remaining disks are healthy

Apparently none of this was done.

4. A number of disk drives on the server displayed hardware failures. These

were replaced and the system was left overnight to rebuild.

This is a standard system administrative function to resolve a failed disk.

Subsequently the server crashed around 03.54 on 3 March and it is believed that the database files on Vubis became corrupted as a result of, or during, the subsequent required reboots.

Mr B: 

This is indeed strange. The Vubis database resides on a different set of disks, so the system disks corruption should not have had an effect on that separated data. What I suspect happened was that when the additional disks were added to the Vubis area, the recreation of the extended storage was allowed to run alongside the recreation of the system's mirroring disks (that is to say the mirror of a faulty disk into a healthy one) When the system crashed, the process of recreation of the Vubis storage was left in an unstable condition, which worsened during the 'subsequent required reboots' and led to complete corruption of both the system and data disks.

Hmm. And if so, in Mr B's view,  this would have been 'an amazingly clumsy piece of work'.

7. A non-corrupted tape back-up from March 2014 is available – this is the last date a tape back-up was carried out as the server was changed to digital backups only following this date.

Mr B:

What this tells us is that 'digital backup' was corrupted as well. If the last backup was taken on December 2015, the latest available backup should have been restored from this point - but apparently it was not. I'm at a loss to know what type of backup Capita did carry out. Was the backup done on the same disks that the data resided on, which would be (censored by Mrs Angry) or did the digital backup never work?

One last comment, which is deeply worrying, if true.

10. ... Wifi in libraries has been restored.

Mr B:

Wifi restored, but since there is no customer data (pin/login credentials), the network is open for anyone to connect to, which means that malicious users can snoop on other's activities. So do not connect to your bank or email from the library wifi.

If this allegation regarding wifi is true, at the very least, surely, all library visitors and users must be made aware of the risks to their own data security: the consequences are serious, and this matter should be addressed urgently, with confirmation as to whether or not proper advice is being given to potential users.

Since the IT crash was reported, Mrs Angry has asked, loudly and continually, to little avail, what implications the failure had for the wider context of all the other former council services run by Capita, in the course of its two massive and highly lucrative contracts. 

There has been a deathly silence, of course, as Tory councillors do not dare to think such dangerous thoughts, and some of the Labour group have also been slow to grasp the scale of failure regarding the library system, rather too easily persuaded by officers - or misunderstanding what they were told - that it really wasn't as bad as all that, and even, it would seem, as we saw, last week, that all problems had been resolved.

But now look: last night it was discovered a report * (this link has stopped working for some reason, try Item 7 here ...) has emerged from Barnet's Internal Audit team on the timely, and very interesting subject of - 

Information Technology Disaster Recovery, (ITDR).

This report is explosive, in terms of significance for the future of the Capita contracts. Or should be, if our Tory councillors had any intention of fully undertaking the role of scrutiny, which is doubtful.

The report, written apparently just days before the library IT crash, explains the purpose of this audit review:

Background & Context

An ITDR programme is the IT component of the wider Business Continuity Management (BCM) programme, which fulfills part of the Council’s obligations to the public and Civil Contingencies Act in the event of a major incident. The purpose of the programme is to recover IT services that underpin Council activities, within an agreed time and to a point in time prior to the outage, to prevent an unacceptable business impact. 

Hmm. If, like Mrs Angry, your mind is already shutting down and slipping into (non corruptible) backup mode, just try and focus on the phrases 'major incident', 'outage', and 'unacceptable business impact'.  

Now then, reader, can you think of any recent 'outage' that was a 'major incident' and has had an 'unacceptable business impact'? No? Then you are an elected member of the London Borough of Broken Barnet, and this is perfectly normal. Or you wrote this report. (No mention of the word, sshh ... 'library' ... )Go back to sleep. The rest of you, read on:

At Barnet, the technical component of the ITDR programme has been outsourced

to Capita as part of the Customer Support Group (CSG) contract.

Now then, do you remember the CELS meeting, where the new Barnet/Capita IT person told us all about our upgraded, Platinum card membership of the Crapita International Rescue scheme, for Catastrophic Events, that have Nothing to Do With Us? 

Platinum level response, from Crapita. Note the smooth running operation of the futuristic 'technology enabled library', top left.  And stop looking at top right.

Hmm. Well, now look at this:

The Council’s ITDR recovery requirements are described in the contract with Capita. It

was noted that the requirements detailed in the contract are not those that are being delivered by the ITDR project. In particular, the Council applications are rated as platinum, gold, silver or bronze based on an assessment of the business impact. Applications rated as Silver and Bronze, are supposed to be recovered within 48 hours with a maximum of an hour of data loss. The current project is not delivering ITDR for Bronze applications and the current provision is to restore Silver rated applications within 96 hours with up to a day’s worth of data loss. There are similar inconsistencies at Platinum and Gold level.

'Inconsistencies'. That's it. Translated: run for the lifeboats. We're all doomed. 

There's loads more like this, however, such as on the subject of 'Interim ITDR Capability':

Prior to the new ITDR capability being implemented at the secondary data centre, we confirmed that an Interim ITDR capability was in place. This was initially a ship to site “data-centre” that contained infrastructure for the Council’s legacy systems. These services were procured from an external supplier by Capita but the contract for these services lapsed in early 2015 and was not renewed. Capita are currently replicating data to the secondary site and taking backups in preparation for the new full ITDR capability, now due in Q1 2016. However, these back- ups cannot be used to restore capability as they have not been tested and there are no documented ITDR plans in place. It was noted that there is currently no alternative interim capability. 

Well, then. A ship to site 'data-centre' sounds awfully good, doesn't it? Except ... oh, hang on, it's looming into view now, that ship ... those lifeboats, where are they? Oh dear: listen, in the distance, just faintly ... Nearer, my God, to Thee ...

The report ends with an acknowledgement:

We would like to thank the CSG IT team and the Information Management team for their time and co-operation during the course of the internal audit.

Aww. 

Uh oh: followed immediately by: 

However, we would also like to note that during the course of the audit we encountered significant delays in receiving relevant information and being able to speak to the appropriate staff within CSG IT. The initial discussions around the audit were held in April 2015. The terms of reference for the review was not, however, agreed with CSG IT until September 2015. Delays were then suffered around gaining access to the right people and receiving the requested information. Ultimately a large amount of information was provided in late December 2015, hence this report has been issued in Quarter 4 as opposed to Quarter 1 as planned.

Well, well, well.

Mrs Angry cannot recall ever seeing a remark like that in any report submitted to a council meeting, let alone from Internal Audit.

It is absolutely clear now that there are fundamental questions to ask about the true standard of performance and value for money delivered by the Capita contracts. 

Aside from the core contract, and the minimal 'savings' seemingly achieved from that, wrapped around this is the enormous extra profit extracted from us in the form of extra charges, apparently every month. And at the same time, the level of standard of services provided is, in the case of IT, in what is supposed to be Capita's area of expertise, an issue of huge concern, in the light of the library system failure.

Right from the earliest days of this blog, and the uncovering of a culture of failure in procurement and contractual processes as a result of the MetPro security scandal, it is often in Broken Barnet, the smallest thread that once pulled, unravels a greater hole in the carapace of this Tory administration. The library system crash was and is a very serious failure, in itself and in terms of the terrible library plans just approved: but its wider significance is in the ominous implications for the future of the monolithic Capita contracts, and the lack of transparency, accountability and 'ownership' of the failure of the library system does not bode well for a realistic appraisal of the real risk to services that any similar weaknesses might pose.

The hole is too big, now, to cover up anymore: and now it is the responsibility of councillors of all parties to take action - and prevent further betrayal of the best interests of the residents and taxpayers of this borough. Will the Tories who nominally at least are in charge of this authority find the courage to act, and for once put what is right before political considerations? I doubt it. I hope I am wrong.