I was at the #PostOfficeInquiry in London to watch Fujitsu whistleblower Richard Roll give evidence. I first met Richard Roll in the Leathern Bottle pub just outside Wokingham in April 2015. What he told me was something the Post Office were explicitly denying - that Horizon errors could cause holes in Postmaster accounts, and that Fujitsu staff could go into those accounts, and change them at will.
By that stage the BBC had commissioned its first Panorama into the Post Office scandal. Mr Roll became a significant contributor to our programme.
His appearance and what he said in public, on the record, became a significant factor in bringing the funders and legal teams together (added of course to the testimony and evidence of hundreds of Subpostmasters) to take on the Post Office in what became Bates v Post Office. Mr Roll was a witness for the Postmasters in that High Court litigation, and the judge (in finding for the Postmasters) called Mr Roll's evidence "very important".You can read more about Mr Roll in my book The Great Post Office Scandal. He has been sworn in to the Inquiry, which means his witness statement is here.
From Roll's witness statement re the Post Office Horizon system:
"My role in Fujitsu SSC was to provide third line support on the Horizon system. The system was hugely complex and included (amongst other things) Windows NT systems and Unix servers; asymmetric digital subscriber line (ADSL), microwave and satellite communications systems; and software written in a variety of languages. Day-to-day tasks were many and varied - for example, we may have been asked to examine database transactions in order to identify errors in the accounting system; examine computer programmes to try to identify bugs; investigate communications failures in the system; rectify failures in the overnight processing of the previous days transactions; or liaise with engineers during the installation of new Post Office counters. We also generated reports for senior management and the Post Office, tested equipment, evaluated new hardware and software etc."
He continued:
"When I joined Fujitsu, I received the same basic training on the Horizon system as the SPM's [Subpostmasters], although as I didn't work in a Post Office I never became truly familiar with the system operation from a SPM's perspective. I don't recall there being any technical training as such —the system was so complex and the role so demanding that you had to be an expert in one or more fields when you started, you then picked it up as you went along."
Roll was at Fujitsu working on Horizon between 2001 and 2004:
"General security protocols were in place at the PO's. Secure passwords were required, which were not supposed to be shared but often were. There was a secure link from the PO counter to Fujitsu's servers - the counters would only espond to requests originating from specific telephone numbers —and all data was encrypted. Additionally, the network was completely (logically) isolated from the internet, it had its own dedicated lines and for resilience these were duplicated, with one to the east of the country and one to the west, I think (physically, the lines were probably shared with other BT customers). However, due to geographical and financial constraints, these lines merged in London and both were routed along one of the Tube lines; I believe there was a minor train accident one day which unfortunately severed both cables and left the north of the country isolated from the south."
"We all had at least two PC's at Bracknell; one 'open', which we used for emails, researching the internet etc, and one 'secure' (completely isolated from the `open' system) for working on the Horizon system."
Mr Roll being asked about coming forward to raise the alarm. He told me several years ago it was the Inside Out broadcast in 2011, but he now can't remember how he became aware of it, or who he contacted to offer his assistance (it was Alan Bates).
Roll does remember being on Panorama programme. He says he never thought of himself as a whistleblower until it was mentioned to him.
We are being taken to his witness statement to the High Court and his evidence about trying to find the source of discrepancies in Horizon code by downloading branch data. Sometimes scrolling through lines of code, sometimes printing the code off and using their own filtering programmes they had written to strip out irrelevant data lines. Having identified a problem, they would then look at the source code to see what might be causing the problem. If it was the source code, they would refer it to the developers and say "here's the problem, this is the source code. This is the source line. It's wrong. It should say minus this value when it says plus this value."
Single errors were easy to identify, but multiple errors were really hard to spot and they would cause problems to "snowball". Jason Beer KC is doing the questioning of Roll for the Inquiry. This is very different from the experience of watching Roll being cross-examined at the High Court. It is in the spirit of inquiry, rather than an attempt to belittle or demolish his recollection and expertise, and is quite measured in its pace (given the High Court trials lasted five or six weeks and this Inquiry has as much time as it needs, that is perhaps not surprising). JB asks if he had to identify AND fix code problems. RR says their "primary aim was to keep the system up and running so it worked, and so that Fujitsu didn't suffer any penalties… if we could identify problems in the code as we went along, that was a bonus."
JB asking about penalties for Fujitsu.
RR is hazy, but if a bank transfer didn't go through in 3 days there would be a small penalty, but can't remember if it was 10p or £10. He says the problem is when you've got 20-40,000 counters and a bug stops all transactions of a specific nature happening it magnifies the small fine into a massive one.
JB So you patched things up as you went along?
RR It was widely accepted that the system was "crap" and needed a re-write but that was never going to happen because of the money involved.
RR says that when an identification of an error was made it was passed to the developers, but he can't remember the names of any developers.
RR says Subpostmasters might be told there was a fix going into the system to sort the problem affecting their branch, but they were never specifically told it was a bug within Horizon. JB pushes him on this:
RR If we were talking to an SPM we'd say there was a problem with their counter "data corruption" or something like that.
JB So it wasn't habitually fed back to them that there were coding errrors?
RR Correct.
JB Was there an official line on this?
RR Not sure.
JB But the practice was not to tell them?
RR Correct.
We go to RR's witness statement when he said: "In my opinion the coding and development of the system did not meet my expectations of quality for a major software project; I considered it to be a very poor system that should never have been deployed but I cannot be more specific than this."JB asked if that related to his earlier "pithy" epithet he gave the Inquiry earlier (ie that it was "crap").
RR Yes
JB takes RR to a section of his High Court Witness Statement "My recollection is that the software issues we were routinely encountering could, and did, cause financial discrepancies at branch level, including “shortfalls” being incorrectly shown on the Horizon system… If we were unable to find the cause of the discrepancy then this was reported up the chain and it was assumed that the postmaster was to blame."
RR struggles to remember examples.
Being asked about how problems came up, he said they could be raised by SPMs or they could find them themselves in SSC.
RR says he "thinks" Horizon improved whilst he was there in terms of coding standards and the documentation "but that's a distant memory".
JB asks why he has that recollection.
RR thinks people just became more professional.
JB How reliable were the Horizon cash accounts?
RR Pretty ropey. I said ‘surely this should be rewritten’ to my manager Mik Peach and he said ‘it's never going to happen’.
JB And was that down to money as you said before?
RR Yes and we didn't have the staff.
JB You say if you could not find a credible cause of a problem then it was assumed the SPM was to blame?
RR Correct.
JB Who?
RR PO management and Fujitsu.
JB How did they come to this assumption? Who expressed it?
RR That was my feeling - if we couldn't find a problem in the code or the data then there is no problem. It must be the SPM.
JB Did you understand action was being taken against SPMs?
RR No.
JB Were you aware people were being prosecuted?
RR Not at the time.
JB Were you aware anyone from SSC being asked to be a witness in a criminal prosecution at Kingston Crown Court - I am referring to Tracy Felstead.
RR No. Can't remember.
JB You describe your work on Horizon as "firefighting"?
RR Yes it was quite hectic at times. Sometimes there'd be a bit of a panic on... all hands on deck to try to fix a problem as soon as possible.
JB takes RR to his second High Court Witness Statement: "I do not recall Fujitsu carrying out any analysis of Transaction Corrections to try to identify if there may be an underlying software error.... I also think it is wrong to say that software errors would occur uniformly across branches, as I explained in the example in para 10 above. My experience was that software errors occurred in very specific factual circumstances, which is why they were challenging to identify and correct."
RR says he has no recollection of the PO ever asking SSC to analyse what they thought was a fault.
RR has not heard of Andy Dunks or knew of his job role. No liaison with him. Andy Dunks supplied the courts with witness statements and oral evidence in court about what RR's team had done in response to calls to the SSC by SPMs.
JB [Do you know why] Dunks, a member of the customer service team, was selected to give evidence about what you and your team were doing at SSC?
RR No.
JB Were you ever party to a discussion… or did you ever hear why Dunks, the crypto key manager in the customer service team was giving evidence about what was happening in your team rather than someone from your team?
RR No.
JB Were you ever asked to give evidence?
RR Don't think so.
JB If they had, would you have described all the problems you have told us about today?
RR Probably, yes.
JB Did you ever hear discussion about who had attended court on behalf of Fujitsu to give evidence about Horizon?
RR No recollection of it.
[So Fujitsu sent a useful idiot, Andy Dunks, along, to give evidence in court to vouch for Horizon, despite him knowing nothing of the problems SSC were dealing with. Feels a bit corrupt, that; in the same way the Post Office sent three useful idiots to Panorama to give an on the record statement about there being no remote access to Horizon, when it knew there was.]
JB is now talking to RR about specific Horizon problems he raised in his High Court witness statements, including at least one where they attempted data recovery and could not retrieve it. And other occasion where RR found a hardware fault.
"An SPM seemed to be switching her laptop off before 6.00pm. Because the power was switched off the Laptop could not generate the end of day financial markers, consequently the accounts were not processed correctly which resulted in transaction data not being sent to the banks and utility companies so Fujitsu missed SLA deadlines. I was asked to investigate; the SPM (a very experienced lady) insisted she was not turning the machine off but the log files on the counter showed that the laptop was being powered down. I arranged for the laptop to be swapped out and returned to Bracknell for testing and found that when the screensaver button was pressed the power to the machine was switched off. When I disassembled the machine I discovered the fault —during the build the wires had been cross connected. I brought this to my manager's attention as I felt it should be investigated further; a few days later he called me over and informed me that the manager of the section that assembled the Laptops knew about the issue already as one of his engineers had told him that he had inadvertently mis-wired several laptops that had been sent out to SPM's. I was told that no further action was to be taken and I was instructed to record the fault as no fault found or something similar —the incident was hushed up without senior management or the Post Office being made aware of it. The faulty laptops remained in general circulation, but as none of the other SPM's used the screensaver button regularly it did not cause a problem, however this raises questions regarding communication, honesty and transparency within and between departments within Fujitsu. For example, mistakes were made when releasing updates to the software and it is feasible that a programming error could have been rolled out to the estate and a fix rolled out a few days later, without anyone in the wider organisation being informed. In this scenario, if an SPM had problems with their accounts then by the time SSC were asked to investigate the fault would have already been rectified so we would not have been able to duplicate the error. It would have seemed that the only logical explanation was that the SPM was to blame, with potentially catastrophic consequences for that individual.''
JB asks RR who asked him to "hush up" the hardware issue.
RR "Mik Peach"
[we take a break]
JB asks if he knows Anne Chambers.
RR No.
JB Recall her as working at SSC?
RR Yes.
JB Her expertise?
RR Very good on the accounting side and databases.
JB Did she have knowledge of the Horizon software or data integrity?
RR Can't remember.
JB Did you speak to her about giving evidence in court?
RR No.
JB Do you remember if she was?
RR No - I don’t remember anyone being selected.
JB is being taken to Andy Dunks witness statement in the @CastletonLee case.
JB Do you recognise the format of this call log?
RR No.
JB Were they printed or always on the screen?
RR Always on the screen.
@CastletonLee JB says the call was opened on 25 Feb 2004 - within your time working in SSC. Underneath it says "Postmaster reporting that they are getting large discrepancies for the last few weeks". JB wants RR's help re understanding the information. RR can't remember this type of data being set out like this.
JB "New call taken by Kujinder Bhachu: [SPM] reporting that they are getting large discrepancies".
RR cannot remember much about this at all.
@CastletonLee JB "NBSC have been in contact with the SPM and can not find any user error".
RR can't remember NBSC.
JB "Checked tivoli events and health checked. Site is health checking okay".
RR says tivoli was a background monitor of activity. RR's memory of the time (18 to 19 years ago) not great here.
RR now discussing the audit trail when SSC engineers went into the code and how it could create error flags. The actual identifying codes they used for the engineers might cause errors. So they only used it when they knew it wouldn't cause problems. RR describes changing Horizon data as it came into their servers and whilst it was on the servers to save going into branches. He also remembers blanking out SSC engineer identifiers, but can't remember why it was done - perhaps to change reference data parameters. RR talks about hacking the Horizon system by creating a session in Riposte to insert transactional data. "We were doing it through the back door".
JB Why describe it as a hack?
RR It wasn't the way things were supposed to be done.
JB Why do it then?
RR Only way we could get the system back up and running. It was a workaround.
JB Was it just you doing this?
RR No, everyone was doing it. We had unrestricted access.
JB Did your managers know about this?
RR Oh yes.
JB How?
RR My colleagues taught me how to do it.
JB Was this hack written down?
RR Don't know. I know there were problems when the auditors came in and found out we were doing it. At first nothing was written down. We were flying by the seat of our pants. It was a mess. Eventually the documentation was created.
RR taken to a document created by Fujitsu on 2 August 2002 which he wouldn't have seen. It's an audit report saying that SSC have unrestricted and un-audited privileged access to all systems including Post Office counter PCs.
JB Is this true?
RR Yes.
JB Was it widely known?
RR Yes. Within SSC.
JB But not outside?
RR No - I doubt the PO would have known. It's only looking back I see how it's "pretty shocking" how much access we had.
RR recalls hot fixes made to individual branches - sometimes they told a Subpostmaster to leave a computer on and not touch it. Other times they did it during lunch hours without telling SPMs and sometimes there was a way of logging on as an SPM (but he doesn't remember this clearly and is concerned he might be confusing a memory with documentation he saw during the High Court trial).
[I've written a chapter on evidence and memory in my new book. Our memories are very unreliable fact-retention units and every time we retrieve a memory it changes without us knowing it. It is very easy for our memories to merge our direct experience of an event and a later experience of encountering the event through other means]
Synopsis posted by Leighton Associates 17 November 2022 – “Trizzy’s Book Club: Book 2 of 3”
[We have stopped for lunch and will reconvene with Mr Roll at 1.45pm]
RR is being taken through an OCR form - Operational Correction Request (these came up in the Horizon trial). Now I know what an OCR is. This document creates a paper trail for changes to the Horizon system; the request, who signed it off, whether it was tested. RR has no recollection of this OCR document, dated 2001.
JB Did you have to fill something like this out when you made changes?
RR No memory of this document before.
JB takes RR through the 29 Jan 2001 protocols for making changes to Horizon system. JB calls it "very involved" and "complicated". RR has no recollection of following these protocols. He says he isn't sure whether that means he didn't - he just can't remember. His recollection is that he doesn't think they followed those protocols, but just can't remember for sure.
JB has finished his questions for the day. Inquiry breaks for 5 mins - there will be questions from barristers representing the Subpostmasters after the break. Sam Stein KC will be first with the questions for RR.
SS You discussed with JB what can happen when a system used by an SPM has a dodgy on-off button. That could and did lead to loss of data integrity.
RR Yes.
SS Can we assume losses of power would have the same effect?
RR It would have that potential.
SS So potential for that when the power goes - and connectivity issues. Could that lose data?
RR Yes but the data would hopefully still be on the Computer in the branch.
SS Would it always?
RR If there was a power failure there is always the potential hardware damage to the disk or boards. If that made the computer inoperable, if you can't recover data from the disk, without a paper trail, you would not know what happened.
SS Was this ever explained to SPMs?
RR I don't remember doing so and I'm pretty sure I didn't. I don't know if anyone else did.
Flora Page FP now asking questions. FP wants to know about tivoli - were there any routine or processes around it?
RR Can't remember.
JB has one last question. JB raises the third witness statement RR said he did for the High Court case. JB thinks it might be an AMENDED second WS rather than a third. Thinks this might be the cause of some earlier confusion. Asking him about it now, particularly the hardware problem which caused a hard power off to the unit. Clarifies RR was told not to put details about the problem in his report at the time.
RR is thanked by Sir Wyn Williams for assisting the inquiry and the many years of assistance he has given in previous years.
The journalism on this blog is crowdfunded. If you would like to join the “secret email” newsletter, please consider making a one-off donation. The money is used to keep the contents of this website free. You will receive irregular, but informative email updates about the Post Office Horizon IT scandal.
Republished with permission. Click here to sign up to Nick’s newsletter on the PO Scandal.
Additional text and image of Mr Roll: Nick Wallis, ©2023.
UPDATE: Nick Wallis interviews Paul Scully MP, who says former Post Office Chief Executive (2012 – 2019) Paula Vennells should be stripped of her CBE. Investigating the Post Office Scandal / Ep35 - Paul Scully MP, former Post Office minister (audioboom.com) (49m 37s)