Editor's Blog6 mins ago
Covid - The Missing Uk Cases.....
35 Answers
seems 16,000 cases were omitted from figures last week because of an "IT Error"
https:/
on various internet platforms, it's being suggested the "error" was because whoever loaded the data into the Excel spreadsheet did so on a case-by-column basis, rather than a case-by-row basis, meaning the spreadsheet maxed out 64 times earlier than expected.
aside from wondering if excel is a suitable database medium for so much data (it's cheap therefore it's good, eh?), these are schoolboy errors that should have been reviewed out before the system went live.
shouldn't they?
https:/
on various internet platforms, it's being suggested the "error" was because whoever loaded the data into the Excel spreadsheet did so on a case-by-column basis, rather than a case-by-row basis, meaning the spreadsheet maxed out 64 times earlier than expected.
aside from wondering if excel is a suitable database medium for so much data (it's cheap therefore it's good, eh?), these are schoolboy errors that should have been reviewed out before the system went live.
shouldn't they?
Answers
Best Answer
No best answer has yet been selected by mushroom25. Once a best answer has been selected, it will be shown here.
For more on marking an answer as the "Best Answer", please visit our FAQ.
This is an age old problem with Government IT.
I remember going to the FSA(as it was at the time) along with a dozen other people from other companies to rip them a new one over MiFID I.
To be fair though one of the reason for using XL is probably due to it being already in most companies. And the reason for using an old file format is because you will find not everyone updates to the latest, just look at the recent problems in the NHS itself using old versions of Windows.
What they could have done is write a Macro (from Excel) that created a file that could be sent. However then you have the issue that many companies dont like Macros from externals due to security risks so maybe they should have just opted for a csv which is pretty universal.
Then of course comes the transmission problem. The public sector seems to like ftp/sftp however the delivery is not guaranteed so you can end up with half a file and not know it so there is need to do checks on file headers and trailers etc although this is not fool proof. So the best thing to do is to process the file on arrival (a 'program' can be triggered auto on file arrival ) do a count and send back a reply to the sender who can check the results.
But they didnt ....
I remember going to the FSA(as it was at the time) along with a dozen other people from other companies to rip them a new one over MiFID I.
To be fair though one of the reason for using XL is probably due to it being already in most companies. And the reason for using an old file format is because you will find not everyone updates to the latest, just look at the recent problems in the NHS itself using old versions of Windows.
What they could have done is write a Macro (from Excel) that created a file that could be sent. However then you have the issue that many companies dont like Macros from externals due to security risks so maybe they should have just opted for a csv which is pretty universal.
Then of course comes the transmission problem. The public sector seems to like ftp/sftp however the delivery is not guaranteed so you can end up with half a file and not know it so there is need to do checks on file headers and trailers etc although this is not fool proof. So the best thing to do is to process the file on arrival (a 'program' can be triggered auto on file arrival ) do a count and send back a reply to the sender who can check the results.
But they didnt ....
> In order to process this data the scribes have been using MS-Excel (which I have used extensively at work and which I use at home to track my finances and gas bills).
It's worse than that. They never should have been using Excel at all, but the version of Excel they were using was Excel 2003. Since Excel 2010, the number of rows that Excel could handle increased from 65,535 to 1,048,575.
It's worse than that. They never should have been using Excel at all, but the version of Excel they were using was Excel 2003. Since Excel 2010, the number of rows that Excel could handle increased from 65,535 to 1,048,575.
//They never should have been using Excel at all, but the version of Excel they were using was Excel 2003.//
Spot on, ellipsis. It beggars belief that such an important task was left to somebody using such outdated software. Everybody who subscribes to MS Office has the latest version. The government is fannying about, closing and opening businesses on a whim (well, it may as well be on a whim for all the reliability they can place on their data). But as you say, Excel is a totally unsuitable tool, even in its most recent version, for such an exercise.
Spot on, ellipsis. It beggars belief that such an important task was left to somebody using such outdated software. Everybody who subscribes to MS Office has the latest version. The government is fannying about, closing and opening businesses on a whim (well, it may as well be on a whim for all the reliability they can place on their data). But as you say, Excel is a totally unsuitable tool, even in its most recent version, for such an exercise.
TTT - It's not just in IT that the public sector has the problem of paying peanuts & employing monkeys. It is, of course, much more noticeable in IT and is the basic reason that so much public sector IT work is outsourced. This outsourcing is frequently very poorly specified because the user departments (especially their so-called managers) don't have much of a clue about how to go about it. We see the chaotic results far too often. (Just for info, my IT debut was in 1964 & I've seen a lot of systems since then.)
remember this little beauty?
https:/
The problem is they spend gazillions to cock it up then pay through the nose to get it fixed. If they'd only hire the right people at the right money in the first place it would end up cheaper.
https:/
The problem is they spend gazillions to cock it up then pay through the nose to get it fixed. If they'd only hire the right people at the right money in the first place it would end up cheaper.
Large amount of taxpayers money have been spent outsourcing ‘Track and Trace’ work. The contracts are awarded without any tendering process, and often to firms without any healthcare experience.
// Deloitte to manage the logistics of national drive-in testing centres and super-labs; Serco to run the contact tracing programme; Palantir and Faculty A.I. to build the COVID-19 datastore.
Fitting with the general theme of complexity, Deloitte then nominated Serco, Sodexo, Mitie, G4S and Boots to staff and manage operations at the testing sites. Those unable to access the testing sites were advised to request home testing kits produced and processed by diagnostics company Randox (in a contract worth £133m) and dispatched by Amazon. //
A lot of private firms have made huge amounts of money delivering a service that is not up to scratch.
// Deloitte to manage the logistics of national drive-in testing centres and super-labs; Serco to run the contact tracing programme; Palantir and Faculty A.I. to build the COVID-19 datastore.
Fitting with the general theme of complexity, Deloitte then nominated Serco, Sodexo, Mitie, G4S and Boots to staff and manage operations at the testing sites. Those unable to access the testing sites were advised to request home testing kits produced and processed by diagnostics company Randox (in a contract worth £133m) and dispatched by Amazon. //
A lot of private firms have made huge amounts of money delivering a service that is not up to scratch.
"Everybody who subscribes to MS Office has the latest version."
Only if on MS365 (Cloud). If it is installed by the company from their own servers then the company may well have access to it but choose not to install it. There actually is very good reason for this, particularly for large companies, as often releases are full of bugs.
"It's worse than that. They never should have been using Excel at all, but the version of Excel they were using was Excel 2003. Since Excel 2010, the number of rows that Excel could handle increased from 65,535 to 1,048,575."
I explained why they may use an old version in my post of 8:45. Did you read it or have I not explained very well?
Only if on MS365 (Cloud). If it is installed by the company from their own servers then the company may well have access to it but choose not to install it. There actually is very good reason for this, particularly for large companies, as often releases are full of bugs.
"It's worse than that. They never should have been using Excel at all, but the version of Excel they were using was Excel 2003. Since Excel 2010, the number of rows that Excel could handle increased from 65,535 to 1,048,575."
I explained why they may use an old version in my post of 8:45. Did you read it or have I not explained very well?
Palantir, a US analytics company co-founded by Trump ally Peter Thiel.
Faculty AI used to be called ASI Data Science. They were responsible for the Vote leave campaign using the disgraced Cambridge Analytica to steal user data. Faculty owner Marc Warner is a friend of Dominic Cummings. Warner’s brother Ben was recruited by Cummings into No.10 as their IT procurement advisor.
Faculty AI used to be called ASI Data Science. They were responsible for the Vote leave campaign using the disgraced Cambridge Analytica to steal user data. Faculty owner Marc Warner is a friend of Dominic Cummings. Warner’s brother Ben was recruited by Cummings into No.10 as their IT procurement advisor.
//Only if on MS365 (Cloud).//
I think that's the only way you can get it now (legally, anyway).
I take your point about new releases containing bugs. However, the latest release of Excel (enabling 1m+ rows to be used) has been around for ages and should be fairly robust for the comparatively simple task of consolidating a number of inputs. As well as that, if contributors have older versions, their submissions will be able to be handled by the later versions.
All of these potential drawbacks should have been identified and addressed before now. Whatever the remedies required they should have been implemented and the whole fiasco is a major blunder. This keeps being portrayed as a "glitch" or an "IT problem". It is not. It is simple ineptitude for which there is no excuse.
I think that's the only way you can get it now (legally, anyway).
I take your point about new releases containing bugs. However, the latest release of Excel (enabling 1m+ rows to be used) has been around for ages and should be fairly robust for the comparatively simple task of consolidating a number of inputs. As well as that, if contributors have older versions, their submissions will be able to be handled by the later versions.
All of these potential drawbacks should have been identified and addressed before now. Whatever the remedies required they should have been implemented and the whole fiasco is a major blunder. This keeps being portrayed as a "glitch" or an "IT problem". It is not. It is simple ineptitude for which there is no excuse.
> I explained why they may use an old version in my post of 8:45. Did you read it or have I not explained very well?
There's no excuse for using Excel 2003 in 2020, especially for such a critical task. The biggest problem with Excel handling large quantities of data was always the 65,535 line limit. It's simply not the appropriate tool for the job.
There's no excuse for using Excel 2003 in 2020, especially for such a critical task. The biggest problem with Excel handling large quantities of data was always the 65,535 line limit. It's simply not the appropriate tool for the job.
Related Questions
Sorry, we can't find any related questions. Try using the search bar at the top of the page to search for some keywords, or choose a topic and submit your own question.