Federal IT systems at risk of 'critical failure,' Trudeau warned in memo

[OceanBonfire]

Jordan Press - The Canadian Press

Newly released briefing notes for Prime Minister Justin Trudeau describe the dire state of federal computer systems, which deliver billions in benefits and are on the precipice of collapse.

Officials briefing Trudeau after his party's re-election noted "mission-critical" systems and applications are "rusting out and at risk of failure," requiring immediate attention from his government.

Some systems are pushing 60 years old and built on "outdated technology" that can no longer be maintained.

The Canadian Press obtained the documents through the Access to Information Act, and much of the text has been blacked out because it is considered sensitive advice to government.

One of the few visible lines says the public service will be working on projects "to stabilize mission-critical systems."

The Liberals promised in the fall campaign to improve services for Canadians.

These back-office problems often require long-term spending commitments and improvements that aren't easily seen by voters, which means they aren't usually top-of-mind issues for politicians. But there has been a growing belief inside the government that how satisfied citizens are with digital services is linked to how much trust they place in their government.

Nowhere is the issue of antiquated systems more pronounced than Employment and Social Development Canada, which oversees child, parental, senior and employment insurance benefits.

The Liberals have already made multiple changes to the federal social safety net that required programming changes to old systems. The documents to Trudeau suggest the aged systems pose a problem for more changes the Liberals have promised.

"The complex array of existing programs and services means that future program changes, to continue providing Canadians with the programs and services they expect when interacting with their government, will need to account for pressures on legacy IT systems, which are facing rust-out and critical failure," part of the briefing binder says.

"These aging platforms neither meet the desired digital interaction nor are capable of full automation, and thus are unable to deliver cost-savings through back-office functions."

For example, the system that runs old age security is almost as old as those now reaching the age to benefit from it.

A separate document says upgrades are taking longer than planned because of procurement delays and the complexity of projects.

Funding pressures are coming in part from the requirement to maintain existing technologies longer than originally anticipated, the document says.

Often, officials didn't look to upgrade these old systems so long as they continued to work, said Andre Leduc, vice-president of government relations and policy with the Information Technology Association of Canada. He said the majority of departmental IT budgets are used to keep old systems running, leaving little spending room to deploy new technology like cloud computing.

"It's not that they're hitting the panic button. There is no panic yet," Leduc said.

"There is a lot of concern, both within the bureaucracy, within the political layer, and within industry about we need to get this ball rolling. We need to help government digitally transform."

Leduc said it won't be an easy fix because of the money required, and getting the government to move ahead with major IT work after getting burned on high-profile projects. The most glaring example is the Phoenix pay system that has left public servants overpaid, underpaid or not paid at all.

https://globalnews.ca/news/6493713/canada-federal-computer-systems-trudeau/

https://www.ctvnews.ca/politics/federal-it-systems-at-risk-of-critical-failure-trudeau-warned-in-memo-1.4793335

https://www.cbc.ca/news/politics/federal-it-systems-critical-failure-1.5448871

 

[Brad Sallows]

"Mainframe modernization" isn't rocket science, and doesn't have to be done all at once (ie. a single budget-blowing expenditure).  I wonder that the "experts" in government can't do what private companies can.

 

[FJAG]

It's really the scope of the thing.

There are tens of thousands of computers and hundreds or thousands of back end systems and server based programs that have been deployed over the last forty or so years. I've dealt a bit with IM Gp and I know they face tremendous challenges when it comes to network and data security, obsolescence and business transformation every time one particular system needs "tuning". The problem is greatly magnified when you go across the whole of government and have to deal with smaller departments whose IM teams are not as robust as IM Gp's are. Just look at the fiasco when they tried to amalgamate the pay system.

When I did a three year project with JAG we had to consolidate hundreds of user accounts and several dozen back-end applications run out of dozens of different stand-alone building/base specific local area networks. We did that by migrating everything into a single location server farm with a Protected B wide area network that could be securely accessed from any unclassified DND WAN computer through an encrypted application that ran off a Secret level USB drive issued to every member of JAG. It took us three years to design and build the servers and records management system (using 2007-8 technology) but after that time we still didn't have a functional case management nor knowledge management system. At the time IM Gp was a bit hesitant of letting us work towards a Protected B environment because the plan was to have a Secret level DWAN (Not the then existing TEMPEST system) within a few years. (I haven't been following what's been going on these days but I'd put money on the fact that the DWAN hasn't essentially changed one bit since I left ten years ago.) On top of that our system had to be a "day-forward system" because we simply didn't have the ability to aggregate and bulk enter our decades of legacy data into the new system (in fact because of DND's cockamamie system of local user management) a vast amount of the legacy electronic data had been deleted every time someone was posted to a new job in a different building in Ottawa or base.

That was just one small group within DND which, while it made things better for us, pretty well put one more user specific IT system into the mix for IM Gp to manage the back end. Multiply that by a factor of several thousand and you'll see what the current state of the system is.

:cheers:

 

[kev994]

Sounds like a problem for the lowest bidder.
About 4-5 years ago they dropped tempest computers on all of our desks, and then apparently abandoned the project, 2 of ~15 are in use. It’s still wasting space, we can’t get rid of them.

 

[Swingline1984]

Sounds like a problem for the lowest bidder.

Why spend money on IT when SSC can add 25 more Directors? If they ever go looking for efficiency they can start there.

 

[Colin Parkinson]

Part of the blame can go to the politicians from both side of avoiding issues or providing "quick fixes". But the meat of the blame goes onto the Senior Management of the Public Service, most have no idea how any of it works and will only do that which ensures their bonuses and/or promotions. I am sure there are a few fighting the good fight, but not enough. When they do come up with a "fix" it's always at the users expense and generally bogs down their productivity. Everyone loves the centralized/ single system stuff. But when it breaks it means everyone in the Department is screwed. I pointed out that a rough estimate of costs to the Department by the daily hangups of Citrix was costing an estimated $46,000 a day, for 15 minutes of downtime per day across the department. (average hangs in one day for just Citrix) 

 

[macarena]

I think a good point of start will always be to whom delegate the task of new developments, to inside the house, or to private companies?
The private companies don't have the knowledge of whole shit envolved on the old systems, as the inside users and supporters have. Then these companies will quote prices based on the profit they want to make, and not based on the real charge of work it will take. Well, they aren't commited to the cause.
To make the job inside the house, they will surely build up a new team for that. I mean, reorganise the development team with new buddies, with experienced project managers, and the most important and dangerous tier, it will be to move out the unfit strategic staff, since they will be refusing new ideas, new directions, due to their lack of knowledge on the actual solutions the market has proved to be effective for many years. Personally, I do consider how long a certain technology has being effective and etternal, since one would like that the new systems will last for a long while (at least for the next 50 years, Jesus).

 

[Brad Sallows]

Private companies which specialize in mainframe modernization exist.  Yes, they like to be paid, as do "inside" workers.  But because it's what they do, I can hazard a guess that the likelihood of delivering successfully is higher.  Paying 150% of budget for a successfully completed project, for example, would be preferable to paying 100% of budget for nothing.

 

[garb811]

Private companies which specialize in mainframe modernization exist.  Yes, they like to be paid, as do "inside" workers.  But because it's what they do, I can hazard a guess that the likelihood of delivering successfully is higher.  Paying 150% of budget for a successfully completed project, for example, would be preferable to paying 100% of budget for nothing.

Exactly, just like the update/upgrade of our email system, a private company ensures success, right?

This article is from 2018...so, 8 years and not even counting anymore. I don't remember the last time I heard anything about this, I'm guessing it was just left to fade away into the ether.

Government email project still in limbo, costing millions
BY MONIQUE SCOTTI GLOBAL NEWS
Posted March 13, 2018 11:34 am

A massive government email project that has cost taxpayers at least $100 million is still in limbo six months after Global News reported on the program’s ongoing troubles.

The Email Transformation Initiative (ETI) was, as of last September, heading into its sixth year and appeared to have stalled indefinitely, with internal documents revealing that attempts to set up other projects to compensate had also faltered.
...

On a related note, I've been getting a notice every 15 min for the last three weeks telling me that I've passed an update deadline. It won't install automatically, won't install manually, help desk is scratching their heads since even admin rights won't force it to happen. I'm taking bets on how long it will be before it causes me to fail the security check on login and my computer becomes a brick.

 

[dapaterson]

Exactly, just like the update/upgrade of our email system, a private company ensures success, right?

This article is from 2018...so, 8 years and not even counting anymore. I don't remember the last time I heard anything about this, I'm guessing it was just left to fade away into the ether.
On a related note, I've been getting a notice every 15 min for the last three weeks telling me that I've passed an update deadline. It won't install automatically, won't install manually, help desk is scratching their heads since even admin rights won't force it to happen. I'm taking bets on how long it will be before it causes me to fail the security check on login and my computer becomes a brick.

You computer will be bricked when the hard drive fills up with multiple copies of the update downloaded and stored daily.  Which is also hammering the network.

 

[FJAG]

Did I mention above how much the requirement to have a bilingual system resulted in our having to reject numerous perfectly usable off-the-shelf software solutions because they weren't customizeable and that the few inadequate solutions that were customizeable would have had us indentured to the provider forever-after anytime we would need to make an amendment?

Kinda sucks when the aim was to get rid of a whole bunch of custom software apps developed in house on Microsoft Access.

:cheers:

 

[Colin Parkinson]

Private companies which specialize in mainframe modernization exist.  Yes, they like to be paid, as do "inside" workers.  But because it's what they do, I can hazard a guess that the likelihood of delivering successfully is higher.  Paying 150% of budget for a successfully completed project, for example, would be preferable to paying 100% of budget for nothing.

Government workers getting paid? Well that would be a novel approach..... 8)