NAB has detailed its 2023 Teradata decommissioning publicly for the first time, revealing how the data team rebuffed pressure to build a ‘like’ replacement, and its surprise discovery of “payments and lending processes” running inside Teradata instead of the core banking system.
Image credit: Joanna Gurry/LinkedIn.
The bank migrated data and “use cases” for that data from its 26-year-old Teradata platform to Ada at the end of 2023.
Ada is the name of NAB’s second-generation data platform. It has Databricks on AWS at its heart, but also has Fivetran, PowerBI and Azure-based components.
NAB publicly marked the decommissioning of Teradata at the end of 2023. It later emerged that the bank live-streamed the cutover internally.
But until now, it had been unclear just how much behind-the-scenes work went into unwinding Teradata and reaching that point of transition.
That effort is now on the public record, courtesy of a presentation at Databricks’ Data+AI Summit in San Francisco earlier this month.
While 18 months have elapsed since Teradata was switched off, the timing may simply be a by-product of the data team having sufficient time to decompress.
Data platforms executive Joanna Gurry said the decommissioning was one of several related programs of work run in parallel, which pushed the data team to its limit.
“Retiring a legacy application is really difficult,” Gurry said. “It’s a strategic decision that significantly impacts your cost and your risk profiles – and, in fact, stability, cost and risk were key drivers in why we chose to decommission Teradata.”
“Not only that, but we also decided to build a brand new data platform that we call Ada using Databricks, and because that wasn’t quite enough we also decided to move from batch processing to data pipelines, and then we also chose, on top of all of that, to take several applications that were built within Teradata and build them back into the core banking and payment systems as well at the same time.
“We also decided to load 50 data sources at the same time that we were decommissioning Teradata for our financial crime operations team.
“We did all of this in the same financial year, at the same time.
“Looking back now, I think it was a bit of a crazy plan. And I can tell you, it took blood, sweat and tears to get us across the line, and at the end all I had left were tears.”
She later added: “A few of them were tears of joy that it was over, [but] also deep pride in what we had done as a team.”
Teradata’s tentacles
Deep integration into banking processes was to be expected for a 26-year-old system.
Gurry said the system “had its tentacles into many, many business processes, models, reports and extracts” and supported 956 use cases across the bank.
“These were absolutely business-critical functions and services that needed to keep running smoothly during the entire transition,” Gurry said.
A natural challenge was that over such an extended timeframe, people who had designed or built various use cases had left NAB, and Teradata’s exact role in processes was undocumented.
“No one knows how it all hangs together,” Gurry said.
“Some teams … didn’t even know what was in their own databases, so every [use case earmarked for transition] was kind of like a project in itself.
“We received some really big surprises along the way – processes that no one knew about [that] were underpinned by complex transformation rules and extracts that were flowing in but also out, and sometimes even in a circular way.”
Gurry – who previously compared the decommissioning of five Teradata cabinets in NAB’s data centre to “watching the Terminator die five times in a row” – compared the approach to use cases as like an episode of ‘Storage Wars’ – a US show that tracks people buying abandoned storage units without knowing the contents.
“We decided to target this particular unit, we made the financial commitment, we had the whole bank behind us, finally agreeing that the time was nigh and we walked up to the roller door and lifted it up and that’s when the real work began,” Gurry said.
“We started finding out exactly what was inside.”
Among the surprises were “business-critical processes and applications [that] had been built inside the database that we didn’t know about.”
“We had payments and lending processes that, instead of running in our core banking systems, were running in our data and analytics platform, in Teradata,” Gurry said.
“There were also really high-impact services that were so important to our customers that we had to protect and be very careful with – services supporting our customers in financial distress or financial hardship, as well as bereavement support and fraud detection.
“There were some regulatory reports running off it as well.”
Applications that were found to be incorrectly running in Teradata were ultimately repatriated to NAB’s core banking system, although Gurry noted the initial expectation of the business was to rebuild them as-is in Ada.
“Because they had been in Teradata, everyone was expecting us to rebuild that in Databricks,” she said.
“However, [instead of] automatically rebuilding these on a data and analytics platform, we stepped back and said, ‘Look, if we were building this application from scratch, would any of the architects say, ‘The first thing you need to do is build a data intelligence platform’?’ Of course, the answer is ‘no’.
“So, we worked really closely with the enterprise architects and all of the application teams that run our core banking systems and had them rebuild those applications for us back in the core systems where they belonged.
“That had to all be done by the time that we were due to exit Teradata, but we had wonderful executive support which made all of those conversations a lot easier, and in fact the teams that run those business systems were really glad to have that functionality back where it belonged in their domain so that they were controlling [it] end-to-end.”
NAB also uncovered “significant dependencies” between Teradata and “a second data warehouse that was on Exadata [that] had data coming into and out of Teradata.”
“We didn’t have time to decommission that system as well, so we took copies of the most critical tables that we needed and we put those into Databricks as a source,” Gurry said.
This is a temporary measure; the data team is now progressively working to “turn off the feeds into those tables”.
“That means we’re hollowing out that source of data from the inside as we go,” she said.
A ‘like’ replacement
Internal pressure to replicate incorrectly housed use cases from inside Teradata to Ada spoke to a broader challenge that the data team encountered in the migration program.
“In doing a really large-scale migration like this, there’s tremendous pressure from the business to absolutely rebuild 100 percent and migrate everything over,” Gurry said.
“The business is going to insist that every report, every model, every data pipeline must be preserved, and you can’t leave anything behind just in case.”
This ran counter to a desire to treat the project as a “once-in-a-lifetime [re-platforming] opportunity” – a chance to take stock and declutter, identifying and transitioning only active use cases across to Ada.
“One of the things that we did, which I thought was really valuable, was identifying what’s actually being used and trying to decrease the scope [of the migration],” Gurry said.
“There was a lot of pushback. It’s really difficult to keep relationships intact when you’re trying to get the business to let go of stuff.
“They are worried because they often don’t know very confidently what is needed, so they want to hang on to everything so that we don’t make a mistake.”
Gurry said the data team used a mix of “probing questions”, system logs and usage patterns to judge if a use case really needed to be recreated in Ada.
“If someone said, ‘We still need that report’, we said, ‘Great. Show us the last time you used it, and we’ll add it to the register’.
“[We found] people just didn’t come back [to us]. They couldn’t find anyone who had been using it.
“We also heard ‘The regulator asked questions about that data, and we need to have it ready going back 20 years’, and we said, ‘Show us the last time the regulator asked for the data so that we can determine what tables are involved’. The regulators never asked for the data.
“Asking those probing follow-up questions had a massive impact.
“We didn’t just rely on business input either. We actually went to the system, checked out logs, dove into hit rates and examined usage patterns, and out of thousands of reports, hundreds of use cases, we could actually see within the system itself what was being used.”
Using this approach, Gurry said, NAB cut the number of use cases supported by Ada from 956 in Teradata “to just over 400”.
Gurry said another way of prioritising what to transition was to ask which users would perform acceptance testing and sign off that the use case is working as intended.
“For the things that were really important, everyone wanted to test and own [them] and make sure that the numbers were correct.
“Then, there were vast swathes of stuff that we couldn’t find anyone who was interested in testing at all, so I think that’s a really good indicator for how to reduce your scope as well.”
To assuage user concerns, the data team created a “comprehensive” backup of everything in Teradata – “not just the data, but the schema and the models, report definitions, example outputs, SQL queries, metadata, HTML code – you name it” – which is in cold storage with an unnamed cloud provider.
“This provided a safety net for the program but also … a security blanket for the business as well,” Gurry said.
“The archive is not really used, and because it’s in cold storage in the cloud, it’s an economical way of still hanging onto that, just in case something was needed one day.
“But again, no one’s really using it, so we’ll look to retire that at the right point.”
Despite all the preparation, Gurry noted that not all use cases were successfully migrated from Teradata to Ada.
“By the end of the migration, about 95 percent of the use cases had been successfully ported over, and they were running faster, with better quality data, [and] amazing outcomes, [with] great opportunities for the business to be leveraging all of this stuff in entirely new ways.
“But there were about five percent of the use cases where the business rules proved extraordinarily difficult to recreate.
“Although they looked correct on the reports or some of the queries that the business analysts were recreating, when the analysts in the business tried to reuse that data to produce new metrics or new reports, they were getting incorrect answers.”
In some cases, Gurry said this was the product of rules being flawed for years, “and not as accurate as the business had understood them [to be].”
In other cases, the transitioned use cases were defective.
“Even after we had decommissioned the [Teradata] system, we had to do quite a bit of refactoring for about two or three months,” she said.
“If I had to do something again differently, I would actually carve time out, have many checkpoints scooping up objections or queries, and spend a lot more time embedding that change back into the business areas.
Honouring the bank’s heritage
When Gurry first spoke of the bank’s decision to livestream the end of the Teradata era, she noted the deep connection that parts of the business had “with the heritage of our existing systems.”
As it turns out, the use of this language was no accident and was in fact a strategic play to avoid using the word ‘legacy’ to describe the Teradata system.
Gurry said the bank and data team were purposeful in wanting to “honour past value”.
“We avoided the term ‘legacy’ and described the Teradata system and the teams [using it] as ‘heritage’,” she said.
“Now, this might seem like a really tiny and small semantic difference, but it signals something really important about how much you value their IP [intellectual property] and their institutional knowledge regarding that system that you’re decommissioning.
“It demonstrated respect for the people who contributed to that asset over the years.”
It was also an acknowledgement of the connection that different parts of the bank had with the Teradata system.
“For people who worked on something that they loved and felt deeply connected to, it was almost part of their identity. When we said that we were going to shut it down, it was profoundly disorienting to a few of them, and it kind of stung a bit. It was like their work life had kind of tilted on its axis a little bit,” Gurry said.
“We gave [those people] a high-profile role in the transition, made sure that they had a unique experience in building the new platform and actually when we finally shut down Teradata we marked the occasion by bringing a film crew into the data centre and live-streaming the shutdown of the hardware and the powering down to the entire company.”
Leadership and top-down reinforcement were also required in some cases to encourage change.
“Many of our data modelling experts, for example, had just spent their entire careers working on this 26-year-old Teradata system, and they didn’t know that the methods that they used to do data modelling were just one choice, that there were other ways of doing it,” Gurry said.
“This is where strong leadership became really crucial.
“When teams push back and they just want to revert to their familiar methods, you really need leaders who can lean in in a technical way and take action and correct course. And sometimes, and especially in our case, that meant we had to make some really difficult personnel choices as well.
“If I could do one thing differently, I would have better anticipated the magnitude of this challenge because I really underestimated how difficult it would be for the modelling team to feel comfortable with making that massive change.”