HOME BLOG
← Back to all articles

Don't Fix What's Visible. Fix What's Causing It.

Novell NetWare 3.12 installation disks

The first lesson of the materials handling company: the problem everyone complained about was never the real problem.

When you walk into a broken IT environment for the first time, there is a strong pull towards quick and easy fixes to show fast results. It looks good but it does not address the root-causes.

The pressure is real when people are complaining that their computers are slow, the email system keeps dropping messages, applications freeze mid-task and staff sit watching hourglasses while their work does not get done. The obvious answer is more RAM, bigger screens, a hardware refresh and yet this answer is almost often wrong or just cosmetics.

I know this pressure because I felt it the moment I walked through the door. What I learned, over the next few weeks, was that resisting that pressure and focussing instead on removing one pain point after another is key.

What Broken Actually Looked Like

The IT landscape I inherited had accumulated its problems the way most do. Not through any single bad decision, but through years of reasonable-seeming choices that had quietly compounded into something genuinely dysfunctional or at the very least very slow. It seems a general rule that most modern IT systems over time become legacy and slow.

The company had exactly that problem because it ran Novell NetWare as its file and print infrastructure. To function across multiple sites, Novell required a specific type of bridge router to propagate its own network protocol — IPX/SPX — so that servers at different locations could see each other. The routers in place were Ericsson NILE units, common in Novell environments at the time, and they performed that job adequately.

The problem was everything else they did to the network in the process. And what was a good technology stack a few years earlier now prevented the new modern architecture of client server applications to work well. The Novell Netware optimized network was poison for the new TCP/IP based client server enterprise architecture.

The problem was that these routers propagated NetBEUI broadcasts excessively across the WAN. NetBEUI was the protocol Windows used for network communication and it was by design extremely chatty. This suited small local networks but was catastrophic on anything larger because it created so called broadcast storms on the network. Those broadcast storms were consuming bandwidth and more critically drowning out the IP traffic that the UNIX systems depended on for their connections. From the UNIX side the network appeared intermittently unstable. Connections dropped without explanation, applications hung and sessions timed out. The old-timer sysadmins claimed that the new UNIX systems were the problem because the Novell server had no problems. But they were exactly wrong.

What is interesting is how the users were reacting. The accountants had adapted. Several of them kept novels beside their keyboards to fill the time spent watching the hourglass (we were joking that Novell made people read novels). It was not a joke — it was a coping mechanism. Other people started an application or job they knew would take some time, and then simply leave their desk and did some filing or took a break. The entire organisation was slowed down and switched to do lower priority tasks out of boredom when waiting for the slow computers.

Meanwhile, the company had perfectly decent leased lines between its UK sites: some were 2 mbit/s, which was generous for the era, and even a 96 kilobit frame relay link to Munich that provided direct IP connectivity to the European parent. The infrastructure was not underfunded. It was being strangled by the legacy architecture sitting on top of it.

There was also, somewhere in the network, a Novell server whose physical location nobody could identify with certainty. When we eventually found it, it was sitting on the factory floor next to the loading bay, housed in a wooden box, cooled by opening the doors when a forklift warning light started flashing, and home to a small but thriving ecosystem of mice, spiders, and frogs. It was, by any reasonable measure, the worst possible environment for a server. It was also, somehow, the most reliable machine in the building. But I digress! This server deserves its own separate story.

The Diagnosis

My first week was spent not fixing things, but watching them, tracing connections, analysing and documenting network protocols and trying to understand what was actually talking to what, and why the results were so unpredictable.

What I found was a system in gridlock. Every problem had generated a workaround. Every workaround had created a dependency. Every dependency meant that fixing one thing risked breaking something else that had quietly come to rely on the thing you were fixing. The whole environment was locked in place by its own accumulated complexity. The network held everything together but was also causing everything to continually break. And no one dared touching the network for fear to make it worse.

The root cause — a term we might use casually today but which represented a genuine discipline of thinking back then — was not the slow PCs. It was not the unreliable email. It was the Novell infrastructure and the network architecture built around it, which was poisoning everything else.

The visible problems were symptoms. The invisible infrastructure was the disease.

The Conversation With My Boss

At the end of my first week I called my boss in Munich and told him what I had found and what I thought needed to happen.

The strategy I proposed was not subtle: move aggressively, replace the bridge routers with proper IP-only routers, and migrate from Novell NetWare to Windows NT. And do it as a deliberate and prioritised campaign. Eliminate the main bottlenecking systems and allow the new Enterprise system to breathe.

There was a catch. This work was largely invisible — it happened in the server rooms and network cabinets, not on anyone's desk — and it would require planned downtime. I would be spending my initial goodwill before I had earned any, by causing all this extra downtime. I risked people experiencing things get worse instead of better, at least initially. How much patience would they have?

My boss was characteristically direct about this. He told me I was proposing to burn capital I had not yet accumulated. But he also understood immediately why it was the right call. He had done this before, four times across Europe, and he recognised the diagnosis. He would prepare the board. He would identify the quieter periods in the business calendar to schedule the most disruptive work. He would have my back.

I offered to do the major interventions at weekends to minimise operational impact. He was, he said, entirely unsurprised that an IT person would sacrifice their weekends for the sake of the infrastructure, and he was entirely supportive. The time would come back to me, either as additional leave or as additional pay.

We had a deal.

Fixing the Foundation

The Novell-to-Windows-NT migration was not glamorous work. It happened in stages, mostly out of hours, mostly without anyone in the business noticing — which was exactly the point. Server by server, site by site, the IPX/SPX dependency was removed and replaced with clean IP routing. The Ericsson bridge routers went. Cisco equipment came in — the new generation of IP routing hardware that my boss had already been considering, and which turned out to be exactly what the environment needed.

As the network noise cleared, something interesting happened. The UNIX systems stopped dropping connections. The latency that had been making applications sluggish began to resolve itself. The accountants' novels became less necessary.

The leased lines that had always been adequate started to feel adequate again, because the traffic they were carrying was finally the traffic the business actually needed rather than a flood of protocol broadcasts competing for the same space.

Eliminating Novell also released budget. Support contracts and licence fees that had been consuming resources quietly for years were cancelled. That money became available for the next phase.

What Came Next

Once the backbone was stable, the desktops came into focus.

The PCs across the organisation were, to my genuine surprise, in better shape than expected — Pentium-class machines that were perfectly capable of running Windows NT Workstation. They were not slow because they were old. They were slow because they were misconfigured and starved of RAM.

I proposed to my boss that we max out the memory in every machine rather than replace the PCs. I could install the modules myself, so there was no need for expensive field engineer visits and delays. The cost was a fraction of a hardware refresh and the performance improvement was immediate.

With the RAM upgraded and Windows NT Workstation installed, the ERP graphical tools that the business had been unable to use properly — because they required a reliable Windows environment that had never existed — suddenly started working. The speed and stability that people had assumed required new hardware had been available all along, sitting underneath the infrastructure problems that had been suppressing it.

The backbone was clean. The desktops were capable. And now business applications started to earn their keep. The IT-obstacles were falling one by one like dominos.

The next obstacle was the email system — a CCMail installation so unreliable that staff had resorted to printing screenshots of digital messages and faxing them to colleagues in the same building.

That is the next story.

← Back to all articles