Happy New Year.
The clip of last year’s festivities at Times Square is not available as Times Square only had rights for a live stream.
Three cheers for copyright protection.
Stress Test 2
Agneta and I drove down from the Granta at the weekend, likely on Saturday. We were heading south. The weather was promising.
I was to teach NT systems programming for a week at IBM's venerable think tank in Hursley. The contractor set up our accommodations. We arrived mid-afternoon.
Pop Goes The Weasel
We wanted a good restaurant for the first night, and the lady in the hotel reception gave us directions. Later that evening, she asked if we'd been in our room singing 'Pop Goes The Weasel'. (Yes.)
Agneta didn't know the song and I knew none of the lyrics, only that it was sometimes used in Jack-In-The-Box toys.
Someone else with the hotel told us the supposed historical roots of the song in the outbreak of Plague in Edinburgh. I told the lady we definitely had not been singing 'Pop Goes The Weasel', that we rarely sang anything together except a Dylan song for a laugh when walking to Sainsbury's back home.
I asked her why she asked. She told us that a hotel guest checked out when we were out dining, complaining that a couple in another room - and she evidently suggested it could have been us, although we'd not yet been in our room or met anyone at the hotel - had been singing 'Pop Goes The Weasel'. Oh well.
We chatted with the bartender at closing time. He stuck around for a while to pour a few nightcaps and to fill our minds with tales of Monty and Ike preparing for D-Day in the area, planes landing and taking off all the time.
'Planes?' asked Agneta. 'Where's the airport?'
'They used the fields', the bartender told her.
This was a course in systems programming. Microsoft were expected to release Windows 2000 real soon. Through our own personal contacts with Redmond - we got the nightly builds from their local head office sent over from Redmond - we'd learned that Dave was essentially gone and in fact had been gone for years, with the periphery of his team in a state of panic, imploring 'Dave is gone, do you realise, Dave is gone, whatever are we to do?'
What they did of course was rewrite the entire NT kernel (bloating it by a factor of 400%, from 16M LOCs to 64M). Evidently DEC caught onto them and demanded it, after receiving a settlement from Mister Bill of several hundred million. (Dave had taken the code from his Seattle lab.)
IBM had gone demonstrably into the Linux camp, but they were evidently still on board with NT to some extent: I was sent to teach NT internals.
I brought along my MSDN discs with the latest 2000 beta along with my modded WINFILE.
https://github.com/Microsoft/winfile
https://github.com/microsoft/winfile/blob/master/src/winfile.c
WINFILE OOTB was unusable. A little tasteful colour added to the toolbar glyphs was nice, but most importantly the app had to be geared for professional use, meaning a shitload of settings had to be tweaked and then saved to the Registry, then exported so they could be imported to other boxes.
On MS-DOS it's called DIR. On a Convergent system it was called SFiles. Yes really. And oh so lame too. Seeing listings scroll past was no happy way to work. On all systems, after first making your own custom text editor, you have to create your own file manager. That's just a fact and it's a sine qua non.
On Unix, they say that if you fully understand the command 'ls' then you understand Unix. I actually say that, and others do as well. It's true.
You have to understand a file system to understand your operating system. There’s no alternative, to quote Roger Moore in The Spy Who Loved Me. The file system is the essence of your computer itself. To see your file system, to have complete control over your file system: that's absolutely essential.
IBM's vending machines in Hursley: You can't put coins in them. You have to buy USB thumbs and put the thumbs in them. (That works out nicely for Big Blue.)
The Hursley estate is expansive, rolling hills, impressive. I had four students, two were blah-hah, two were either exceptional or on the way to being exceptional. The two good ones were a Scot and a dude from Africa. The African didn't yet have the skills of the Scot but you could see he'd get there.
I brought along my WINFILE setup and the 2000 CDs. We put 2000 on an extra box. Interest was minimal. I never got to fully know what they were up to, but learning about NT seemed a bit like a shell game because everyone was otherwise using Linux, Linux, Linux. Which is hard to fault, really, after the Baskin-Robbins flavours came out, and studiously avoiding the microkernel issue, the UI issue, and the fact that FreeBSD was already there. But the four from IBM swore by Linux, with the Scot affirming over and over that it was 'bulletproof'. And he'd had experience working with Dave's VMS, so he understood 'bulletproof'.
‘Bang, it’s there!’
WINFILE wasn't working right on this latest 2000 beta. Oh sure, you could spend endless minutes configuring the whole thing, but those settings wouldn't stick in non-volatile. This of course prompted the project to create Xfile for Windows ('X-file') which took one year ten months.
Handling removables was dicey. You obviously couldn't check them on every refresh, so you had to develop a way to get around that. And the mucky design of Microsoft's tree view control meant that the application had to be built a bit like the SAS Viggen: you sort of assume a state of data instability so corrections are going to be necessary further down the line.
Reuters in London
New Zealander John Townsend contacted us from Reuters in London. John had the unenviable task - a half-year gig, he said - of dealing with all the Y2K mail. There were tens of thousands, hundreds of thousands, perhaps millions of messages in the Reuters inbox, and he had to sort through them. To make matters worse, he had to connect to the mail server over an extremely slow Ethernet pipe with 1/10 normal throughput. He was given Microsoft Windows Explorer for file management and that wasn't doing it for him. Listing a single directory would take many minutes, often long enough for him to go off for coffee and a refill. So somehow he found us, tried our Xfile, and things were better.
But he also noticed a curious thing after the fact. Because his mail server pipe was so slow, he could actually see, in realtime, how an Xfile refresh was actually not one but two refreshes. We could in no way see it, as our own connection was at least ten times as fast. So back to the garage.
We'd had to sprinkle our code with diagnostics that could be turned off on builds. If we turned them on, we could see exactly where we were in the code, and we could trace control flow. (The MS debugger could not be used as tree view interactions screwed up everything.)
'You are exiting function foo.'
'You are entering function bar.'
So I was able to see how the refresh code looped back on itself. Getting rid of that meant Xfile for Windows was now twice again as fast. Hopefully John and Reuters were pleased.
Xfile for Windows was necessary because WINFILE no longer worked on 2000, and WINFILE was necessary because it was the only way to actually see what you had on disk.
The Windows Explorer Shell Namespace method is pure shit. Conceive of a hierarchy in the left pane where the topmost node is 'Desktop' but about a half mile down that pane, deep in the hierarchy, you discover the same 'Desktop' all over again? And what is real and what is fake? Control Panel is NOT a folder. But Microsoft now wanted to make it appear to be a folder. WTF is that?
What is real and what is fake? You can't know on Windows.
Our friend John Lettice at El Reg noticed a curious thing with Windows. Namely if you reassigned app connections under HKEY_CLASSES_ROOT away from default Microsoft apps, the affected files never turn up in Microsoft searches.
That was very tacky.
All we want to do is see what we have on disk. That's all.
How else can anyone manage a system?
Second Upload, New, Improved
The above clip is a second upload of the 'Stress Test'. It's made because Google have a nasty habit of cluttering screens so you don't see the final frames properly. The first 1:35 or so is still excruciatingly boring, and the full clip is truncated, as no one wants to watch an enumeration of 235,000 directories. But the speed is roughly the same as seen in our other clip 'A Walk On The Wild Side'.
Apple Not The Answer
For a while - for the longest time, we are stubborn - we opted for OS X for several extraordinary reasons.
Migration away from Windows is necessary for the web health of the planet. Windows is not secure and wasn't even designed with security in mind. (Dave Cutler's VMS/NT is another matter, but that OS does not make Windows secure.)
Getting Joe Blogs to change his habits is key. FOSS is all over that place, factionalised. Nobody's going to get behind it. People who only think about themselves can opt for this Linux or that FreeBSD, but that's not important. The web health of the entire planet is what's important. So you have to fornicate with the devil. You need a major corporate player who has the marketing clout to unseat filthy dirty Microsoft.
Apple inherited, through Steve Jobs, an unbelievable bag of goodies. Steve Jobs owns the programming language Objective-C. The origins of that ownership and how it got owned in that way are still clouded in mystery, but Steve Jobs bought the language, right before making the trek back from Redwood City. Objective-C is the only production language to this day that correctly addresses the paradigms of programming with a graphical user interface. Objective-C is also the world's only truly authentic object-oriented language. Alan Kay invented the term 'object orientation', and he doesn't have kind things to say about the pretenders, and it's easy to understand and agree with his reasoning. Objective-C is simply a high-power production model of his Smalltalk which was an experimental laboratory language for use at Xerox. As it's interpretive, it's not suitable for production. Objective-C is.
NeXT hardware. It runs rings around anything seen from Microsoft or Apple. Colours? Microsoft offered 14. Apple offered 2. NeXT offered thousands and potentially millions. The software-hardware interface is scalable on NeXT systems, using floating point rather than the integers seen on Microsoft or Apple systems.
That's basically it. But against this list one must factor in the 'Mac' mentality, the 'Mac' community, and the 'Mac Derangement Syndrome' prevalent within Apple, present to this day. Apple's so-called 'engineers' never got it and still don't get it.
We offered to build a hybrid Linux built on GNUstep and FreeBSD, but no VC was ready to finance it. Shuttleworth and his friends think an OS is better simply because its icons look better. Sorry, Mark.
All of which inevitably returns to the same eternally eternal question: why even programmers - at least the more common and less gifted ones - don't seem to give a flying F about what they have going on in their own systems?
We've been good friends with highly regarded IMS programmers who follow the recipe without having a clue what they're doing. We have seen shoddy work like you wouldn't believe. We're going to estimate that fewer than one in one thousand programmers actually know what the fuck they're doing. And we're going to undersign Steve Jobs' own assessment, time and again, that the 'above average' programmer is not two or three times better than the average but two to three hundred times better.
This is a crisis of intelligence. Statistics show that people in the West are getting increasingly dumber by the year. George Carlin warned of it. I can imagine, for example, that BWK might not have the tidiest file system ever, but I couldn't imagine that of DMR, and looking at the file system of 'ken' would likely be daunting, as he'd have a level of organisation that totally boggles the mind. But Joe Punchkey who writes code for HP? Or Microsoft? Or Apple? No. Microsoft's former Hungarian HR manager summed it up perfectly. He was never looking for geniuses. They create trouble. He looked for 'average stupid' - just enough smarts to find the power button and to know they have to follow orders. Carlin probably talked about this, and it's true.
And so if your technical giants don't seek intelligence then it's not going to be reflected in your target demographic either. Make complex products for a market you're trying to dumb down? What do you expect?
NeXT wasn't a company like that. Bell Labs wasn't an organisation like that. Commercial giants care not for the future but for the false god of today's bottom line.
Calculations
Expanding that OS X system from root takes just under one minute forty seconds, or one hundred seconds. This makes calculations easy.
The command for counting directories is:
sudo find / -type d | wc
The results:
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
238905 323053 36172919
The command took about forty seconds to complete.
Driving Instructions (From North/East Via M3)
Leave M3 at Junction 12, signposted A335 Eastleigh (N).
At roundabout take first exit signposted to Chandlers Ford and Otterbourne.
Proceed for 200 metres and at mini-roundabout turn right down Otterbourne Hill towards Otterbourne and follow this road northward for just over a mile.
At the mini-roundabout just after The Old Forge pub turn left down Poles Lane signposted Hursley.
Follow Poles Lane for 2.5 miles and at the junction with the A3090 signposted Winchester (part time traffic lights) proceed straight over into *unmarked entrance* to Hursley Park (note tree with stone circular seat).
Continue on site perimeter road for *1 mile* to Hursley House and follow signs for visitors parking.
Our recollection was that it was actually difficult to find our way, despite our hotel being nearby. The lady in the hotel reception told us to keep a lookout for an unpretentious sign on the left leading to an innocent woodland drive for a mile after which things would open up and we’d appreciate the vastness of the estate.
We never approached what they seem to call Hursley House. And I have no recollection what Agneta did all day long when I was on the Hursley estate teaching. Presumably she went out on her own, as the 'city' itself was in walking distance of our hotel.
Again? Aw Come On!
The last thing you want to do when finally migrating away from Windows is to build another blasted file manager.
We'd seen Tom Greene and colleagues of Mark Ward make the move. Greene converted his entire household to Unix. We conceived of the Red Hat Diaries project, as we were in close physical proximity to Red Hat's HQ in the Research Triangle Park (RTP).
After sending out queries on our mailing list for appropriate Wintel hardware and coming up with zilch, we were about ready to give up. One memorable visit to a Gateway Country just about sealed the deal - against going that route. We wanted notebooks, but the notebooks we'd seen were overpriced trash. A friend advised us to 'look into the seams' when inspecting the hardware. Worst of all was that almost all boxes were downright ugly.
Coincidence and further research led us to Apple. We’d heard ugly rumours about Gateway return rates, found two consumer feedback sites, finally looked to see who rated highly, and found IBM and Apple at the top of the lists.
We knew nothing about Apple, only that under the bonnet the traditional Apple boxes were a mess. But this OS X - this was something else. And we knew about NeXT. They'd pursued me in Stockholm, keen to set up a permanent display in our HQ. Likewise, a demonstration of their software creation capabilities at the Stockholm Unix Expo was convincing, to say the least.
Agneta was all over the Sun exhibit. They had lots of free stuff in Sun purple, and Agneta had this thing for purple. But I'd found the discrete NeXT booth with my contact Andreas standing there all alone.
Andreas had already gifted me with a pilot's case full of NeXT paraphernalia: books, the complete x86 setup, NeXT The Movie on VHS cassette, and more.
I introduced Andreas and Agneta and then asked Andreas if he could demonstrate how the NeXT boxes worked. 'Let's build a phone book application for her, Andreas!'
'But I don't need a phone book', Agneta protested.
'But this is only a demonstration', I told her.
So Andreas fired up what must have been Interface Builder and started laying stuff out. First name, family name, phone number, address, etc etc etc.
'Oh, as an afterthought, let's put a controller on the phone number field so it only accepts numerical input', said Andreas, who dragged something from another toolbox window and dropped it on the entry field. Said and done.
And he did all this without at first using the keyboard a single time. He just dragged shit from toolboxes in other windows, held down a shift and expanded a single entry field into a matrix, and so forth.
It took him all of one or two minutes. At the very most. 'OK, it's ready, try it!' he said.
'Try it, Agneta', I told Agneta.
'What should I do?' she asked.
'Put in a name.'
'What name?'
'Our daughter's. Put her in.'
So Agneta typed away. First name, family name, phone number.
'Now save', said Andreas. Agneta clicked a 'Save' button. I told her to name the file 'Black Book'. Then we called the file back up again.
'OK, so I'm supposed to be impressed', Agneta said. 'But why?'
'You remember the phone book app I wrote for work the other week? Remember how long that took? That was for Windows. Remember how long?'
'You said you wrote it in a day.'
'Pretty much, yes - I pulled an all-nighter. Twenty-three hours straight.'
'Oh.'
'Twenty-three hours. For Windows. But here... How long did it take here?'
That's another thing about NeXT. They have SOS Interface aka Interface Builder at NeXT. No one else has anything like it.
Twenty-three hours versus five minutes. Photo finish.
So trying out OS X seemed a good way to waste time. We found a dealership in Raleigh and their 'showroom', big enough for four laptops, looked like a scene from Joe Versus The Volcano where Joe looks for luggage. They had the TiBook and a white magnesium iBook. I ordered a TiBook. Sydney wanted an iBook but wanted a bit of time to think about it.
No one had looked at the system's file management. (Who could have suspected?)
As a good friend who's big in network consulting in the US told us:
'Yes the hardware can be dazzling. People see all these enormous possibilities. But then they ask to see the system file manager and the rep pulls up Finder.
‘Then they walk away.'
It's only gotten worse over the years. Betas for Apple's latest iteration had a file manager so destructively buggy that the thing would crash all the time, and what's worse, its crashes would wreak such havoc that only a cold reboot could save the day.
All that nonsense for a file manager. But we knew nothing about that at the time. After years of intensive programming for Windows, teaching interface programming for Windows. teaching NT internals, and spending twenty-two months building a decent file manager for Windows, I'd sort of reckoned with taking it easy for a while.
Not to be.