Approximate Date: 1998-1999 Computer: Web Servers
Tools & Technologies that didn’t exist: REST APIs
Historical remarks: The web standards debate was still raging with the two biggest players being Netscape Navigator and Microsoft Internet Explorer. Java was still very much a toy that was used to do fancy browser things on the client side but IBM had literally come to the offices trying to give away their latest Java Websphere server software. When I pressed them they admitted that their version of Java wasn’t quite “Exactly the same” as the version of Java from Sun who owned Java at that time. The world was very much caught up in deploying intranet websites and services. XML and SOAP had just been “Rediscovered” for IT use and were being touted as the silver bullet for everything. VB 6 and SQL were still the standard desktop tools while ASP (NOT.net) and Lotus Notes/Domino were the web tools of the day. Finally, although some transactional web processing existed in ASP (not .net), its critical to point out at this time that REST APIs or RESTful services didn’t exist at all. In fact, the term wasn’t even introduced until April 2000
The Web Class
The Management Board Secretariat had been in talks with KPMG to run an employee satisfaction survey for the Ontario Provincial Services employees. They had found paper based solutions to have the lowest success rates and wanted to look at an internet based solution. Since I was their more advanced web developer, they came to me with the request.
I sat with the KPMG staff and explored their requirements. They needed a tool to ask an unknown number of question. The questions could be of all types, freeform answer, check boxes, and radio buttons, anything. The size of the text for each question was also unknown. What was known was that the answers to the surveys couldn’t be traceable back to the person who completed the survey even through the use of server logs. The survey had to work on all browsers and since this was across all of Ontario and different ministries, all of the different mail products with highly variable feature support had to be accounted for.
The Ministry was using Lotus Notes/Domino. Domino had a simple designer front end meant for use by non-developers and this is what the government usually relied on for its web pages. Although some dynamic capabilities could be implemented this way you usually needed tricks and stunts to pull it off and there was always the need to define a limit. When the other consultants in the department heard what I had to do, they all insisted the best approach would be to build a huge 100 question form with tricks to vary the question type. It would have been massive, unwieldy and would still be traceable. No this was too complicated. There had to be a simpler, more elegant way.
As I considered the problem, I realized that as unique as the Domino product was it was still functioning on a common medium, the internet. In order to be successful on the internet it had to conform to the same data structures and rules and every other server and browser out there. So I began searching for how Domino interfaced with the rest of the world. It wasn’t long before I discovered the CGI variables from the internet that Domino was using and how to access them. Although Domino had a simple end user designer, Lotus had also licensed Visual Basic 5 as a core language within the tool which was so well connected internally that you could not only do anything you wanted with Domino from your VB5 code, you could even run VB5 code to do normal VB5 things like access SQL servers and build dialog boxes. In this case the integration to the web was so perfectly seamless that the CGI variables were already in the VB5 language within Domino ready to retrieve and do what I wanted with. With this I could now control all communication between the internet and Domino.
I began building a VB5 class which I imaginatively named “Web Class”. Through it I would capture all incoming data. The focus of the class was to differentiate between URL or FORM based communications (Get/Post) and build variable/value pair lists to feed back to the calling code for use. This was the key to reading any response from the internet without having a predesigned form to start with.
On the output side, the class focused on overcoming some common string handling bugs like the 2K limit on strings and it also focused on formatting page content with the appropriate HTML tags for a standard page and form headers. This also meant the class could choose to go beyond HTML in representing data and offer other presentations in MS Word or Excel through Mime type controls in the page headers.
I now had all of the pieces I needed to control both the creation of my web survey forms and manage the responses. Now I just needed to find a way to build the survey form itself.
Now that all of the end user simplifications had been bypassed in Domino, the answer was pretty simple. I Designed a Domino “Question” form that would allow a single question to be defined. It’s type, its placement in the list of questions and all of the details of the single question itself. Now all the KPMG staff had to do was create a separate record for each question and the code could assemble the questions into a single form for completion on the web and to obfuscate the responses while keeping them statistically viable, record the answers back to the database as a single isolated answer record for each question on the form. I called the effect, throwing a tomato (The internet form responses) at a screen door. This gave us limitless real-time flexibility. KPMG was impressed!
Now the environment was still a nasty mix of mail packages. Lotus Notes, Outlook, Eudora, GroupWise and CC:Mail to name a few. This was critical because to ensure employee access to the survey while eliminating the possibility of duplicate participation, I had to engineer dynamic URLs that were unique to each user. The URL was uninterpretable by people but upon seeing the URL the survey engine would validate access to the survey and construct the form for completion by the visitor. Once the survey was successfully submitted, the user would be marked complete and their access to the survey terminated. The ability of the mail package to deal with something as simple as an URL had to be determined.
To establish URL functionality I used the employee mail list to build what I called “The Probe”.
The idea was simple. I fire an email containing an URL back to the probe engine. If a user clicked the URL in the email the engine would capture which browser they were using and mark their mail installation as compatible with URLs. Once it was ready to go, I met with managers from every ministry in the OPS to advise them of what I was doing and coordinate my efforts. When everyone was ready I launched the probe and within a couple days I had confirmed over 3000 employees and had identified a mail implementation that was not supporting URLs. What’s more, I could relaunch the probe at the outstanding employees who were unable to use the link as needed by their mail administrators until all issues were resolved.
Within a couple weeks of the probe, the survey launched to over 3500 employees and was a complete success. KPMG touted the results as the highest return they had ever seen on an employee survey before and were very interested in my survey engine. Since the government loves its internal surveys, I modified the survey engine into a fully automated and user managed survey toolkit for use internally. It met with massive success and completely eliminated the need for consultants to spend weeks building survey forms, a fact that garnered me considerable distain from ministry contractors. The core class was so successful that one of the contractors stole the code for use at his other customer sites, a fact he proudly informed me of as he left the government.
This was the first ever use of REST web API on a Lotus Notes Domino server. In fact, ASP was the only other tool on the web at the time that could be accused of even trying to do REST APIs. Not even Java had been developed to that point yet. Not even the term “REST” had been invented yet! I used the tool for years after to solve many Lotus Notes/Domino problems requiring a stateless, transactional approach with Restful API.
Later I repackaged the class into a product for resale to the Lotus Notes development community. I called it Goodwin ePages since it offered developers the way to build fully transactional REST solutions in Lotus Domino and I developed a full manual and sample databases to get adopters started. The internet link to the downloadable sample was attacked on IBM developer forums as being totally impossible and thus amounting to nothing more than a phishing attack.
It was sad that I couldn’t find a wider audience for the solution but the fact that the Domino developer community attacked it so widely as being impossible was a testament to how advanced the technology I developed in 1998 – 1999 was.
Download the Goodwin ePages manual here.
A word about Lotus Notes/Domino
Let’s call this a public service message to the business community.
Anyone who has read though my blog posts knows my programming expertise extends from the lowest levels of the machine to the highest Object Oriented architectures and my data expertise has equivalent coverage. So my opinion on the question of languages and development tools should have some relevance especially where more simplistic tools are concerned.
I’m a power programmer, hard core, I’ve always been. So when I saw Lotus Notes/Domino for the first time in 1993 I thought it was a ridiculous joke, but being Lotus Canada’s Product Area Expert on LotusScript I had to get to know it. What I can say today (2019) is that the loss of this product from the IT landscape has probably cost industry well into the billions of dollars. Here’s why..
Prior to Lotus Notes popularity, office staff used spreadsheets to track all kinds of things that had nothing to do with accounting. Project plans, meeting agendas, an unimaginable and wildly varied array of tasks recorded in spreadsheets. This was done because they lacked a more suitable tool for the job. And the resulting mess from a data management, sharing and organization standpoint is an absolute travesty! I’ve literally seen large banks with shared drives loaded with THOUSANDS of spreadsheet files used to record data. Before you talk about Lotus Notes not being relational, Excel isn’t either and a drive full of spreadsheet files is by no means organized or usable from a mass analytics perspective.
There is a huge technology gap between the common spreadsheet and a coded solution that Lotus Notes used to fill. This is where your staff are trapped! Most office tasks are far too small to justify a programmer and a database developer and the months of planning and construction required to automate or organize the work. The Lotus Notes tool was developed specifically for the average office person who just wants to store and organize things. A quick form complete with data backend could be thrown together in literally minutes with no development skill what so ever, evidenced by the massive consultancy boom that was associated with the tool in the late 90’s. A view of all records could be created with equal ease and everything was designed to be WYSIWYG. One of the biggest joys for me was watching average employees building a little database for themselves to help them do their work and keep themselves organized. They would refine their little database over time into amazing works of automation. It was empowering like nothing else.
Today Lotus Notes is dead due primarily to narrow technical redirect leaving users stuck with a choice between a coded database solution or a spreadsheet. Users are forced to make do with spreadsheets.