Iris: Iris Today & Iris Cafe, Iris 2001.

YEAR: 2001
Login Login
User: Anonymous

LABEL: ASP - Application Service Providing | Lotus Domino
PEOPLE: White, Brian
THINGS: ASS | Notes | RNext
TIME: 2001
RNext & ASP

This month's Iris Interview is with Brian White of Lotus and Peter Mierswa of Iris. They are here to talk about the Application Service Provider (ASP) project for Domino Rnext (Rnext is the code name for the next major release of Notes and Domino).

Also, during the month of February 2001, Brian and Peter will also be taking your questions about the ASP project in the
Iris Cafe Developer Spotlight Forum here on

Brian White

Brian is the product manager for the Lotus ASP Solution Pack and, as such, he is responsible for integrating application service provider features into the core Domino server. Since joining Lotus in December 1996, Brian has been involved with helping service providers use Domino and Lotus technology in their offerings to end customers. Before joining Lotus, Brian worked as the Manager of the IP Network Design group within the IBM Global Network. In his "spare" time, Brian is pursuing a High-Tech MBA degree from Northeastern University.

Let's begin with an explanation of what an Application Server Provider (ASP) is. Would you define that term for us?

Today, the term ASP is probably overused, but it is primarily used to refer to a service provider that hosts applications for their customers at the ASP's data center and provides access to those applications via a network. The hosting services can range from an outsourcing model to that of a shared environment with hosted applications. In an outsourcing model, the ASP dedicates hardware and resources to each individual customer. In a shared common environment, the ASP hosts applications for many customers, each in their own secure, private environment. A shared environment reduces the total cost of ownership for the ASP and helps the ASP offer profitable services.

Brian White

What is the Domino ASP project and how does it relate to how Domino is used by ASPs today?

At Lotus, there is actually a lot of different activity going on with ASPs. Today, we are actively involved in the ASP market and we have been for some time. In the early days, we worked with service providers in our Network Notes program to provide basic Notes and Domino outsourcing, where Lotus was dedicating hardware and resources for each individual customer. At that time, we developed clustering, partitioning, and billing, all of which helped service providers offer dedicated hosted solutions for customers.

Recently, we have placed more focus on the hosted applications segment of the ASP space to reach a new set of customers, the small-to-medium businesses (SMBs). In early 1998, Lotus released Domino Instant! Host, which provided a hosting infrastructure for single-point applications. This past August, Lotus released its second generation of hosting platforms with the
Lotus ASP Solution Pack, which has the underlying Lotus Hosting Management System (LHMS). With this release of LHMS, a service provider can offer a suite of applications across Domino, QuickPlace, Sametime, and the IBM WebSphere servers.

With the Domino ASP project for Rnext, we are enhancing Domino itself, by taking some of the features we developed in LHMS and addressing them in the core Domino server. Our ultimate goal is to resolve core features directly within Domino and to use LHMS for the add-on functions to help with application management, user management and licensing across the wider set of application servers including Domino, QuickPlace, Sametime, WebSphere and others.

Companies have been outsourcing their computing services for years. How is the hosted application business different from this older, traditional outsourcing?

Using the traditional outsourcing model, a service provider would eventually take over some or all of the operations that had been done by the enterprise IT staff. This would include any applications that the enterprise developed on Domino; it could even include supporting the Notes clients. The service provider would dedicate Domino servers for each customer, running them at the service provider's data center or managing them remotely at the end customer offices. The end customer is essentially giving the service provider the responsibility of running the Domino infrastructure. Today, outsourcing is typically limited to the high-end enterprises due to the costs associated with total outsourcing.

In the hosted application model, the ASP selects a set of solutions that they offer to potential customers. This model is about applications, or solutions, which the potential customers need; it is not about the infrastructure. The hosted applications business is geared to allow the small office, even the home office, to get a hosted application up and running quickly, instead of being forced to support the application in-house. The ASP service model is helping customers by reducing their costs, eliminating the need to hire the IT staff, and enabling them to quickly deploy applications that would have taken months to deploy in-house.

Will the ASP develop these common applications or will this be an opportunity for existing ISVs (Independent Software Vendors) to sell their applications in a different way?

ISVs will definitely be involved in developing applications that will allow them to pursue a new market. The ISV will be able to place more focus on application development and less on actual sales to end customers by leveraging the ASP in their sales channels to small and medium businesses. Some of the ASPs will totally rely on the development skills of the ISVs, while other ASPs do have development skills themselves. There will be a mix of different business models where ASPs and ISVs partner at different levels to offer hosted applications.

A vertical market opportunity also exists. One of the problems with the ASP business today is that many ASPs are simply trying to resell applications on the Internet. They are not differentiating their services, so the opportunities they project are cost of ownership and price. Over time, I don't think this model will succeed. The ASPs that pursue vertical markets and offer a set of solutions for a vertical industry such as the legal or medical industry are really going to be able to differentiate themselves. It is frequently the new ASPs that pursue the vertical markets, and in many cases the new ASP is an existing ISV that has expertise in a particular vertical industry. So an ISV who is developing applications for health care, for example, might become an ASP who pursues the vertical market of health care with their own offerings as well as additional health care solutions by other ISVs.

What is the main product goal for the ASP project? What are the key areas of interest?

The main product goal of the ASP project is to gain new customers, primarily in the small and medium businesses. This is the end-customer base that is not yet tapped by Domino, or by our competitors. This is the area of growth. The enterprise market is largely saturated in the areas of messaging, Web servers, etcetera, but the smaller businesses are now looking for solutions. The ASP project is a way of offering solutions that will allow our technology to be hosted for use by a much wider customer set.

Key areas of interest for Lotus include not only leveraging Domino, but leveraging Domino with our other applications, such as Sametime, QuickPlace, Domino.Doc, K-station, and Lotus Discovery Server, as well as integrating solutions with IBM and IBM offerings. So, our technology can be used alongside IBM technology, as well as our competitors' technology to offer a suite of services to customers.

Why should a customer use an ASP business model? What are its advantages from our customers' perspective? Does the ASP business model open any new areas for them?

From Lotus's standpoint, there are three different customers to discuss. They are the end customers themselves—the hosted organization; the Independent Software Vendor (ISV) who is developing the applications, and the ASP who is hosting the services. From the perspective of the end customer, the benefits revolve around the total cost of ownership, the rapid deployment of applications, and the ability to have these applications run by experts that the end customer does not have to hire. From the perspective of the ISVs, the ASP business model has helped them and our business partners reach a broader and wider customer base. Today, many of our business partners don't have the sales force needed to reach all of the small and medium businesses, so this is a new channel for our business partners who are developing applications. They are able to offer their applications to a wider customer set and grow their customer base accordingly.

From an ASP perspective, we are expanding the markets that the ASP can access. When an ASP offers only an outsourcing model, they are typically limited to a high-end customer base. Our hosted applications business model allows the ASP to use a common infrastructure to support many hosted organizations with a broad set of applications, allowing them to go downstream at a lower cost of ownership to the SMB market. The shared infrastructure and delegation of administration functions provide the ASP a better return on investment when targeting this market.

What are the impacts of this type of business model on application developers and on administrators?

Application developers have to understand how to develop solutions for this business. Today, many of the Lotus business partners develop and complete 80% of an application, and then customize the last 20% on-site with the enterprise customer through a services engagement. The hosted application model is very different. The application must be 100% complete, have an intuitive interface with online help, and should generally provide some level of personalization within the application itself. The most important of these requirements is the intuitive interface with complete online help. We need to ensure that the end customers can navigate the application, and that they benefit from using the application without requiring hands-on resources or consulting services from the ISV or the ASP.

Application developers must also consider that the application will be utilized by a service provider in a shared environment. We will leverage all of the Domino security in our ASP offering, and enhance some of this Domino security to keep Customer A separate from Customer B, but still, there are some functions that will not be permitted in a particular application offered in the service model.

On an administration level, an ASP also considers how much of the provisioning can be automated and what administration tasks can be delegated to the hosted organization. Leveraging the shared infrastructure and reducing the total administration costs of the service provider are the two key areas that help improve the ASP's return on investment. Delegating administration tasks such as create, modify, or delete user is critical to reducing the overall administration costs of the ASP.

What advantages does a Domino ASP offering have over other service suppliers?

Domino is a very good technology for an ASP due to the application provisioning that we can offer. For example, by leveraging Domino templates (NTFs) and security models, we are able to offer a single application securely to Customer A and Customer B on the same shared infrastructure. With Domino, you can even run many different applications with many different customers securely on the same infrastructure. Additionally, by using LHMS with Domino Rnext, we can help the ASP create and provision services for each customer as they self-register.

Domino Off-Line Services (DOLS) provides Domino solutions to a user who is connected over the network or to a user who is offline. Both users are able to access their files, such as mail files, and are also able to access applications with a browser, either on the network, or locally. They can switch from online to offline, working locally, on a desktop, or on a laptop. DOLS allows a mobile workforce to use a hosted solution both while connected to the Web or disconnected from the Web, such as when traveling. This is a strong differentiator from other hosted solutions that require that an end user always be connected to the Web to gain access to their applications and files.

Domino has also proven to be adaptable to new technologies. Domino used to be a client/server enterprise model. Internet standards came out and we adopted those Internet standards and we fully support them within Domino. Wireless technology is now becoming very hot, so we have moved into the wireless space and Domino servers can now be used in a wireless environment. Domino does not pigeonhole an ASP to a single set of technologies; instead, Domino will adapt to new technologies as they come out, allowing Domino to exist side-by-side with a competitor's technology in an ASP server farm.

What are you supporting in terms of Internet protocols/clients initially, and in the future?

Since our main focus is working with the ASPs to target the SMB market, we are primarily focused on Web technologies that don't require a "fat" client. In our initial release of ASP, we plan to support Web services via HTTP; mail via HTTP, POP3, and IMAP; directory services via LDAP; and offline services via Domino Off-Line Services (DOLS). As we move forward, we see the integration of Domino in a wireless environment as very important. Today, wireless services are already being used widely within many European communities. While Domino Everyplace already offers this wireless integration, going forward, we need to look at the integration of wireless access in our ASP architecture.

What platforms will you support initially and in the future?

Initially, we will support the Sun Solaris 8, IBM AIX 4.3.3, and Windows 2000 platforms. In the future, we will look at additional platforms that fit the model of hosting many customers on a single box, such as the AS/400 platform, which already has a port of LHMS underway.

Many ASPs use Linux today for offering services using dedicated hardware for each customer. Until a recent version, there has been an issue with the thread support in Linux scaling up to the very large end systems we see the ASP shared services using. We will continue to look at Linux, especially as its scalability increases. I see a good fit for Linux as a protocol server in our future multi-tiered architecture.

How does the ASP project relate to other Lotus or IBM offerings and strategies? Can you give us any details about these?

The ASP project that we are talking about today is an evolution of what we have been doing here at Lotus. The Lotus ASP Solution Pack with LHMS is available today, using R5 technology. The ASP project in Rnext builds on that work, improving Domino as an application server in an ASP environment.

We also work very actively within the greater IBM Software Group to ensure that we have a single Lotus/IBM strategy around hosting tools for ASPs. The hosted applications market is about applications, so the wider the set of applications available to the ASP, the better the opportunity the ASP has.

Lotus has recently announced
Lotus Collaboration Services as a new software service initiative. While this may appear as directly competing with our ASP partner base, in reality the service is focused on the very large enterprise accounts that would not separately invest in a Sametime infrastructure for instant messaging and meeting center services. The first solutions require separate dedicated servers such as we see in the outsourcing model, but we will be working collectively to move Sametime into a shared infrastructure offering. Leveraging the Rnext ASP enhancements, particularly in the directory space, and the LHMS tools, will help with this effort and ultimately allow Lotus to bring hosting technology to its core products faster.

The ASP industry is still very young. How do you know that the services you are offering are exactly what is needed?

To be frank, we don't. We've been working with service providers on an outsourcing basis for quite a while and we understand their requirements. Many of the solutions and features that we are adding to Domino Rnext will help these customers with their outsourcing issues. We've also been working with hosted applications since 1997 with our initial investments in Instant!TeamRoom and Domino Instant! Host. So, as you can see, we've had some experience in this area. We were in this market before the term ASP was created.

We have also been working quite actively to gather input from customers, as well as from the ASP consortiums and conferences. We are working with ISVs and ASPs to understand their needs and to determine how we can work toward both our purposes and theirs. We believe that the features we are adding will greatly assist our customers, reduce the ownership costs of the ASP, and bring them to a profitable business much quicker. We will be actively working with them in the future to continue to improve and modify the ASP environment.

Peter Mierswa

Peter Mierswa is the project leader of the Networks/Clusters team and the Application Service Provider team. The Networks/Clusters team is responsible for all of the common network and cluster support services for both the client and the server. The Application Service Provider team is responsible for overseeing the modifications of all Domino services to be suitable in an ASP environment. Peter has been involved in network engineering for 30 years and has been with Iris for the past 3 years.

What were the key development goals at the start of this project?

The ASP project was started over a year ago. Our primary goal was, and still is, to provide low-cost services to multiple small and medium businesses (SMBs). We have determined that the most efficient way to do this is to use a shared server model, that is, to service multiple SMBs on a single partition on a single server. To use the shared server model, we found that we needed to improve security, server reliability, availability, and scalability. Our billing and reporting technology also needed enhancement. These improvements and enhancements became our key development goals. We then considered each of these goals with each of the key Internet Protocols we plan to support: HTTP, LDAP, POP3, IMAP and Domino Off-Line Services (DOLS).

Peter Mierswa

An ASP model makes assumptions that are different than those made by a traditional client/server approach in terms of requirements, such as the need for secure access by multiple organizations and the ability to handle a large number of users, to name just two. What assumptions did you make and what questions did you answer for this ASP model and what were their implications in terms of the development effort?

The basic requirement for an ASP customer is to keep costs down. The overriding principle of the architecture is to do this by placing multiple hosted organizations on a single server—a large number of hosted organizations and a large number of end users with common hardware, common administration, and common management. In order to do this, a fairly long list of architectural features has to be available in every part of Domino. First, there is security. It is essential that no hosted organization see the identity or data of another hosted organization. Each hosted organization thinks that the service they buy from the ASP is private and belongs to them.

We also looked at scaling. The ability to scale is very important because the more hosted organizations and users that you can fit on one machine, the lower the cost to the ASP and, ultimately, the lower the cost to the hosted organization.

Availability and reliability are two more features that we looked at in-depth. In an ASP environment with a single machine or set of machines hosting many organizations and many users at those organizations, a failure in one application or in one server component affects a great many people. Those failures have to be very few and far between. The administration costs have to be low as there are now very many users in different organizations on a single server. Administration must also be hierarchically distributed. There may be multiple administrators working for the ASP, creating the organizations and modifying the organizations' data. Creating and administering end user information needs to be distributed to the hosted organizations themselves.

We also looked at billing. An ASP must be able to collect sufficient billing information. The ASP has to determine how to bill their customers and then implement that billing strategy; therefore, the billing interface has to be easy-to-use and configurable for each ASP. The ASP also needs sufficient tools to do capacity planning because each ASP's business will continually grow. ASPs have to know how that growth is affecting availability on their machines, for example, when new machines are needed, when new memory is needed, and when new disks are needed.

Availability and development of applications are also key areas. The most important characteristic is the set of applications that the ASP offers to the hosted organizations. The ASP must be able to partner with ISVs to find existing applications, and to request that new applications be developed. When the applications are developed, they must be easily managed, installed for the organizations that buy them, and properly maintained.

How much of this does Domino offer today, and how much more attention is needed? Can you give specific examples?

Today, we sell Domino to a number of ASP organizations who offer a similar service by installing a separate partition for every hosted organization. They are doing well, but our model is one in which the same applications are sold to a large number of small and medium organizations. A single partition for each one of the many small and medium businesses is just too costly to manage and administer, and it doesn't perform well. All of the characteristics that I mentioned in answer to the previous question, that we are planning for our ASP service in Rnext, are available in Domino today. They just aren't quite evolved enough yet to please our ASP customers. We haven't actually introduced many completely new architectural concepts. Instead, we've improved each one of these areas, that are already very good in Domino, to be suitable for the serious ASP customer.

How did you go about planning the new features to support ASPs?

Over the last year and a half, we met many times with every major team at Iris, and worked with this set of ASP requirements: security must be achieved; scalability and availability must be improved; and low cost must be ensured. We spent many hours with each team discussing the characteristics of their portion of Domino and how those characteristics would work in an ASP environment. We met with development engineers and quality assurance engineers. We tried very hard to find everything that would be a problem in an ASP configuration, everything that we could improve in our product for an ASP configuration, and we negotiated each one of those tasks with each one of the teams. All of those teams at Iris took on some of the ASP tasks.

Are all the features that you identified at the beginning of the project planed for Rnext?

No, when we started the ASP project, we set very high goals. From the architectural discussions, we decided to create a farm of servers that could be increased almost trivially, and to provide scaling, reliability, and hot backups. That architecture is complete and the coding and tasks necessary to implement it have been planned but the work is too extensive to do within the Rnext schedule. The architecture for establishing a farm of independent protocol servers and database servers has been scheduled for a release subsequent to Rnext.

How will multiple hosted organizations connect to the same server and know that their data belongs only to them and is accessible only by them?

There are two distinct configurations that ASPs may use due to the nature of TCP/IP on the Internet today. The number of IP addresses available for assignment to companies is becoming very short. The standards bodies in the Internet community have developed a new standard for Internet Protocols. This new standard, IPv6, will alleviate the shortage of available addresses, but it is estimated that it will take many years to implement IPv6. In the meantime, the availability of IP addresses is short so let me describe the two different configurations in which an ASP may choose to run.

In one configuration, each hosted organization is assigned its own IP address and its own DNS [Domain Name Service] names for SMTP addresses, IMAP, POP3, and LDAP servers, and for Web server application sites. In this configuration, when a connection is received by a single server that is hosting multiple organizations, the server examines the IP address that it received as the target of the connection. The server uses that IP address to determine what organization the connector is a part of, and it uses that knowledge of a person's identity to constrain that person to their hosted organization's portion of the Domino Directory, and to constrain that person to their hosted organization's portion of the data directory. Each end user will only see people in their organization's portion of the Domino Directory and they will only see their organization's applications.

In the second configuration, a single IP address is assigned to the entire ASP, but different DNS names are assigned for SMTP domains, IMAP, POP3, and LDAP servers, and for Web sites. For each hosted organization, a similar model is used. When a connection occurs, the server doesn't know which hosted organization the end user is a part of. But, when the end user performs their first request, the server can determine the hosted organization. If the request is to a Web server, the URL is the identifier. If the request is to a POP3 or an IMAP server for example, the server uses the identification of the user. When that identification comes over the wire, the server again uses that information to determine which hosted organization the user is in and then performs the same virtualization of both the Domino Directory and the applications in the data directory.

What new security services were required to accomplish this?

Two mechanisms are used to secure the data that is available to end users on a server that is shared by multiple organizations.

One mechanism, the extended ACL (xACL), is used to secure databases that are shared across hosted organizations. The xACL mechanism was developed to satisfy the emerging security requirements of the LDAP standards on the Internet. This mechanism allows the ASP administrator to control what data can be extracted from a database based on the name of the person and the name of the note in the database.

The two most important shared databases are the Domino Directory itself and the administration requests database. Both are single databases in the Domino architecture that need to be shared by all of the hosted organizations. In the Domino Directory, all of the Person and Group documents have hierarchical names. The hosted organization name is part of that hierarchy. The xACL mechanism ensures that a person in company Acme can only see the Person documents whose organization name is Acme. It's a very strong, low-level mechanism.

We added a second mechanism that hides all of the application databases that belong to one organization from all of the other organizations. This new mechanism is similar to the dirlink mechanism that exists in Domino today. [For a definition of dirlink, see
this sidebar.] During hosted organization registration, the new mechanism automatically creates an ACL file in a subdirectory of the data directory. In the ACL file, an administrator can specify who can see the subdirectory and who can access its databases. In this way, the ASP administrator can use the ACL files in every organization's subdirectory of the data directory to specify that only the ASP administrators and that particular organization can see the subdirectories. Hosted organizations don't know about the data files that belong to other hosted organizations. ASP for Rnext offers all of this in addition to Domino's very strong standard set of security services.

What are your goals in terms of scalability? What size services are you planning initially, and then down the road?

We set very high goals for the engineering unit tests in the ASP project. Our goals are to locate any impediments to large scale deployment of the ASP product, and to fix those impediments. The actual number of users and the number of hosted organizations that will fit on any one machine, or set of machines, at an ASP site will be determined later when our performance team does their formal measurement tests of the Rnext product. So to rephrase, there is no architectural limit or coded limit for this. It is really going to depend on the users' user profiles, what actions the users are performing, and what applications they are running. For example, we could place 100,000 mail files on a single modest-sized box, reboot and crash the system, run full-text indexing on all mail databases, and then monitor every Domino feature that suffers a performance problem as a result of this test. When we place a large number of users and databases on a computer, we are testing with a number that is far beyond what we would expect any customer to use. This allows us to find the difficulties of the product.

How are you ensuring this scalability?

We are not putting any constraints into the system, instead, constraints are related to user profiles and to the quantity of user activity. Consider a ratio of registered users versus active users. Each organization might have 100 employees. During a certain period of time, the percentage of active employees will depend on the customer set and the applications that are in service. For example, using POP3 for your mail service requires low overhead because the mail resides on the client machine, but using Webmail is more server-intensive because the management of the mail files and the reading of mail is done by code in the server. The nature of the applications determines a great deal of the scalability.

You mentioned that the ASP's profit making potential depends on the ability to support many hosted companies with minimum ASP resources. What improvements allow a single server to support more users and more databases?

There is actually a fairly long list of modest to major improvements that provide more scalability. These improvements have been initiated and engineered across all of the teams working on Domino. I'll just discuss a few of them.

One is the introduction of the Configuration Only Domino Directory. This new directory configuration provides the ability to store the Person and Group documents, which are the majority of the volume of the Domino Directory, on one or more servers that are dedicated to doing directory lookups. All other servers in the domain only have Configuration documents in the Domino Directory—no Person or Group documents. This allows those servers to dedicate their processing and disk capacity to running applications for the end users while all of the name lookups are done on the full Domino Directory.

The Web Server team has initiated a major enhancement to the HTTP server process for Rnext that will improve scalability and performance, and provide new services.

A third improvement initiated for Rnext is the shared template service that allows similar databases to share a single template. This shared template service allows all mail files on one machine to use a single template containing the definition of the forms that are used in the mail database. This includes all of the code used in the mail database to support calendaring and scheduling. This greatly reduces the disk footprint and provides a modest performance improvement because of the way Domino caches.

Next is the IMAP service. The job of the IMAP server is to meet incoming IMAP protocol requests from IMAP clients by accessing Domino mail files. The nature of the IMAP protocol was slightly different than the design of the mail files in Domino R5 and some processing had to be done to meet this difference. In Rnext, the IMAP server is being modified to use some new core services in Domino that reflect the exact IMAP protocol requirements to better fit the design of the mail file. This greatly improves the IMAP performance, and again, because the performance is better, it will scale better.

Then, for all of the Internet servers, we have introduced a networking improvement that was initially introduced only for the Domino database server in R5. This improvement, IO Completion Port (IOCP), enables a small number of threads to service a very large number of sessions. In R5 all of the Internet servers have a single thread per incoming session. The cost to most operating systems to service multiple threads is fairly high. For example, servicing 5,000 connections with 50 threads is a major improvement in performance, and again, in scalability.

Finally, the replication mechanism that Domino is quite famous for, and that we are very proud of, has undergone some improvements in Rnext. These improvements bundle together a number of the data items that must be replicated between two servers, thereby reducing the number of packets that are sent and compressing the data during the sending. Again, it improves performance and provides better scalability.

We investigated, across each of the engineering teams, the effect of placing a large number of hosted organizations and databases on a server. We wanted to determine how the server reacts when excessive demands are placed on it. We found a few places in the code where things weren't optimal. For example, restarting the system might have taken 15 minutes with 100,000 databases. For each instance of poor performance due to large scale, we've introduced caches and we've improved the code so that the number of databases and organizations does not slow down a restart of the system and does not impede normal production.

An ASP requires a high degree of availability/reliability to compete. How will you help to provide this?

Although there are many textbook definitions of availability, the important one for the ASP is to ensure that no end user serviced by the ASP ever calls and complains. The end user who is using the services of the ASP expects constant service. There are two major ways in which an ASP can provide this high-quality service to the end user; one is reliability. If the servers never go down, the end users are happy. In the real world there are power failures, there are machine failures, and there are software failures. The second way to provide this high degree of availability is to have backups for all of the components of the ASP's service offerings. When one component fails, another component can be brought in quickly. The end user, again, sees constant service.

We have done some things that have improved reliability. First, we have made a number of process changes in development to meet our goal of shipping the first release of Rnext with the same quality as that found in the third, quarterly release of previous versions.

Second, we have added the ability to automatically restart the Domino server following a software failure and we have reduced the time to start the server.
To increase availability, we have investigated all the mechanisms that provide backups and we have ensured that all are available in ASP configurations. These include Domino clustering, which gives you backups for failed CPUs and failed disks, as well as OS clustering which will backup CPUs, power supplies, and processes. We also support the use of network disks with shadowing to provide backup for disk failures, as well as supporting network sprayers to redistribute incoming connections to hot backups. I believe that ASPs have a broad range of choices for providing hot backups and all are supported by Domino ASP.

An ASP's profits also depend on low administration costs. How are administration costs kept down? How are you ensuring administrative ease-of-use?

There are two changes that we have provided specifically for the ASP environment. Before I continue, let me just say that we believe the Domino administration client and its set of tools is very, very good. It administers all of your servers from one client and provides a wide range of tools to provide easy, low-cost administration. We have added a new service in Rnext, specifically for ASPs, which provides a single user interface for creating a new hosted organization. This should lower the cost of adding new organizations at sign-up.

We know that distributing the administration can lower the costs. The ASP has the choice of doing all of its administration from the ASP site, or distributing the administration of hosted organization end users and groups to the hosted organization themselves. This will be done through use of the Web administration client from the hosted organization. Modifications are being made in Rnext to both the administration client and the Web administration client to make it quick and easy to administer a single hosted organization.

Let's talk about activity logging and billing. How are you handling those elements?

We have definitely improved the gathering of billing data in Rnext. In the past, our billing mechanisms in Domino have not been the best. The major change is the addition of the new activity logging service. [For a definition of activity logging, see
this sidebar.] To determine what we needed to do for the ASP project, we examined every use of the services in Domino and determined we need to be consistent about the data that is collected from every service. We also noted that the data we collected had to be complete. Our research led to the development of the activity logging feature.

Now that we are collecting complete data from all the components of a server, the data collection must be fast and must never fail or lose data because a queue is full or processing power is not available. The mechanism for the activity logging service is an extremely low-cost, high-density mechanism. Activity data is queued in memory, well-compressed, and large records are written periodically to the log file. When the data needs to processed, it can be done at an off-time, or it can be done on another machine. The collection speed is fast enough to ensure that no data is ever lost. There will be APIs in the SDK [software development kit] that allow the ASP to write billing software to use the data in a way that works best for them.

The activity logging mechanism is also configurable. The ASP can decide what services they are billing for, and then only collect the data they need to bill according to their billing model.

What tools will you offer for detecting problems and monitoring for capacity planning?

The existing tools in the administration client are actually quite good for monitoring servers, but some improvements have been made for Rnext. We made some modest changes to the way in which log entries and events entries are presented to the end user to make it easier to use logs and events. Second, there is a new Server Health Monitoring service that watches for capacity problems. [For a description of Server Health Monitoring, see
this sidebar.]

Will you talk a little about Web application development? Will the ASP model change the way applications are developed? Are there areas that application developers will have to look at differently, such as application design, support, and management?

As I've mentioned before, this release of Domino ASP offers many improvements. Domino has been providing Domino data-based and Domino Designer-based Web applications for a number of years. Many customers are very happy with that. Additionally, on the Internet there is now emerging a fairly-standard Java-based application development environment. In Rnext, we are providing this Java-based application development environment. In addition to using the Domino Designer, we are making it easier to use non-Designer based application development tools. Both of these services, the support of Java-based development environments and the use of non-Domino tools, makes it possible to shop from a broader range of ISVs and a larger market of application developers.

Second, there is an important consideration in application development for the ASP. When a single company develops an application, and the application has some problems such as periodically consuming a little too much CPU time or crashing once in a while, only that organization is affected. In the ASP environment, if the ASP adopts an application without first providing strict quality controls, if it uses too many resources or it fails, all of their customers are affected. So, an important business practice of the ASP, independent from our products, is to ensure that every application they obtain is secure, reliable, easily configured for multiple hosted organizations, and provides low cost data management for each hosted organization. We are providing, again, the standard Domino tools and some new tools to make it easy for both the ASPs and the ISVs to meet these applications requirements.

Third, we are providing for the management of applications for multiple organizations. To meet this need, we are integrating with the currently available product, Lotus Hosting Management System (LHMS) which provides a set of management tools to distribute applications across multiple hosted organizations. This will definitely be a major part of the ASP offerings for Rnext.

Were new technologies used to achieve the features you wanted? Was there specific synergy with other development efforts?

The rate at which technology advances is much faster than anyone can keep track of. I was definitely surprised by the size of the machines we used for testing. I was quite shocked when I received my first complaint that on a Sun Solaris 64-processor system a particular software problem appeared—and what were going to do about It? I had not realized that there was such a thing as a 64-way system. I guess the real challenge for us was to set up test environments in which we tested the scaling and performance of very large systems. We have a completely independent group at Iris whose sole job is to do this testing and they have purchased very large systems with very large client simulators, and we've done a fair amount of very big ASP testing.

Have there been any particularly difficult technical challenges?

When we initially set our goals, the goals were to improve scalability and performance. If we came to a particularly knotty problem, there was often a compromise. The solution may have improved performance, but it wasn't ideal for the security of Domino. There is no compromise with respect to security. The security must be complete. That's probably been the hardest challenge for us. I think by using the xACL mechanism, and the ACL database mechanism, we are providing ASPs with the necessary complete security services.

Were there any ASP requirements that caused improvements in Domino that will be seen by all Domino customers?

Actually, except for the virtualization of the Domino Directory, everything we've talked about in this interview is an improvement that will be seen by anyone running Domino Rnext.

After the initial offering with Rnext, where do you see the ASP project heading—both in terms of features and technology?

There are two different directions in which we want to proceed at the same time. First, we want to provide the ability to build a farm of protocol and database server machines that greatly improves scalability and reliability. Second, we will listen to every ASP customer who uses our new service and we will make plans for the next release to please those customers. We recognize that the ASP industry is still an emerging industry. There is a small number of ASPs, and we believe as do industry analysts, that the ASP industry is going to grow significantly over time. Every successful company in the computer industry understands that you must pay attention to the emerging marketplace. It will always change in a direction you hadn't foreseen and at a rate that you couldn't predict.

During the month of February 2001, Brian and Peter will be taking your questions about the ASP project in the
Iris Cafe Developer Spotlight Forum here on


Mary Jrolf is a Principal Technical Writer and project leader at Lotus. She is responsible for planning and writing several sections of the Domino Administration documentation, as well as the ASP documentation.