Looking to implement a SaaS solution (ERP or CRM solutions such as SAP, Salesforce or Workday) and think because it is a SaaS solution (all cloud and all mighty) you really do not need to assess the Performance and Scalability. You might be wrong – a real expensive WRONG!

Of late I have met one too many customers who think based on their understanding of SaaS solutions (or what they have been sold) that they do NOT need to performance test SaaS solutions. Now I’m not talking from the perspective of a solution provider but from a customer or business assurance perspective.

The big question: As a customer who is purchasing a SaaS solution, should I assess the Performance of the solution at my end?

The short and absolute answer is : YES. If you are keen to understand the nitty gritty’s before making an informed decision, then proceed below.

Caution: The information is based on my experience and discussion with industry experts. This is an attempt at consolidating our thoughts to educate the few who otherwise assume that a cloud solution means “infinite” and the responsibility for good performance has been delegated to the provider. Not always.

 

So what is a SaaS?

SaaS stands for “Solution-as-a-Service”. A standard solution being sold as a service where the charging model is usually based on number of users or business functional units or time based. However the backend is usually a shared environment (a cloud vendor or a private cloud setup). With cloud everyone thinks “infinite” and “scalable” however that is far from reality. The “smart” part is in managing the underlying resources efficiently and effectively. That is what SaaS providers set out to do – to manage their operational costs. They manage the underlying resources for you so you can just concentrate on the business aspects and not worry about the non-functionals. This sounds very good and yes delegating that responsibility is absolutely great however the ability to assure if the solution meets your non-functional requirements becomes difficult.

There are one too many solutions being rolled out and sold to clients as a solution out of a box. Most of these solutions are just amazing. They’re really solving excellent problems. With intuitive and easy to navigate UI designs, seamless asynchronous widgets, features such as aggregated data visualizations, popular process workflows and the likes; all this seems too good to be true and they truly are. I’ve seen some really good software being written out there that is changing the conventional way of software design approach. Yay.. That’s a super win for Technology!!!

So whats the fuss about?!

 

SaaS myths

Well we firstly need to get over the assumptions that are knowingly or unknowingly sold to clients by solution providers such as:

• Handles any user volume (or fluctuations too) using Dynamic resource scaling – fancy term eh?!
• Data security is the highest
• Near zero outages
• Always available
• Highly scalable

As these are the core and ideal features every solution architect dreams of building and at times is quite successful too, it is important to understand the factors that affect user experience and performance of an application. Once we are able to understand those factors, it is easy to understand – just because something is in the cloud or being provided as a SaaS doesn’t really mean those benefits will be visible or available to the customer.

 

Factors that affect Performance

Before we dive into the factors that affect performance, we need a hard look at how do we define “good” performance of an application? The easiest answer is “to have expectations”. If you do NOT have expectations you can’t define if your observations as either good or bad. This is where NFRs or Non-Functional Requirements are necessary. A good tester would know; an even better tester would know the reality that at times we never get NFRs. However we need to push for a practice where NFRs are being defined without which we cannot say if something is good or bad.

Some NFR examples:

• All content pages need to load within 5 seconds.
• Search responds 50 records or below within 3 seconds.
• Documents should load within 9 seconds.
• Reports should generate and load under 12 seconds.
• Backend XXXXXXX job with an average size of YYY records should finish under 2 hours.

The above examples are some high level NFRs that help to define what the business expectations are out of a Performance Assessment and also help in negotiating with the vendor in ensuring these are met.

Once we understand what the business expectations are, we can attempt at establishing the factors that affect Performance. Obviously these vary with every solution however the below factors are some common areas that we need to consider while assessing the performance of a SaaS solution.

• Where are the service datacenters located? What is the network latency?
  • Need to understand the network hops between the user and datacenters
• How will the application be accessed?
  • Network (office LAN and WiFi / external home networks / mobile network like 3G/4G)
  • Supported Devices (Laptop / Tab / Mobile)
  • Supported Browsers (IE 11, IE 12+, Edge, Chrome, Firefox, Safari)
• What are the Customer / Business flows and estimated expected usage? The flows can be:
  • Data intensive
  • UI intensive
  • Process intensive
  • Resource intensive

 

SaaS vendor contract is key

As more and more organisations move towards SaaS, it is imperative to separate the hype and capability from actual facts and best practices. At times due to the myths and hypes and missteps in a contract limits the full realisation of the SaaS value proposition. A clear example is when most if not all SaaS cloud subscription agreements don’t reflect a truly on-demand highly flexible and scalable model. The contracts are mostly never structured for actual consumption calibration. They are usually more like traditional on-prem software contracts with fixed fees payments for a specified-set of products and committed number of users.

In addition to the above at times the commitments are absolute and for multiple years, with obligations to make upfront payments of the fees associated with each year of the term. Through such cloud subscription agreements, SaaS vendors are acting like traditional software vendors where they look to protect predictable revenue streams by motivating customers to sign fixed-fee contracts with upfront commitments with very little flexibility.

Being a technologist I will try not to focus too much on the contract aspects of SaaS vendor management, however in my opinion it is key to involve technologists to help consider technical aspects of adopting a SaaS.

 

The bottom line is that when an organization is considering a SaaS solution, it is critical that they think as much about how they structure the subscription agreement with the right expectations by first recognising what they are and by involving experts at the right stage to assure quality expectations are met.

References : SaaS cloud subscriptions: When “On-Demand” really isn’t