What is a performance benchmark?

In my experience, when adopting a buyer’s perspective,  I have seen 3 main classes of performance benchmarks:

  1. Check-mark benchmarks
  2. Decision confirmation benchmarks
  3. Decision driving (risk-reducing) benchmarks

 

Check-mark benchmarks are usually driven by internal processes tied to best practices associated with quality processes or required by laws and regulations.

In most cases the benchmarks falling into this class are perceived as a pure cost that needs to be minimized: an industry standard benchmark is usually the cheapest answer to the need for a benchmark result.

The Wikipedia article on the general performance benchmarking subject adopts a perspective that matches very well this type of benchmarks.

The key principle, from the 7 proposed Benchmarking Principles, that in my opinion position the article as a description of “check-mark benchmarks” is the one called Representativeness “Benchmark performance metrics should be broadly accepted by industry and academia”.

Several years ago Curt Monash wrote here that “The TPC-H benchmark is a blight upon the industry”.

Not only I fully agree with him about TPC-H, but I would expand the statement further: as of today all the industry standard benchmarks serve (some) vendors, but not the buyers.

I find the rest of the principles listed in the article sound and relevant for all the classes I’m describing in this post, but I use a slightly different definition for relevance (the test should measure metrics relevant to the business problem that needs to be solved by the technical solutions tested) and transparency (not only the metrics need to be easy to understand, but also the test conditions and how changing these condition can influence the results should be clear).


Decision confirmation benchmarks are executed to demonstrate the correctness of a decision that has been already taken.

When running such a test there is a high risk of a confirmation bias coming into play in the way the test is defined with the tests favoring the technical solution that has been selected.

Because the decision is already made the benchmark is seen as a cost to minimize rather than an investment also in this case.


Risk-reducing benchmarks are executed, as the definition implies, to minimize the risks associated with the selection of a specific technical solution to address a set of business needs.

The costs associated with the selection of an incorrect or sub-optimal solution can be very significant for an enterprise with the direct ones (incurred to implement the solution) usually being just a fraction of the total.  The cost of (lost) opportunity is usually the largest part.

When looking at the performance benchmark from this perspective the buyer sees the costs associated with the preparation and execution as an investment like it would be the case for an insurance.

Minimization of cost is no longer the main design criteria and is replaced by the balance between the ability to predict the future behavior of the different technical solutions when implemented with the buyer’s specific processing pattern and the cost of defining and running the test.


A single exercise might show characteristics of more than one of the classes, but in my experience the (mainly) risk-reducing performance benchmarks are a very small fraction.

What is your experience  in this regard?

Performance benchmarks for the operational people

In August I promised (here) to post a target table of content of this series of posts about performance benchmarks and delivered the first part covering the subjects I believe are relevant to managers.

I label “operational people” any individual or organization that is involved hands-on in the definition, implementation, execution and technical evaluation of a performance benchmark.

Today you find below the candidate list of subjects I believe are relevant to the operational people.


1) Knowing your starting state

  1.   The system(s) you are going to replace
  2.   The interaction between the systems
  3.   The workload on each system

2) Simulating your final state

  1.   Black box definition of the final state
  2.   Defining the workload insisting on the black box based on the current state
  3.   Defining the workload generated by functional and/or organizational changes

3) Ensuring the technologies are set to perform the best way for the desired final state

  1.   Defining the metrics
  2.   Defining how the metrics are grouped
  3.   Sharing the metrics with the vendors/implementation teams

4)  Executing the tests and reviewing the results.


 

Performance benchmarks for the manager

I promised last week (here) to post a target table of content of this series of posts about performance benchmarks.

It is a quite long list of topics that I split in two main areas:

  • Topics that are relevant for everyone in the organization, labeled “for the manager”,
  • Topics that are of interest mostly for the technical people, labeled “for the operational people

Below you find the candidate subjects that I believe are of general interest.


1)   What is a performance benchmark

2)  Types of technical benchmark

  1.   The check mark benchmark
  2.   The confirmation benchmark
  3.   The industry standard benchmark
  4.   The valuable benchmark

3)  Organizational challenges to implement a valuable benchmark

  1.  The IT architecture perspective
  2.  The IT operations perspective
  3.  The CFO perspective
  4.  The CEO perspective

Next post will contain the list of technical subjects while the following ones will start to dig into each subject in the two lists in an orderly fashion.

As I wrote earlier: your feedback will be key in shaping how this will move forward. Feel free to comment here or to reach me directly.

Disclaimer

The opinions expressed in this blog are my own, not the opinions of my current employer or the opinions of any of my former employers and might diverge from their current, past or future opinions on the subjects I discuss. 

The opinion expressed are not the opinions of any of the associations I was part of in the past or I am part of today.

All the content of this blog is mine: it can be freely quoted in academic papers and free publications as long as credit is provided and the content is linked, but can’t be included into any paid publication like books, e-books, web content behind a paywall or any money-generating media without prior written authorization.

I reserve the right to change my mind on any of the subjects I write about and the freedom to share or not to share my revised opinion.

While I take no responsibility for the comments made on my blog I reserve the right to delete any comment as I see appropriate without the obligation to provide an explanation of my action.

I provide credit to my sources to the best of my knowledge: I will be happy to revise and correct any quote and credit if notified.

While I share my experience (positive or otherwise) in full honesty your mileage may vary and what worked for me might not work for you: the readers are the sole responsible for taking or not taking any action or decision (financial or otherwise) based on what I write in this blog.

I am not making money out of this blog nor get free products or perks of any type associated with my blogging activity.

There is no conflict of interests between this blog and my work or financial activities.

I reserve the right to change the terms of this disclaimer at any time.

 

2014: eating @ Bar della Crocetta in Milano

I used to eat at Bar della Crocetta since I was in my highschool days and the place was managed for almost 30 years by the same crew.
It was the best sandwich place of the entire city without any doubt: rich, creative, balanced in the mix of tastes.
Just fantastic.
But two years ago the owners have sold it.

three days ago, as I arrived at the door with my wife, I immediately noticed that something had changed: cleaner place, more light, no more sausages in display, female waiters.
In theory all good changes, but I had a negative gut feeling.

The place was almost empty, but it was the day of the soccer world championship final so it was reasonable.

The list of sandwiches got axed: about 70% less choice than in the past.
Also the list of different types of sausages halved.
I went for a classical sandwich (cipollata) that I had dozens of times in the past with the previous owners while my wife got something new based on Praga ham.

Then we started to wait.
More waiting.
After 30 minutes (with 12 customers in total in the place) we finally got the sandwiches and delusion with them.

The taste of my sandwich was just a pale memory of the original one and using the same name is absolutely inappropriate
Cipollata refers to the fact that it is based on onions (cipolle in Italian) but the quantity was so little and the onions so sweet that their contribution to the overall taste was minor.
The melted cheeses were too hard and not well amalgamated.
The thickness of the sandwich interior was about 60% of the old one while price adjusted with inflation was constant. Ans this was the XL version.
My wife complained tha the ham was sliced so thin that is was almost ethereal in her mouth and about the overall lack of taste. This never happened with the old owner.

To summarize: maybe now it is a healthier place, but it is no longer a place worth visiting.
The Bar della Crocetta si dead and anyone that experienced the place in the past should avoid it to avoid getting the same delusion I had.

Why We Get Fat by Gary Taubes

I love food.
A lot.
Really.

It is too bad that this usually goes together with consequences: extra weight and all the rest that then, in most cases, follows.

According to the author this is not necessarily always the case.
He explains in the book that it is more about quality of the food rather than about quantity: insulin production is the pivotal factor in storing energy as fat.

To support his point he shows several cases of populations with high fat ingestion that are not fat and that start to become fat once they start to use the “western diet”, groups of people with limited amount of available daily calories that become obese anyway, experiments with mice in laboratories.
His conclusion is that the foods that drive up the insulin production are the ones that will increase the fat buildup in a person.

On the basis of his findings the author suggests a diet that goes in the opposite direction of the ones promoted by the national health advisory councils in the western world: skip cereals (even unrefined), vegetables with high amid content and sugar while getting all the desired proteins, non-caloric vegetables and fat.

Sooner or later I’ll give a try at this shift of eating habits: most of the food I like better already falls in the categories he suggests hence I should be keep up with the change at least for a while.

Channel 4 on demand

In the past I’ve really appreciated the first series of Black Mirror that was briefly available on youtube.

As I was recently in the UK I decided to give a try to the second series as the streaming is not available in Italy.
The experience was far from stellar.

Channel 4 asked for a registration (a couple of minutes as email confirmation is needed) and then I got access to over two minutes of boring advertising (not something like the Superbowl ads) that I could not skip.
Fair enough as I was not paying.

Finally the show.
Actually the first 10 minutes, then again over two minutes of advertising.
The rate of advertising to content is really excessive: I’m never going to use the service again.