What is a performance benchmark?

In my experience, when adopting a buyer’s perspective,  I have seen 3 main classes of performance benchmarks:

  1. Check-mark benchmarks
  2. Decision confirmation benchmarks
  3. Decision driving (risk-reducing) benchmarks

 

Check-mark benchmarks are usually driven by internal processes tied to best practices associated with quality processes or required by laws and regulations.

In most cases the benchmarks falling into this class are perceived as a pure cost that needs to be minimized: an industry standard benchmark is usually the cheapest answer to the need for a benchmark result.

The Wikipedia article on the general performance benchmarking subject adopts a perspective that matches very well this type of benchmarks.

The key principle, from the 7 proposed Benchmarking Principles, that in my opinion position the article as a description of “check-mark benchmarks” is the one called Representativeness “Benchmark performance metrics should be broadly accepted by industry and academia”.

Several years ago Curt Monash wrote here that “The TPC-H benchmark is a blight upon the industry”.

Not only I fully agree with him about TPC-H, but I would expand the statement further: as of today all the industry standard benchmarks serve (some) vendors, but not the buyers.

I find the rest of the principles listed in the article sound and relevant for all the classes I’m describing in this post, but I use a slightly different definition for relevance (the test should measure metrics relevant to the business problem that needs to be solved by the technical solutions tested) and transparency (not only the metrics need to be easy to understand, but also the test conditions and how changing these condition can influence the results should be clear).


Decision confirmation benchmarks are executed to demonstrate the correctness of a decision that has been already taken.

When running such a test there is a high risk of a confirmation bias coming into play in the way the test is defined with the tests favoring the technical solution that has been selected.

Because the decision is already made the benchmark is seen as a cost to minimize rather than an investment also in this case.


Risk-reducing benchmarks are executed, as the definition implies, to minimize the risks associated with the selection of a specific technical solution to address a set of business needs.

The costs associated with the selection of an incorrect or sub-optimal solution can be very significant for an enterprise with the direct ones (incurred to implement the solution) usually being just a fraction of the total.  The cost of (lost) opportunity is usually the largest part.

When looking at the performance benchmark from this perspective the buyer sees the costs associated with the preparation and execution as an investment like it would be the case for an insurance.

Minimization of cost is no longer the main design criteria and is replaced by the balance between the ability to predict the future behavior of the different technical solutions when implemented with the buyer’s specific processing pattern and the cost of defining and running the test.


A single exercise might show characteristics of more than one of the classes, but in my experience the (mainly) risk-reducing performance benchmarks are a very small fraction.

What is your experience  in this regard?

Performance benchmarks for the operational people

In August I promised (here) to post a target table of content of this series of posts about performance benchmarks and delivered the first part covering the subjects I believe are relevant to managers.

I label “operational people” any individual or organization that is involved hands-on in the definition, implementation, execution and technical evaluation of a performance benchmark.

Today you find below the candidate list of subjects I believe are relevant to the operational people.


1) Knowing your starting state

  1.   The system(s) you are going to replace
  2.   The interaction between the systems
  3.   The workload on each system

2) Simulating your final state

  1.   Black box definition of the final state
  2.   Defining the workload insisting on the black box based on the current state
  3.   Defining the workload generated by functional and/or organizational changes

3) Ensuring the technologies are set to perform the best way for the desired final state

  1.   Defining the metrics
  2.   Defining how the metrics are grouped
  3.   Sharing the metrics with the vendors/implementation teams

4)  Executing the tests and reviewing the results.


 

Performance benchmarks for the manager

I promised last week (here) to post a target table of content of this series of posts about performance benchmarks.

It is a quite long list of topics that I split in two main areas:

  • Topics that are relevant for everyone in the organization, labeled “for the manager”,
  • Topics that are of interest mostly for the technical people, labeled “for the operational people

Below you find the candidate subjects that I believe are of general interest.


1)   What is a performance benchmark

2)  Types of technical benchmark

  1.   The check mark benchmark
  2.   The confirmation benchmark
  3.   The industry standard benchmark
  4.   The valuable benchmark

3)  Organizational challenges to implement a valuable benchmark

  1.  The IT architecture perspective
  2.  The IT operations perspective
  3.  The CFO perspective
  4.  The CEO perspective

Next post will contain the list of technical subjects while the following ones will start to dig into each subject in the two lists in an orderly fashion.

As I wrote earlier: your feedback will be key in shaping how this will move forward. Feel free to comment here or to reach me directly.

Performance benchmarks

It is time for me to give back.

Dealing with performance benchmarks has occupied a fair share of my life from my early days in the computer world in the mid ’80s.

In the beginning it was mostly reading, with just a bit of writing, that today I would be ashamed of, in one of the early Italian BBS “newspaper” called “Corriere Telematico“.

At the time I could have never imagined that benchmarks would have a very large role in my career to the point that for about 8 years they even defined my job title.

Now, as I my transition into a new role is almost complete, it feels like the right time to write something about benchmarks that can help many people in the industry.


I recall reading in one of the paper magazines of my early days something along the lines of “benchmarks don’t lie, but liars do use benchmarks”. I believe it was on MCmicrocomputer but I can’t bet on this.

This bleak statement about benchmarks was true 30+ years ago and it’s still true now, but we should not throw the good away together with the bad: proper benchmarks were and still are useful tools for individuals and organizations alike.  It’s all about defining “proper” correctly in each context.

For a while, given the scarcity of published material on the subject, I was thinking of putting together a book, with the help of a friend of mine.

I fear I will not be able to put in all the time needed to complete it in a reasonable time frame and for this reason I decided to blog on the subject instead.

In the coming weeks (or months, I don’t know yet how this will work) I will share what I learned in many years as a source for anyone wanting to get closer to the holy grail of the “proper benchmark”.

I will be vendor and technology neutral, covering both the business and the technical sides.

Your feedback will be key in shaping how this will move forward.

In the next post I’ll share the target table of content of this series of posts.

Disclaimer

The opinions expressed in this blog are my own, not the opinions of my current employer or the opinions of any of my former employers and might diverge from their current, past or future opinions on the subjects I discuss. 

The opinion expressed are not the opinions of any of the associations I was part of in the past or I am part of today.

All the content of this blog is mine: it can be freely quoted in academic papers and free publications as long as credit is provided and the content is linked, but can’t be included into any paid publication like books, e-books, web content behind a paywall or any money-generating media without prior written authorization.

I reserve the right to change my mind on any of the subjects I write about and the freedom to share or not to share my revised opinion.

While I take no responsibility for the comments made on my blog I reserve the right to delete any comment as I see appropriate without the obligation to provide an explanation of my action.

I provide credit to my sources to the best of my knowledge: I will be happy to revise and correct any quote and credit if notified.

While I share my experience (positive or otherwise) in full honesty your mileage may vary and what worked for me might not work for you: the readers are the sole responsible for taking or not taking any action or decision (financial or otherwise) based on what I write in this blog.

I am not making money out of this blog nor get free products or perks of any type associated with my blogging activity.

There is no conflict of interests between this blog and my work or financial activities.

I reserve the right to change the terms of this disclaimer at any time.

 

Google maps killed my earlier habit of blogging here about places and food

When I started this blog a fair share of the content related to my travel and food experiences around the world.

Formally blogging is, in my opinion, a fairly involved process to ensure not only the content is relevant, but also the writing is at least properly structured. This, together with the fact that blogging is not a job for me and I have other hobbies too, led me to write only for significant experiences rather than always.

The excitement of writing was drying up over time (it is easy to see in the posting history) when the mechanism to contribute to Google Maps became available.

Albeit reviews on Maps tend to be shorter and a bit “twitter-style” the mechanism to contribute them is so convenient that I ended up being much more active than I was before.

An added stimulus to keep writing there is the monthly feedback showing the level of visibility of my contributions: reviews on Google Maps have a visibility that this blog never reached nor was ever going to reach while keeping its nature of a small side project.

If you are interested in keeping up with my reviews you should be able to find them here

I the end Google Maps contributions might be considered the last nail in the coffin of my writing, but I rather like to look at them just like an evolution of it and a greater value for the community than the earlier model.

Smart car software update: chronicle of an unacceptable journey.

I recently posted about my very unsatisfactory experience with service personnel while attempting to get a few problems fixed on my Renault Megane.
The mechanics had no clue about how to fix them, but a factory reset of the on-board computer (like on current personal computing devices) did the trick.

I inferred from this fact that updating the software, again like in personal computing devices, was the way to go to avoid facing the same problems in the future and started my long journey to accomplish this.

I followed the manufacturer instructions and downloaded the software downloader on my notebook, inserted in it a 8GB USB flash drive previously initialized in the car and, after a byzantine procedure requiring web interaction to select the updates that then the application would fetch, I started downloading.
Again. And again. And again…

What looked strange is that the download counter made it to the full size, but then continued!
After a few dozens attempts all failed in the same way and with no success in sight I decided to get in touch with the country support.

As a reply to my first contact I received a cut&paste of the standard procedure.
This is a fairly common practice in every sector and makes a lot of sense because most people is not reading the manuals.

Unfortunately I was already following the standard procedure so I replied back with more data including the fact that to get 5.4GB of updated maps the tool had downloaded already over 113GB (from a non-Renault domain) without success.
The solution proposed was to use a larger flash drive.
I could not obtain from them an answer about why to get 5.4GB an empty 8GB drive was not enough.
And a 16GB drive was not a fix for the problem anyway.

During the fruitless exchanges with the support I kept attempting the download until it finally worked. On the 8GB drive.
I believed that even if this was not communicated to me they had fixed whichever issue there was and I was happy with that.

A few months later I found out that it was just one lucky astral alignment.
The situation is back where it was: tens of downloads attempts needed to get an updated version of the maps and failed downloads leave the flash drive in an inconsistent state where the car tries the update anyway only to fail after a few minutes.

I was guessing in my earlier post that the challenges I faced were due to the time needed for the knowledge to move from the top of the manufacturer organization to the service people.
But from my experience attempting to do the software update it looks like I was wrong: even at the country level the manufacturer appears unable to support the smartness they are putting in the vehicles.

According to the discussions I had with a few colleagues in the office other manufacturers have a much smoother user experience.
In my opinion Renault really needs to evolve quickly to stay relevant.