What is a performance benchmark?

In my experience, when adopting a buyer’s perspective,  I have seen 3 main classes of performance benchmarks:

  1. Check-mark benchmarks
  2. Decision confirmation benchmarks
  3. Decision driving (risk-reducing) benchmarks

 

Check-mark benchmarks are usually driven by internal processes tied to best practices associated with quality processes or required by laws and regulations.

In most cases the benchmarks falling into this class are perceived as a pure cost that needs to be minimized: an industry standard benchmark is usually the cheapest answer to the need for a benchmark result.

The Wikipedia article on the general performance benchmarking subject adopts a perspective that matches very well this type of benchmarks.

The key principle, from the 7 proposed Benchmarking Principles, that in my opinion position the article as a description of “check-mark benchmarks” is the one called Representativeness “Benchmark performance metrics should be broadly accepted by industry and academia”.

Several years ago Curt Monash wrote here that “The TPC-H benchmark is a blight upon the industry”.

Not only I fully agree with him about TPC-H, but I would expand the statement further: as of today all the industry standard benchmarks serve (some) vendors, but not the buyers.

I find the rest of the principles listed in the article sound and relevant for all the classes I’m describing in this post, but I use a slightly different definition for relevance (the test should measure metrics relevant to the business problem that needs to be solved by the technical solutions tested) and transparency (not only the metrics need to be easy to understand, but also the test conditions and how changing these condition can influence the results should be clear).


Decision confirmation benchmarks are executed to demonstrate the correctness of a decision that has been already taken.

When running such a test there is a high risk of a confirmation bias coming into play in the way the test is defined with the tests favoring the technical solution that has been selected.

Because the decision is already made the benchmark is seen as a cost to minimize rather than an investment also in this case.


Risk-reducing benchmarks are executed, as the definition implies, to minimize the risks associated with the selection of a specific technical solution to address a set of business needs.

The costs associated with the selection of an incorrect or sub-optimal solution can be very significant for an enterprise with the direct ones (incurred to implement the solution) usually being just a fraction of the total.  The cost of (lost) opportunity is usually the largest part.

When looking at the performance benchmark from this perspective the buyer sees the costs associated with the preparation and execution as an investment like it would be the case for an insurance.

Minimization of cost is no longer the main design criteria and is replaced by the balance between the ability to predict the future behavior of the different technical solutions when implemented with the buyer’s specific processing pattern and the cost of defining and running the test.


A single exercise might show characteristics of more than one of the classes, but in my experience the (mainly) risk-reducing performance benchmarks are a very small fraction.

What is your experience  in this regard?

Performance benchmarks for the operational people

In August I promised (here) to post a target table of content of this series of posts about performance benchmarks and delivered the first part covering the subjects I believe are relevant to managers.

I label “operational people” any individual or organization that is involved hands-on in the definition, implementation, execution and technical evaluation of a performance benchmark.

Today you find below the candidate list of subjects I believe are relevant to the operational people.


1) Knowing your starting state

  1.   The system(s) you are going to replace
  2.   The interaction between the systems
  3.   The workload on each system

2) Simulating your final state

  1.   Black box definition of the final state
  2.   Defining the workload insisting on the black box based on the current state
  3.   Defining the workload generated by functional and/or organizational changes

3) Ensuring the technologies are set to perform the best way for the desired final state

  1.   Defining the metrics
  2.   Defining how the metrics are grouped
  3.   Sharing the metrics with the vendors/implementation teams

4)  Executing the tests and reviewing the results.


 

Performance benchmarks for the manager

I promised last week (here) to post a target table of content of this series of posts about performance benchmarks.

It is a quite long list of topics that I split in two main areas:

  • Topics that are relevant for everyone in the organization, labeled “for the manager”,
  • Topics that are of interest mostly for the technical people, labeled “for the operational people

Below you find the candidate subjects that I believe are of general interest.


1)   What is a performance benchmark

2)  Types of technical benchmark

  1.   The check mark benchmark
  2.   The confirmation benchmark
  3.   The industry standard benchmark
  4.   The valuable benchmark

3)  Organizational challenges to implement a valuable benchmark

  1.  The IT architecture perspective
  2.  The IT operations perspective
  3.  The CFO perspective
  4.  The CEO perspective

Next post will contain the list of technical subjects while the following ones will start to dig into each subject in the two lists in an orderly fashion.

As I wrote earlier: your feedback will be key in shaping how this will move forward. Feel free to comment here or to reach me directly.

Performance benchmarks

It is time for me to give back.

Dealing with performance benchmarks has occupied a fair share of my life from my early days in the computer world in the mid ’80s.

In the beginning it was mostly reading, with just a bit of writing, that today I would be ashamed of, in one of the early Italian BBS “newspaper” called “Corriere Telematico“.

At the time I could have never imagined that benchmarks would have a very large role in my career to the point that for about 8 years they even defined my job title.

Now, as I my transition into a new role is almost complete, it feels like the right time to write something about benchmarks that can help many people in the industry.


I recall reading in one of the paper magazines of my early days something along the lines of “benchmarks don’t lie, but liars do use benchmarks”. I believe it was on MCmicrocomputer but I can’t bet on this.

This bleak statement about benchmarks was true 30+ years ago and it’s still true now, but we should not throw the good away together with the bad: proper benchmarks were and still are useful tools for individuals and organizations alike.  It’s all about defining “proper” correctly in each context.

For a while, given the scarcity of published material on the subject, I was thinking of putting together a book, with the help of a friend of mine.

I fear I will not be able to put in all the time needed to complete it in a reasonable time frame and for this reason I decided to blog on the subject instead.

In the coming weeks (or months, I don’t know yet how this will work) I will share what I learned in many years as a source for anyone wanting to get closer to the holy grail of the “proper benchmark”.

I will be vendor and technology neutral, covering both the business and the technical sides.

Your feedback will be key in shaping how this will move forward.

In the next post I’ll share the target table of content of this series of posts.

Smart car software update: chronicle of an unacceptable journey.

I recently posted about my very unsatisfactory experience with service personnel while attempting to get a few problems fixed on my Renault Megane.
The mechanics had no clue about how to fix them, but a factory reset of the on-board computer (like on current personal computing devices) did the trick.

I inferred from this fact that updating the software, again like in personal computing devices, was the way to go to avoid facing the same problems in the future and started my long journey to accomplish this.

I followed the manufacturer instructions and downloaded the software downloader on my notebook, inserted in it a 8GB USB flash drive previously initialized in the car and, after a byzantine procedure requiring web interaction to select the updates that then the application would fetch, I started downloading.
Again. And again. And again…

What looked strange is that the download counter made it to the full size, but then continued!
After a few dozens attempts all failed in the same way and with no success in sight I decided to get in touch with the country support.

As a reply to my first contact I received a cut&paste of the standard procedure.
This is a fairly common practice in every sector and makes a lot of sense because most people is not reading the manuals.

Unfortunately I was already following the standard procedure so I replied back with more data including the fact that to get 5.4GB of updated maps the tool had downloaded already over 113GB (from a non-Renault domain) without success.
The solution proposed was to use a larger flash drive.
I could not obtain from them an answer about why to get 5.4GB an empty 8GB drive was not enough.
And a 16GB drive was not a fix for the problem anyway.

During the fruitless exchanges with the support I kept attempting the download until it finally worked. On the 8GB drive.
I believed that even if this was not communicated to me they had fixed whichever issue there was and I was happy with that.

A few months later I found out that it was just one lucky astral alignment.
The situation is back where it was: tens of downloads attempts needed to get an updated version of the maps and failed downloads leave the flash drive in an inconsistent state where the car tries the update anyway only to fail after a few minutes.

I was guessing in my earlier post that the challenges I faced were due to the time needed for the knowledge to move from the top of the manufacturer organization to the service people.
But from my experience attempting to do the software update it looks like I was wrong: even at the country level the manufacturer appears unable to support the smartness they are putting in the vehicles.

According to the discussions I had with a few colleagues in the office other manufacturers have a much smoother user experience.
In my opinion Renault really needs to evolve quickly to stay relevant.

Smart cars without smart mechanics in the long run are not going to work as a business model.

A few months ago I started to drive a 4th generation Renault Megane in the (Italian) Bose trim:
in this version you get almost as many gadgets as possible.

While they all work driving the car is a very enjoyable experience for the vehicle class, but as soon as problems started to appear and I was looking for a fix, I realized that the support personnel was left behind in the product evolution.

After a few months the electric massage seat and the lumbar support stopped to work, some time later the rear cam did not disengage anymore as soon as moving forward, after some more time parking sensors stopped working and also the lane assist stopped to produce the sound feedback, finally the HUD was resetting the position to default every time I was turning off the engine.

I an attempt to get the issues resolved I have contacted 3 different mechanics from the official support network getting vague statements about what the problem could be, but all of them agreed that it would take multiple days to get it fixed. One stated “for electric problems you need to plan at least a 3-days stop”.
I tried contacting the online support describing the issue and all I got back was the link to the list of services.

None of the mechanics offered a replacement car during the troubleshooting and repair even if the vehicle is well within the warranty period: very upsetting.
I started planning the right time to bring the car in when I could stay without it for an extended period of time when, by pure chance, I ended in a menu of the car computer that offered a reset to factory defaults.
Having some past experience with consumer electronics I decided to trigger it counting on the fact that worst case if the car stopped completely I could call the service to pick it up free of charge.
With my surprise all of the problems I was having suddenly disappeared.

How it is possible that not only 3 authorized services had no clue about this basic troubleshooting, but also the online support did not come up with the advice to reset?

In my opinion putting cars ahead of the support structure is not a safe bet.
Not for the for the manufactured nor for the consumers.

Thinkering with my desk: an attempt at IoT as in “intranet of things” rather than “internet of things”.

Microsoft sends me a periodic email (TechNet Flash Newsletter) listing the news related to their product and ecosystem that I read in a sporadic way, but a few weeks ago thanks to the Christmas holidays I had a bit of extra time and read through one containing an invite to a challenge on hackster.io.

Joining the community was quick and straightforward.
After a few days I put in an idea for the challenge pre-contest and earlier today I found out that it was selected and I should get the Genuino MKR1000 to make it a reality.

Tools are installed on my W10 phone and notebook ready to consume my week-end spare time for the next 46 days.

I believe this marketing initiative is a very smart one giving good visibility of MS tools in the IoT space to the people who should really care (developers and tinkerers) and can create the tools, applications and devices that will feed the Azure infrastructure with major volumes of data in the coming years.