Tuesday, 18 February 2014

Tender evaluation and linearity

As two of the founding members of Market Dojo are engineers and the other a physicist, we take our maths very seriously.  One of the reasons for developing our auction platform for the public sector during our first grant is that we wanted to demonstrate how easy it was to set up and execute a weighted tender.  The specific area we want to address here is the scoring system.

We have been baffled by the illogical and non linear nature of many of the scoring systems in the market, and were determined to provide something better.

Firstly we would like to describe how we have approached the problem and then look at why some of the other common techniques on the market are challenging to use from the buyer and supplier side and how some are even fundamentally flawed.

Here is a section of our capability; the full version can be seen here.



We think any scoring mechanism should be linear and straightforward.  Thus if you are running a weighted tender you need to be able to clearly communicate the mechanism for converting the various elements to scores and how they are combined.  

[You can try out all that you read here by just registering with Market Dojo for free and experimenting with setting up weighted events in the Sandpit.]

Questionnaire scoring:

We have made the set up for a questionnaire very easy, providing many different question types.  On top of this you can create a questionnaire which is scored and also weighted by question, section or both.  The simplest way to show you what this means is an example output.



Here you can see that this questionnaire is scored and also weighted by question and section and the total weighted score is the final result for the questionnaire for this participant.

Price Scoring:

We have two techniques for price scoring on our application.  

Pro Rata: The system scores the price based on where it sits between the lowest price and the qualification price. This is a simple linear method of determining differences between prices. The qualification price is set before the event and is the highest bid you are willing to receive. The lowest price is the current lowest bid in the event. [One could look at a further extension to this method by setting the lowest bound but this presents its own challenges].

You might use this method when you are looking at many complex factors and need to have greater control about making the scoring methodology suit your tender. You can judge your price scoring against your highest acceptable price.  

Percentage of leading bid: The is a simple and powerful technique.  As an example, if the highest bid in a reverse auction is £100 and the next best bid is £110, then the lowest bid gets 100% and the next highest bid gets 90%, as it is 10% over.  This is also a linear mechanism which does not need a qualification price.  One potential disadvantage is that if a bid is double the lowest bid, the participant will get a score of zero, although that may be a fair outcome.

One benefit of this method is that the non-price elements can be equated directly to price.  For example if the non-price element is worth about 1% of the estimated final price then you can weight the non-price elements to be 1% and the price 99% and the evaluations works out consistently (see the example below).  

This method can lead to difficulties in the public sector arising from rogue bids and/ or participants having the same score of zero when they have different bids.  In the private sector though you have more flexibility and can decline participant during an auction.

Combining scores:

The weighting for the price and the questionnaire should be set before the event. The participants bids are ranked on their score, calculated as:

[Questionnaire weighting] * {Questionnaire score} + [Price weighting] * {Price score} = Overall score.

For example:


You can see in the scenario above that the ‘Percentage leading bid’ method works very well.  Here in round 2, one supplier with better payment terms but a worse price has an equal overall score.  The added value of 90 days payment terms compared to 60 days equates very nicely to 1% of the overall score.

Auction:

During the auction you want to easily see the overall score live and to exactly determine how the weighting is effecting the result.

Here you can see the leading supplier based on the overall score:



And you can also drill into the weighted scoring per bid:


The result is an easy to use weighting mechanism that can be used during a live auction with good visibility of all the related scoring..

Issue with current scoring mechanisms on the market:

Whereas any scoring system will have potential flaws, there are some systems on the market which are more flawed than others and some that are simply incorrect.

The multiplier:

This method uses the questionnaire score to create a price multiplier during the auction.  We find it hard to imagine something so incorrect.  There are many issues with this type of mechanism, which we can only think is a result of a software provider not developing an appropriate mechanism but fudging an exchange rate approach to solve the issue,  such as:

1.  In a weighted auction, if the participant has a price score of zero, they should still have a questionnaire score.  However, with the multiplier mechanism the overall score will become zero.

2.  The methods we have seen for determining this ‘multiplier’ are very much non linear and are impossible for the buyers and suppliers to truly understand how the mechanism would react.

3. This method skews non price elements. Let’s say that quality is the non price element used to determine the multiplier.  During the auction the multiplier will create different results based on the price.  Let’s say Supplier A has a price of £10 and Supplier B has a price of £12.  Let’s also say the multiplier is only applied to Supplier A and is 1.1.  In this case the overall score for Supplier A would be £11 to put them in first place.  Alternatively, let’s say Supplier A had a price of £100 and Supplier B with £102.  Here the price is still separated by £2 but with the multiplier applied to Supplier A (£110) they now fall into second place.  In other words the non price element with the multiplier means different things at different prices.

Percentage of leading bid:

For a forward auction, the equation for percentage of leading bid is a simple linear scoring result : ‘Bid divided by Highest bid’.  However, in the public sector we have often seen the inverse of this formula used as the price score calculation in a reverse auction, i.e. Lowest Bid divided by the Bid.  This is mathematically incorrect and actually gives a non linear result, as shown below.  It makes it even harder for the buyer to know how the scoring mechanism would react.



Normalised questionnaire scoring:

We encountered this being used at a council who had applied rigorous OJEU mechanisms to every type of tender, of any value.  In their questionnaire scoring they took the participants’ results and normalised them.  So for example let’s say that the questionnaire is worth 50 points.  Participant A scored 5 out of 50 and Participant B scored 10 out of 50.  After normalisation Participant A gets 25 and Participant B gets 50.  Thus you can see that even if the participant does incredibly badly in the questionnaire, they can still get high points if the other participant has also scored badly.  From the actual points you can imagine how this could drastically change the tender.  In some ways it is saying that there is not room for improvement.  Imagine if the questionnaire was related to quality?  The normalisation could transform a low quality score into a very commendable one, which sadly masks the reality of the situation.  

Conclusion:

In summary the mathematics of a weighted tender are actually quite simple and we are proud of the straightforward linear methodology we have developed.  Not only do we believe it is easy to set up but you can also analyse what is going on during an auction.  This is exactly as it should be: fully auditable and compliant.

We are amazed by some of the current mechanisms used in the public sector today. At best they can distort the results in an unpredictable way and at worst are incorrect and non compliant.  Suppliers involved in these tenders might even have the ability to challenge the results which, if you think about it, could create quite a stir.

The biggest disadvantage of these mechanisms is that they are just plain confusing.  Any buyer setting up a tender using these techniques cannot fully grasp how the mechanisms will behave.  And if the buyer will have difficulty, just imagine what the suppliers will think.

13 comments:

  1. I think this is a useful and well-thought-through discussion. How can I save this as a document in my laptop for later reading / use?

    ReplyDelete
  2. Hi Ian. If you just drop a mail to info@marketdojo.com we can send you a word version. Otherwise you can simply copy and paste the text out of the blog.

    ReplyDelete
  3. Hello,

    Could you please explain why having a non-linear result is incorrect? My workplace has used 'lowest tender bid * 10 / tender price' to get a score, but we're looking at other options. However, i'd need to able to explain why the current method is no good.

    Thanks, Fiona

    ReplyDelete
    Replies
    1. Many thanks for your comment and I will try to reply as best I can..

      Firstly to say it is incorrect is perhaps overly harsh and subject to personal opinion. If you let the suppliers know what is happening then this is allowed in the public sector. However I can certainly explain why we do not believe this is good.

      The actual method of {Lowest Bid / Bid} gives a non linear result as mentioned above. This would mean it is much more difficult to understand how a mechanism would work in reality for the buyer and supplier. Different price decrements mean different scores at different prices.

      If you are going to use an inverse law then why not do 2* or 3* to create different non linear scoring mechanisms. In reality the only way to be confident in your scoring system is to keep it linear and logical and then it is clear for all concerned. Please give us a call if you would like to discuss further.

      Delete
    2. I think that there is a possible bit of confusion here..... The "incorrect linear" result is in fact the result of a proportional scoring mechanism.This is a popular scoring methodology in public sector and is never in practice called a "linear" scoring methodology. What usually happens is a buying authority will chose either proportional scoring (most popular as it is less harsh) OR linear scoring.

      The two are different and in my experience (until I read this article) are rarely confused with each other. Care should be taken when trying to explain the flaws of a scoring mechanism.

      Delete
    3. Hi Anonymous, Many thanks for your comment. There are two concerns noted in the article above.

      Firstly the confusion actually arises where we have been involved in tenders where they have said they are using a percentage of leading bid calculation and use - Lowest bid/ bid - which is simply not correct with respect to their naming.

      Secondly I would question the use of this type of non linear mechanism especially in actually understanding how a non linear mechanism reacts to various bidding scenarios. I also would question how many suppliers completely understand this mechanism during tender evaluation.

      Perhaps you can explain the how you would use this mechanism against the concerns highlighted? It is challenging to call this a proportional mechanism given it is unlikely to pass through '0'.

      Delete
  4. I am a bit confused. In the round 2 example should the score for Supplier B be 99.99 (98.99 +1). At present it shows with them with a lower score for price (as there tender is more expensive) but gives no benefit for the extra payment days valued at 1%. At the moment the example shows both suppliers with an overall score of 99.

    ReplyDelete
    Replies
    1. Hi, apologies we left out one important step in the calculation to go from 98.99% to an overall score of 99%. The 98.99% is the Price Score out of 100. What we then do is take into account how important the Price is in the overall tender evaluation. In this case we said that Price is 99% of the ratio and Payment Terms are 1% of the ratio. So we then multiply the Price Score of 98.99% by price weighting of 99% to get a weighted price score of 98%. We then add that 98% to the 1% from the Payment Term to get 99% overall for Supplier B.

      All this maths isn't easy to read but what it does translate to is that in order for Supplier A to present the client with a matching offer to Supplier B, who is offering better payment terms, Supplier A has had to reduce their quote by £4400.

      The client stipulated that better payment terms were worth 1% to them, and so the example proves (1% of £440000 = £4400), just through good, clean linear maths.

      Delete
  5. Thanks, that makes a bit more sense to me now. My only comment is that it is not easy to describe this methodology to potential bidders which could cause confusion and complaints. Whilst not ideal for the numerous reasons you have covered at least they do understand price being ranked against the lowest price. The trick is to have a good specification and product line description so that bidders use like for like products so they can be fairly compared. It is still possible to build in payment day scoring and anything else important to the client into the evaluation methodology. Nice ideas discussed above but I think we still need to find a method easy to understand and evaluate. I'll keep trying!

    ReplyDelete
  6. Many thanks for your comment. You are quite right that this particular analysis is rather complex. The client in question was trying to create a method to directly compare non price elements to price in a comprehensive manner. The method above does this rather well. With respect to telling the bidders, you would simply need to say which scoring method will be used and also the percentage weighting. You are quite right though that a defined specification and supplier liquidy are needed to make an eAuction a success. If you find any other simpler methods, we would be interested to hear. We had a grant from the TSB (Technology Strategy Board) and a part of it was to make the scoring in a weighted tender very linear and logical so it is easy for all to understand. We hope we have achieved this.

    ReplyDelete
  7. By assigning the pricing score to 100% of the pricing weighting are you not automatically "normalising" it. You have indicated this as bad practice for capability but are happy to do it to pricing

    Does this not mean that in a 50-50 capability-pricing tender you are actually weighting pricing higher (as 50/50 is achievable but very unlikely in capability)

    ReplyDelete
    Replies
    1. It is an interesting point.

      The best price is given 100%. If the price is the best in the tender then 100% would be a fair result. If you have reason to suspect that you can get a better price from another supplier, they should be involved in the tender. If the supplier that can offer a better price doesn’t take part, then they wouldn’t represent the best price.

      One could create a pricing system that I have heard suggested before where you pro rata the price between a qualification price and an artificially set minimum. However this would present its own set of challenges. Such as two suppliers getting the 100%.

      However, I have also suggested above that it would not be best practice to automatically set the best questionnaire score to 100%. This all comes back to how you have set up the event. In the case of the price, the best price, IS the best price. This does not apply to the perhaps more qualitative aspects of questionnaire scores.

      Let’s take a scenario where you have based the questionnaire on quality. And you have set the bounds of the quality. You have set what represents the worst and best quality. Then let’s say the best supplier gets 5/10, 50% of the total questionnaire score. This is entirely logical. If you say that the supplier gets full questionnaire marks then this makes a mockery of your scoring system and means you have different bounds on your quality score. If you were to re-tender the same category the following year, using the same criteria, and a new supplier scores 8/10, and the current supplier scores 5/10 (same as year before), how do you allocate the scores?

      Quite simply put, if you say to the suppliers that 50% of your marks come from the questionnaire, and this questionnaire is based on quality, and these are the bounds of the quality, you can not then say that a supplier with a poor quality gets full marks. If you do say then this would call into question your definition of how you set your SLA’s and what you actually want from the supplier and what you want your supplier chain to strive for.

      Also, suppliers are playing by the same linear rules and let’s not forget you can change the weighting between the questionnaire and pricing.

      Delete
  8. As brought to our attention by Peter Smith's from Spend Matters.

    He pointed out a technicality with the naming of our percentage of leading bid scoring system which we stated as 'the' percentage of the leading bid. A more apt description would be 'a' percentage of the leading bid. In other words the value comes from 'a' percentage of the leading bid but is not derived from dividing the total value from the leading bid itself.

    However, we both rigorously agreed that it should not be the 'lowest bid/bid' which is non linear.

    ReplyDelete