Software Performance


When discussing performance within auction software, we are discussing the speed with which the software performs certain auction-related things. In auction software, performance is an extremely important metric. Certain algorithms can be very slow if the software isn't written correctly. Algorithms may work quickly on few products and bidders, but may fail miserably as the number of bidders or products increases. Additionally, some algorithms, specifically the combinatorial clock auction, have the potential to never finish calculations.

Well written algorithms are crucial in auction software - a well-written algorithm that can take 3 seconds to finish its calculations can take 17 minutes to run if it's written only slightly less efficient. The results are exactly the same, but the time difference is a huge factor in the overall effectiveness of the software.

Optimal Auctions employs multiple software programmers, people who majored in computer science in college and are familiar with the best practices to run algorithms efficiently and quickly. The team is well versed in "Big O" theory, which measures how well algorithms scale when the number of inputs (bidders and products) increase. Additionally, our vast auction experience has shown us the best practices to ensure maximum performance in all our auction calculations.

By far the biggest advance in software in the past 2 years is the ability for algorithms to take advantage of all the resources available from the underlying hardware. Servers as recently as 4 years ago were single-core machines, where the processor can only do one thing at a time. Everything had to run "in serial", one-after-the-other, waiting for things to finish before it could start something else. Recently however, servers have become multi-core machines, with 4, 8, or even 16 cores. This means that the processor can do things "in parallel", running 4, 8, or 16 things at the same time.

The resulting possible benefits to auction software are tremendous - because many of the algorithms in auction software take the form of "for every bidder, do X" or "for every product, do Y", parallel processing can offer tremendous speed gains. Older servers would have to approach algoritms one bidder at a time, or one product at a time. Newer servers can instead look at the algorithm with an approach of 8 bidders at a time, or 8 products at a time. In effect, running things in parallel offers the ability to run algorithms 8 times faster!

Being a software company first and foremost, Optimal Auctions is at the screaming edge of computing performance. In 2013, we rewrote all our algorithms to run everything in parallel, taking full advantage of our underlying hardware. In one particular auction, as expected, we saw performance on the calculations go from 14 seconds to 2 seconds.

The beautiful part of this fundamental change in computing is that we are now positioned to take advantage of hardware improvements with no change to our software at all. In 2 years, when 16-core servers are common, our software will run even faster than it does today.

Specific Auction Metrics

Specifically for auction software, Optimal Auctions focuses on 3 parts of the auction when measuring our software's performance, and when comparing it to our competitors' software.

  1. How fast the auction opens in round 1
  2. How fast the auction calculates results each round
  3. The maximum number of bids that can be crunched in less than 10 minutes

All our performance tests were run on a cloud server with 8 cores and 16 GB of RAM

Open Round 1

In our auction software, when the auction opens many different things need to happen. The products need to be initialized with the proper prices and attributes, bidders need to have their data prepared for the first round, and all the auction rules and data need to be configured. This metric is a good first measurement, and also a good test of the database performance.

Here are our metrics from our performance testing on our latest release.

Open Round Performance

As the test results show, even with a very large auction of 1,000 bidders and 1,000 products, the auction opens in 2.2 seconds, which is extremely quick.

Calculation of Results Each Round

Each round, the results of that round are calculated and winners are determined. These steps involve analyzing each bid on each product, and calculating winners and next round prices. It also places any bids for bidders who did not bid in a round. It's a good measurement for what happens every round in a typical auction.

Here are our metrics from our performance testing in our latest release.

Calculation Performance

The test results show amazing performance at crunching numbers quickly. Even more importantly, the numbers stay excellent if you increase the number of bidders, or increase the number of products. The time is nearly the same when the auction has 100 bidders and 100 products as it does when it has 10 products and 1,000 bidders. These are indicative of a well-designed algorithm, when the time increases slowly as the inputs into the algorithm increase rapidly. This is termed "a scalable algorithm", since it's proven to grow reasonably within the expected parameters of a standard auction.

Maximum Number of Bids in 10 Minutes

Finally, the last performance test we like to run is to figure out how many bids can be crunched in 10 minutes. Why 10 minutes? It may seem like a made-up number, but it was actually the required maximum calculating time in early auctions, so we continue to use it internally as our preferred maximum throughput time measurement.

As a sidenote, back in 2003 when we first started measuring the maximum number of bids the algorithm could work through in 10 minutes, the number was 400. 40 bidders and 10 products would have hit the limit back then, so performance was a legitimate concern.

Because our calculations in step 2 showed that our algorithm doesn't grow horizontally, but instead grows linearly, we can extrapolate out the maximum number of bids possible in 10 minutes using our existing data.

10 Minute Maximum Bids

Our performance testing indicates that our software can handle a little over 350,000 bids per round. This can be broken down between bidders and products, so for example you could have an auction with 350 products and 1,000 bidders, or an auction with 2,000 products and 175 bidders. Obviously, just based on those examples, those are extreme auctions and well beyond what a client's auction will likely entail. It's beyond anything we've run in our history. But, it should be reassuring to know that Optmal Auctions software can easily handle any auction you may have.

If our competitors think their software can measure up to the blazing speed of our software, then let's see their numbers.

We issue a challenge to all of our competitors - Show the world your performance metrics like we have!