miserware only delivers a 2.7% power reduction

15/2/10 NOTE: this post has content that is seen as inaccurate by Miserware and you are advised to read all the text before forming conclusions to give Miserware a fair hearing.

I used a power meter to measure the electric draw of the PC using and not using miserware

I used a power meter to measure the electric draw of the PC using and not using miserware

Interested by some of Alan’s posts at the Open Sourcerer, I’ve been conducting an experiment in recent weeks with the Miserware power saving software that claims to save energy by better controlling CPU management.  I run a new (late 2009) pc with an Intel Core 2 Quad Q8200 (4x 2.33) CPU with a 450w PowerCool 80+, and have given it a go.  However, I didnt rely on the software’s own claimed power savings, I put an electric meter between the PC’s plug and the socket, and directly measured the power usage over 3 weeks, both with and without the software installed, set at my current domestic electricity rate pkwh.

Here is my data:

miserware installed? date £ of electricity used £ per day from last reading £ per day annual cost annual saving
no 19/01/10 14:00 0 pc booted up
no 20/01/10 18:00 0.16 0.1371
no 23/01/10 22:00 0.62 0.1453 0.1431 52.22 miserware installed
yes 24/01/10 08:00 0.67 0.1200
yes 26/01/10 19:00 1.02 0.1424
yes 30/01/10 08:43 1.53 0.1428
yes 08/02/10 08:33 2.81 0.1423 0.1418 51.77 Uninstalled 8 feb
no 13/02/10 05:12 3.53 0.1481 0.1481 54.07 2.3 test stopped

What I did was to run the PC without the software installed for 4 days from 19-23 Jan, installed miserware from 23 Jan to 8 Feb, then ran the PC again from 8 Feb until today.  Congratulations to the miserware support though, when I had issues installing they were right on the ball, and FOC.

The results are pretty unimpressive.  Despite a claimed 15% power saving claimed by the software (sudo grep Estimated energy /var/log/* )  it actually made very little difference at all.  In fact, if you average out the estimated annual usage for the 2 periods without running miserware ((52.22+54.07)/2) = 53.15 and compare that to the annual cost running miserware of £51.77, you see a 2.7% power saving thats worth about £2.30 a year.  OK, its a saving, but its not worth a lot to me.

The scary observation is that if you leave a PC on 24/7, its going to cost you over fifty quid a year at current online dual fuel electricity tariff rates!

  • http://www.mercianlabels.com Adrian

    This morning I received the following email from the CEO of Miserware:

    —– Original Message —–
    From: “Kirk Cameron”
    To: “adrian steele”
    Sent: Sunday, 14 February, 2010 05:10:24 GMT +00:00 GMT Britain, Ireland, Portugal
    Subject: From MiserWare RE: Your Recent Blog Posting

    Dear Dr. Steele,

    First of all, thank you for your interest in MiserWare. We recently
    read your blog entry entitled “miserware only delivers a 2.7% power
    reduction”. We are very interested in independent third party
    benchmarks of our software. However, power benchmarks are notoriously
    hard to make rigorous which is why our license agreement (see below)
    requires written approval of the methodology prior to publishing
    benchmarking results. We must respectfully ask you to temporarily
    remove your current benchmark posting and all associated links until
    receiving written approval for your independent testing methodology.

    We would like to work with you to facilitate a publishable,
    independent, third-party validation of our software. The goal of
    requiring written approval is not to influence the outcome of your
    tests, but to ensure that your methodology meets current industry
    standards and is accurate, documented, and repeatable. To this end, we
    could loan you more accurate power measurement equipment and help with
    their experimental use in data collection and benchmarking.

    We appreciate your passion for your blog and understand your desire to
    give your readers the most accurate information. We hope you will
    accept our invitation to work together to conduct an independent
    validation of the current beta software.

    Sincerely,

    Kirk

    Kirk W. Cameron, Ph.D.
    CEO
    MiserWare, Inc.

    —————————————
    Detailed Licensing Information

    Below is the full license agreement you signed on January 19, 2010.
    Please forgive the unicode copy/paste errors. The pertinent section to
    this discussion is Section 5b excerpted here:

    “(b) You are hereby authorized to disclose information regarding the
    performance of the Software, provided that such information is (i)
    provided by Licensor to You in a software report; or (ii) obtained
    using techniques approved in writing by the Licensor.”

    —-Begin Full License—-
    [full license SNIPPED by me for brevity but available on request]

    END

    First of all I would like to thank Kirk for taking some time to read the post. I genuinely was not aware of the content of the license agreement when I did the work and wrote the blog post, as I dont have a copy on email from when I signed up to try the software.

    I am in two minds as to why you want me to take down the post. It could be because there is a fundamental flaw in my methodology and it significantly misrepresents the software’s performance, and you make similar requests to everybody who posts feedback without getting approval in advance for the methodology, regardless of the outcome of the test. Alternatively it could be that your firm wishes to use a license clause to limit feedback on your product where you feel that such feedback could be perceived as negative to Miserware. I somehow doubt I would have received such a request if my results had shown a 40% power saving though!

    As scientists with PhDs we will both be aware of the importance of data overiding commercial interests, and being open and transparent about the methodology of experiments to make sure that they are not flawed and are reproduceable. I dont think we are going to disagree on this!

    I am quite happy to take down the content of the post Kirk if we can agree a replacement methodology. I would be pleased to rerun the experiment using a more accurate power meter and I will accept your offer to lend me one to use. I really like the idea of the your software and would like to contribute to its development if I can in a domestic environment. When this replacement methodology is agreed I will remove the content of this post or otherwise indicate that I have temporarily withdrawn it. I am struggling to see where the methodology is flawed though, with the exception of the fact that there are no doubt more accurate power meter available than what I used, but the comparitive data is pretty compelling. Please feel free to advise me on where the methodology is flawed with your expertise in this area.

    If the intention is to use your license clause to make me take down the post “temporarily” and not actually undertake any further work, then I have to balance my indicated agreement to the clause (which I was not explicitly aware of as like most people I dont read all the T&Cs of every webpage agreement I make – I dont have the time ) and moral obligation to comply with contracts, against the importance of allowing others who are interested in the subject to be aware of my data and conclusions, even if it is not positive towards your product.

    Kirk, please reply through this blog and I will publish your responses in full and unedited.

    Best Regards

    Adrian

  • http://www.mercianlabels.com Adrian

    Alan at the Open Sourcerer has made an observation that I didnt state before, which is the CPU is only one of the power consuming units ina PC, and depending on the hardware setup, if the CPU useage is small, even a big reduction in CPU power useage from using Miserware will have comparitively little effect on the total electricity draw (which I measured in this experiment), which what matters.

    You can read his post on this at http://www.theopensourcerer.com/2010/01/18/miserware-energy-saving-software-now-cross-platform/#comments

  • http://www.mercianlabels.com Adrian

    This is the reply from Kirk Cameron of Miserware today. My response is below.

    —– Original Message —–
    From: “Kirk Cameron”
    To: “adrian steele”
    Sent: Sunday, 14 February, 2010 21:18:14 GMT +00:00 GMT Britain, Ireland, Portugal
    Subject: From MiserWare RE: Your Recent Blog Posting (2nd request)

    Dear Adrian,

    Thank you for your prompt response. MiserWare software saves
    significant energy (up to 35%) while controlling the performance
    impact of power management. This is in direct contrast to other power
    management techniques (e.g. the PowerNow daemon) that can save energy,
    but do not allow user control over the performance impact. The actual
    savings achieved will depend upon your system characteristics and
    usage. Our software technologies are based upon a decade of
    peer-reviewed, award-winning research on improving server energy
    efficiency (see http://scape.cs.vt.edu/ ). Our company website
    (http://www.miserware.com) contains testimonials from several
    independent users that readily received approval for their evaluation
    methodology prior to publicly disclosing benchmark results. Our
    software has been independently validated privately by Fortune 500
    companies in the financial services, government, and media sectors. We
    have made the consumer version of our beta software free for personal
    use with the hope that hundreds and eventually thousands of people
    will use the software to reduce energy waste without noticeably
    affecting their productivity. The consumer version you installed
    reports CPU-only energy savings using techniques described in the FAQ
    accessible via your secure account and pasted at the end of this email
    for your perusal.

    We are happy to discuss the limitations with your technical evaluation
    of our software. Since our software was designed to save energy and
    minimize performance impact, any evaluation of MiserWare software
    should capture both energy use and performance of the system with and
    without our software running. As part of our freeware license
    agreement, we require evaluators to obtain written approval of their
    methodology prior to (not after) publishing their results since
    suspect results in either the positive (great energy savings, no
    performance loss) or in the negative (no energy savings plus
    performance loss) help neither party to truly evaluate the software. I
    hope that you will agree your results were actually positive (3%
    energy savings), so we are not trying to bury negative results in this
    case. Rather, we want to make sure evaluations of our software adhere
    to industry standards. For the average case, after thousands of
    benchmark runs on hundreds of systems, we commonly see 20-25% total
    system energy savings with limited to no performance impact. Our
    software matches the energy used to the load of the system which means
    savings will vary with system characteristics and usage. As to your
    blog posting and conclusions, your initial assessment is correct.
    Specifically,

    “there is a fundamental flaw in [your] methodology and it
    significantly misrepresents the software’s performance, and
    [MiserWare] make[s] similar requests to everybody who posts feedback
    without getting approval in advance for the methodology, regardless of
    the outcome of the test.”

    The industry standard for energy benchmarking is the SPEC benchmark.
    Strict adherence to the SPEC methodology would satisfy our
    requirements and result in written approval to publish. Among other
    things, this benchmark has a defined methodology to capture
    performance and energy use and a set of run rules that can be found
    here:

    http://www.spec.org/specpower/docs/SPECpower-Methodology.pdf
    http://www.spec.org/specpower/docs/SPECpower_ssj2008-Run_Reporting_Rules.html

    Your techniques violate a number of industry standard run rules.
    Firstly, you are not capturing the performance of your system during
    both the control runs (without MiserWare) and the other runs (with
    MiserWare). Heavy or light use of the system during such intervals can
    greatly affect the outcome of your test. For example, running an
    intensive graphics application without a high-end graphics card our
    software will maximize performance and the system will probably use
    maximum CPU energy while the app is running. During another period
    while the system is nearly idle our software will minimize energy use
    and may save significant energy. For these reasons, the load the
    system is under during benchmarking matters immensely and must be
    quantifiably measured to ensure fair comparison across samples.
    Subjective “interaction” or “user experience” is not typically
    acceptable as a performance metric though you probably didn’t notice
    any performance degradation when using the software. Keep in mind that
    turning your system off is the best way to save energy; MiserWare
    software is designed to minimize energy waste while your system is
    being used. We do not advocate leaving systems on 24/7 and hope you
    will shut down your system when it is not in use for the sake of the
    environment.

    Secondly, the duration of your tests (i.e. your sample data) does not
    appear to be based on standard statistical techniques that consider
    variance in the input variables thus giving no indication of the
    statistical significance of your conclusions. Variance can come from a
    variety of sources including the accuracy of your power meter, its
    sampling rate, etc. Were you capturing performance data, variance can
    come from measurements in the load variance on the system as well as
    the variance in the tools used to capture performance. This may
    explain why simply using a more accurate power meter could help, but
    is not sufficient to ensure your conclusions are statistically
    significant.

    I hope that you understand that we greatly appreciate the effort
    you’ve undertaken to benchmark our software and our offer to work with
    you to meet the SPEC requirements (or another methodology we can
    mutually agree to) is genuine. We believe in our software and have a
    decade worth of peer-reviewed quantitative data to support our claims
    as to its benefits.

    However, I am concerned you are in admitted violation of the end user
    license agreement you signed. Not reading the license agreement does
    not preclude your legal responsibility to adhere to the terms. In
    addition to the full agreement you signed digitally on January 19,
    2010, the top of the web page with the license agreement provided the
    following bulleted list to draw your attention to key portions of the
    agreement you were signing:

    * You may publish anything output by the program (Section 5.b)
    * You are allowed to release benchmark results with prior written
    approval of the techniques (Section 5.b)
    * You use the software at your own risk (Section 7)’

    Since you have publicly acknowledged violating the terms and
    conditions of the agreement you digitally signed on January 19, 2010,
    I would politely ask you once again to adhere to the terms and
    conditions you agreed to and respectfully ask you to immediately
    remove your blog posting entitled “miserware only delivers a 2.7%
    power reduction” as well as related links that contain benchmarking
    information since your techniques have not been approved in writing
    and may misrepresent the performance of our software.

    With our experience and expertise, we can work with you to reduce the
    time, effort and cost required to adhere to the SPEC standard
    methodology while still producing statistically significant,
    reproducible results. As I mentioned previously, we have no intention
    of influencing your conclusions only that the methodology is fair to
    all parties. This would allow you to both meet the moral obligation
    you have to your readers as well as operate under the terms and
    conditions of our end user license agreement.

    I appreciate your immediate attention to this matter.

    Sincerely,

    Kirk

    Kirk W. Cameron, Ph.D.
    CEO
    MiserWare, Inc.

    —-Begin Related MiserWare FAQ Entry—-

    How are the savings numbers computed?

    The savings reported by MicroMiser and ServerMiser ES are processor
    power numbers; the consumption and savings numbers are based on the
    models of processor power consumption we have created at MiserWare.
    Basically, we track how long you spend in each available power state
    and calculate savings by comparing against the default highest
    performance state for your system. Our techniques and assumptions are
    based on a decade of experience and validation using hardware power
    meters attached to servers, PCs, and laptops.

    The energy, dollar, and carbon savings numbers require some
    assumptions about the relationship between processor speed and
    processor voltage, and furthermore on some assumptions about your
    processor type and the power required to run it. In the current
    version of MicroMiser, the assumptions are very simple.

    * First, we assume that you are using a 90W processor, typical of
    many server and desktop processors available today, although laptops,
    some desktops, and now some servers may be equipped with lower power
    processors; if you know your processor power consumption, you can
    scale the results reported by MicroMiser appropriately to get a more
    accurate measure of your savings.

    * Second, we assume that your processor has a dynamic power range
    of 70%, meaning that 70% of the processor’s power is affected when
    processor speed is changed.

    * Finally , we assume that the dynamic power changes linearly with
    the processor speed, so that running at half the maximum speed would
    halve the dynamic power consumption of the processor.

    Here is an example. Let’s say your processor has two speeds, 2GHz and
    1GHz. Based on our first two assumptions, we assume your processor
    consumes 90W at 2GHz with a dynamic power of 90W * 70% = 63W, and a
    static power of 90W – 63W = 27W. Based on our third assumption, we
    estimate your processor power consumption at 1GHz to be 27W + (63W *
    (1GHz / 2GHz)) = 58.5W. So if your processor ran at its highest
    performance setting (2GHz) for an hour, it would consume 90W * 1 hour
    = 90Wh = .09kWh, and if it ran at 1GHz for an hour, it would consume
    58.5W * 1 hour = 58.5Wh = .0585kWh, with a savings of 0.09kWh -
    0.0585kWh = 0.0315kWh.

    For each time step, the energy savings are calculated in this way, and
    the savings are added up over time to yield the savings numbers
    reported by MiserWare software. The MicroMiser settings page allows
    you to set the assumption for the cost per kWh conversion to dollars
    (or in GBP). The conversion to carbon is based on standard techniques
    similar to those used by the U.S. Department of Energy. If your system
    has multiple physical processors (i.e. multiple CPUs in different
    sockets), MicroMiser reports an aggregated total savings achieved for
    all physical processors.

    Future versions of MiserWare software will include a more
    sophisticated power model that takes other factors into account when
    calculating power savings and consumption.

    —-End Related MiserWare FAQ Entry—-

    Thank you for this detailed response Kirk.

    I would like to clarify a few things:
    1) I did not notice any performance degredation whe I was using Miserware. I should have said this in my initial post, and I apologise for omitting to say this.
    2) I agree that the software did reduce my power usage, and therefore using it does derive a benefit. Agreed.
    3) I find it perfectly reasonable that others, with different usage profiles and hardware will see much bigger savings than what I experienced.
    4) I acknowledge that I may have inadvertently published data in contravention of the license agreement.

    The PC I use is on 24/7, as are many, many PCs. This is what I need to earn a living. Indeed Miserware will show a greater impact on PCs that run 24/7 than those that are on occasionally (obviously).

    I do not doubt the extensive intellectual input and effort that has gone into the product, but at the end of the day, the only thing that matters is how much electrical power demand is reduced for no (or an acceptable) reduction in performance to the end user. In this respect I stand by my data and approach. If on my hardware, when used by me in a way that suits my work pattern, there is very little benefit in power reduction, then essentially I dont see much of a benefit to me. This is the perspective of an end user who is not a specialist in the field. It is not a ‘like for like’ comparison against your, or the industry standard, methodology.

    I accept that my methodology does not meet your industry standard. However, I do not agree that because my testing methodology is not an industry standard then it is not valid. I also note that you have not indicated any specific flaw in my methodology that seeks to establish the total power saving other than it is not the industry standard. My methodology is comparative, and provided that the power meter behaved consistently throughout the tests then I cannt see a flaw in it. With more accurate metering a different result may be obtained. I also offer to give you exact details of my hardware profile so that you can reproduce the results in your own lab,

    The methodology I used and my data are clear for all to see and draw their own conclusions. If they think its rubbish, then I’m happy for others to form the view that the method and therefore the results are flawed. My simple approach does not set out to establish what the effect of Miserware is on CPU power usage, that is quite beyond my abilities. However, it does seek to establish what size of saving the software makes when used in combination with all the other variables in my unique working hardware/software/working pattern mix. It may well be that the software saves 15% of CPU usage, but if for example the CPU is only drawing 18% of the total power drawn into the system through the power meter, then that would give a 2.7% indicated total reduction.

    There may be other reasons for the result I got, which is why I’m not seeking to discredit your product, its a a great idea, its just that for me, it didnt produce much of a saving. I would like to help it its development if I can but I’m not about to enter into the world of research into CPU power usage.

    If you would like to propose a methodology that I can do from home, using my existing power meter or a meter you can lend me then I will happily run further experiments and publish the data. The burden here however does lie with you Kirk to propose a revised methodology that is practicable to do and measures the reduction on my electricity bill from using Miserware. As an end user, this is all I would be interested in.

    I propose to fulfil the “moral obligation [I] have to [my] readers” by operating in complete transparency and letting others draw their conclusions from the piece. I acknowledge that I may well be in breech of the user agreement (as I admit I didnt read it), but on balance I feel that the piece is fair and sufficiently clarified and caveated to enable others to judge the worth(or otherwise ) of the work. Your firm is being given unlimited right of reply to say anything you wish about it, and I will continue to do this. Honestly, I’m not out to attack your product just share my experiences!

    However, I am not prepared to take the piece down until a revised improved methodology is agreed between us to rerun an experiment that I can do and you are happy with, as I dont think its inaccurate or unfair, and it may be of interest to others. If I do agree to take it down now, then I can see circumstances whereby nothing further will happen from your side and the results will be permanently gone. I am sure that you dont want to be seen to try and enforce a contractual clause to supress real data that for some reason you dont want published (please can you explain why you want it removed?), especially as it is in fact quite positive about your software. The technical support was excellent, there was no noticeable change in performance, and it did deliver a power saving. You should be proud of it. But, for me, using my hardware and my methodology, it only delivered a small saving.

    In the long run Kirk, your product will only succeed if it gives a benefit that end users can see in a real world environment, and they will assess this in their user experience (as you correctly say) and in their electricity bill. That is all I measured. If you dont hear it from me, then somewhere down the track others will do just as I have, and indicate the extent of savings in the same manner.

    I await advice from you as to how I should rerun an experiment from home, using my power measuring kit or yours, to run a new experiment. If you could simply state for my benefit and others a simple protocol that we can do to test the efficiency of the software ourselves, then this is a great opportunity for you to allow many others to test out your software in the real world. The best improvement I can see would be to run for longer time periods? The equipment available is a willing operator, a PC with an OS that can support miserware, a power meter and a pen and paper to record results.

    I look forward to hearing from you Kirk and once again that you for your time in this exchange which I hope is proving useful for us both.

    Adrian