My Barrett IO-540/EFII Dyno testing

What REALLY happened at Barrett.
Just the facts. My opinions are just that and spelled out accordingly....
Now that the details of the tests are out, I wanted to do a write up about my Barrett Engine/Dyno/EFII testing and break in we did back in June. My other write-up got sidetracked onto the tests. If you subscribe to Kitplanes, which most of you do, it’s in the latest issue (Nov 2014)

-First, a quick backstory on how I ended up where I was (and am). It’s obviously no surprise that I ended up choosing Barrett Precision to build my engine, so that doesn’t need any explanation. Plus, they are 95 miles from where we are building the plane, so it was nice to be able to see the shop prior to our purchase and pick the engine up ourselves.
Literally the engine was decided from day one. That left me trying to figure out the ignition system. This was a task I did not take lightly. First, as I have only been a pilot for a couple years, I researched magnetos and their technology, went and talked to my local A&P about the service and maintenance of them. Armed with that info I started seeing what else was out there.
I came to realize, there were not that many other options:
Mags: Pros and Cons… in my mind, I wasn’t going to put 100 year old technology in a 2015 plane. Just My Opinion. Yes, they are self-powered, but that’s the only thing going for them.
P-Mags, not an option for a 6 cylinder engines then or now (as far as I know, plus, seems like a weird way to do it)

-That leaves Electronic Ignition: Which the only real negative is the need to have an external power supply to fire it. Which, to be honest, with current battery, alternator and electrical technology, is a pretty marginalized factor. I had already planned on having a redundant electrical system, so not much to change. My current system has 2 PC680s and a single 70amp Alternator. I may add a backup alternator later after some testing. At the moment I don’t think I will need it. I designed my system with an “Essential Bus” switch, which is actually a LOAD SHEDDING switch. Should there be any sort of electrical issue, one flip of the switch and I can drop down to one screen, one com, one ECU etc. (the particulars are still a work in progress with much consulting with people above my pay grade)

-I looked at Lightspeed. I called Allen Barrett and asked his 2 cents. When he told me they had issues with Lightspeed ignitions and refused to work with them, I of course became worried. So, I called Klause. Let’s just say the conversation did not leave me feeling warm and fuzzy.
-More research brought me to EFII. I liked what I read. Again, a call to the MAN building my engine. Allen said they done one or two EFII installations on 4 cyl engines, liked the product and had a few more in production including a 540 like mine. All the research seemed to add up. I found that EFII/Protek Performance were just a short hop up from me in San Diego to Cable Airport. I decided, it’s an important decision, I want to see for myself the product, the facility, the people behind the scenes…
That info is spelled out in this write up:
http://www.vansairforce.com/community/showthread.php?t=108661

I had made up my mind on the ignitions system and had planned on going with the standard Airflow Performance mechanical fuel injection. (Don’t get me wrong, Airflow Performance makes a very nice product and Don is super knowledgeable and friendly, in fact he spend about 30 minutes on the phone explaining the basics of fuel injection 101 to me last year…) However, once I learned that already having the ECU put me half way there, it wasn’t hard to realize the benefits of a full EFII set up. Mainly, not having to deal with the mixture. LESS PILOT WORKLOAD is always nice especially in IMC, in which this plane will surely end up. The options you have for mapping the fuel curve are great. You can get max horsepower when you want it, lean in cruise and have the mixture return, all automatically without having to do anything. No pilot error of flying into detonation…

So… that’s the decision factors FOR ME… your results and findings may vary…


At Barrett for the testing on the Dyno. As it turns out, Allen for some reason was having a heck of a time getting their stock Bendix fuel servo and spider to run properly. I flew out on a Monday night, with Tuesday supposed to be just playing with the EFII set up before packing it up and taking it home. Turns out they spent part of the weekend and most of the day Monday just getting the Bendix set up going.
I arrived at Barrett Tuesday morning to see and hear my engine running (On Mags and Mech Fule Inj), it sounded great. It was a good crew, Allen and Monty Barrett, Robert Paisley from EFII and Dan Horton to do the write-up for Kitplanes, and my beautiful custom painted freshly built IO-540. I was like a kid in a candy store.
So, Robert was primarily there to get BPE up to speed on his system and leave them with a set up so when clients wanted an EFII set up, they could actually install it and dyno it there. Leaving the ECU and a few other parts to stay permanently at Barrett. Well, due to the lack of time available for Robert to catch a plane, and me and my in-laws (Shayne and Phyllis McDaniel, this is plane #4 for them, definitely repeat offenders) set to leave with the engine the next day, we had a very short time to rip one system off and get the next one installed. Literally a few hours. We all pitched in, literally about 5 of us doing whatever we could to get it done.
We got the EFII system installed, however, for the fuel system, we (as a team) made a big no-no. Apparently, due to the set up of fuel flow calculations on the Barrett dyno, returning the fuel from the fuel rail back to the tank wasn’t an option. So, since we didn’t have time to fab up new hoses and modify the system, we had to return the fuel into the system just before the pump. (side note: if you didn’t know, in an electronic fuel injection system, much more fuel is sent to the injectors, with much more unused needing to return back to the tank. It’s literally a loop, just like in a car). So, we actually ended up setting the system up NOT ACCORDING TO THE MANUFACTURERS GUIDELINES…. Robert had just given us a lesson on how NOT to do that earlier that morning, but it was the only option to get the system running in a short time frame. WE didn’t really over think it, it was just what needed to happen to get the engine running.

The results:
The big one that we were all watching was of course the WOT HP reading. It showed a slight DECREASE in HP with the EFII set up? Which left more than myself scratching our head going ?That just doesn?t seem possible?. Throughout most of the other data points showed similar HP to the Mags/Mechanical set up, but better fuel economy.
What I did notice (I shot video of it running on EFII, but I didn?t shoot video during the first run on Mags) was that the engine ran noticeably smoother on EFII. And I do mean both visually and audibly. The first run, the exhaust housing above the engine had a tiny shimy, on the EFII, nothing, not a flinch, smooth like butter, and it started quickly and easily.

So, it was getting late, we shut the engine down, having collected the data and everybody went off for the night. The next morning, we arrived at the shop to dismantle everything that had to come off the engine and prep it for travel in the truck to it?s new home. That?s when we got an email and call from Robert, now in Texas at Titan to perform a similar test. He pointed out the flaw in the set up and that that had to explain the system not performing properly. At that point, we couldn?t do it again as the engine was mostly packed up.

-Here?s my 2 cents, and that?s about what it?s worth so take it for that. But it is my engine, and I was there.

Robert?s explanation sure seemed to make sense. Hot bubbly fuel getting pumped back into the system is a bad thing? everybody knows that. So, in my mind the results of THIS test will always have an * by them. The EFII set up was A: not set up properly, B, set up quickly and not tuned, vs the Bendix system that had 2 full days getting set up?I wish we had done a third test with the mechanical fuel injection and the EFII ignition to just get an ignition apples to apples comparison.

Plus, I personally have spoken to about 6 EFII owners that were flying before switching over. Some just ignition, some both ignition and fuel. In EVERY CASE, they reported improvements. Not only in actual numbers (Faster TAS, GS and better fuel economy) but in literally being able to FEEL the difference in their plane. Including Danny Riggs with a Barrett IO-540 saying ?it feels like it?s on speed now!!? Which led me to believe that our results had to have been somewhat flawed. (again, the engine is producing PLENTY of HP either way, so I?m not too worried about it)

To Cap it all off, Robert, followed up with me two days later after his tests at Titan on an IO 360. At Titan?s dyno they had different fuel set up so he was able to plumb the system correctly. The EFII system performed as expected, increase in HP and Torque, better fuel economy.
All of the above leads me to believe that our test results got scewed, We?ll never know for sure, unless we do it again. (Which we will?..stay tuned)

Love the facts......thanks

Bluelable,

I also read the article Dan wrote in Kitplanes and found it most odd. I run engines everyday here at Titan and our experience is that EFii is everything Robert claims both on the dyno and in my aircraft. I have done direct comparison with Efii and traditional injection and electronic ignition and there is really no comparison. Max power is only slightly higher on Efii but every other aspect of operation is by far better, smoother, and more efficient.

I have raced the system at Reno and installed it on everything from 340, 360, 370, 409, GTSIO-520 race engine for my NXT. We are finishing up our R540 and I can't wait to run it based on my experience with getting the 6 cylinder Continental running.

Thanks again for setting the story straight...

Forgot the best part

Relentless said:

Bluelable,

I also read the article Dan wrote in Kitplanes and found it most odd. I run engines everyday here at Titan and our experience is that EFii is everything Robert claims both on the dyno and in my aircraft. I have done direct comparison with Efii and traditional injection and electronic ignition and there is really no comparison. Max power is only slightly higher on Efii but every other aspect of operation is by far better, smoother, and more efficient.

I have raced the system at Reno and installed it on everything from 340, 360, 370, 409, GTSIO-520 race engine for my NXT. We are finishing up our R540 and I can't wait to run it based on my experience with getting the 6 cylinder Continental running.

Thanks again for setting the story straight...

Click to expand...

What wasn't "straight" about the story in Kitplanes?

I find it hard to challenge the results of a test where the rules were clear and everyone involved accepted the conditions beforehand.

Well, I guess we saw this coming.

It didn't make sense to me that the EFII would not match the mag system in horsepower. One would assume similar max HP but more efficiency with the EFII. Looks like the EFII System was not getting enough fuel flow. The mag system, however, was using an RSA-10 servo rather than a stock RSA-5 and this allows more airflow and more fuel flow.

However, extraordinary claims require extraordinary proof. The ball is in the EFII court.

Kyle Boatright said:

What wasn't "straight" about the story in Kitplanes?

I find it hard to challenge the results of a test where the rules were clear and everyone involved accepted the conditions beforehand.

Click to expand...

Kyle,
I think what Kevin is referring to is the set up story. I was just relaying all the facts. Which were, we all had plane tickets and deadlines. Robert and I flying in from parts of Southern California, Dan coming in from Alabama, and my in-laws coming from Mo. As you can imagine, just coordinating it with all of us was a pain to begin with. We all had schedules. The "plan", as best I know, was for the engine to be running on Mags/Mech Monday morning, do the runs, collect data, then Monday afternoon switch over to the EFII set up, then have Tuesday to tune the system, train Barrett and us, then Wed break it down to take home. Well, as you can read, that didn't happen. It turned more into a reality show type deadline racing the clock. The EFII system literally got slapped on, ran for about 5 minutes before the runs to take data were done. Hence the improper set up.
That's what I think Kevin means by "straight".....

vlittle said:

Well, I guess we saw this coming.

It didn't make sense to me that the EFII would not match the mag system in horsepower. One would assume similar max HP but more efficiency with the EFII. Looks like the EFII System was not getting enough fuel flow. The mag system, however, was using an RSA-10 servo rather than a stock RSA-5 and this allows more airflow and more fuel flow.

However, extraordinary claims require extraordinary proof. The ball is in the EFII court.

Click to expand...

Comparative dyno testing is best done by changing only one variable at a time (the scientific method) to judge relative change. Having a smaller servo in place on one test pretty much does not allow one to draw valid conclusions about anything. I am assuming here that the -10 has a larger throttle plate than the -5 but couldn't find any specs.

Increased airflow matched with increased fuel flow usually equals more power.

Bluelabel already covered the return plumbing mod which was certainly not a good idea if one wants to make valid comparisons. Properly plumbed, the EFII system has vast reserves of fuel available at the port (like enough for over 500 hp).

Repeat the test with the same servo and proper fuel system plumbing. Let the results tell the story then.

One factor at a time....uggggg.

OFAT (scientific method) is 19th /20th century stuff long abandoned by top quality experts. Today it's a DOE (design of experiment) set up as a fractional factorial or if money and time allows, a full factorial experiment. Either way, you learn so much more besides which variables are significant, but also if there are interactions between and within variables. All this while in a shorter timeframe and lower cost. (Interactions is really good info, and pretty common. OFAT will make you go broke before you get enough information about interactions)

Either way, all this publicity should be good for all parties assuming we can fail to reject the null hypothesis. (Null would be EFII is better, . )

Cj

rv6ejguy said:

Comparative dyno testing is best done by changing only one variable at a time (the scientific method) to judge relative change. Having a smaller servo in place on one test pretty much does not allow one to draw valid conclusions about anything. I am assuming here that the -10 has a larger throttle plate than the -5 but couldn't find any specs.

Increased airflow matched with increased fuel flow usually equals more power.

Bluelabel already covered the return plumbing mod which was certainly not a good idea if one wants to make valid comparisons. Properly plumbed, the EFII system has vast reserves of fuel available at the port (like enough for over 500 hp).

Repeat the test with the same servo and proper fuel system plumbing. Let the results tell the story then.

Click to expand...

Kyle Boatright said:

What wasn't "straight" about the story in Kitplanes?

I find it hard to challenge the results of a test where the rules were clear and everyone involved accepted the conditions beforehand.

Click to expand...

Interesting use of word "Rules". I thought the objective was to fairly compare and then put out to the community solid information, not to disprove a new technology. We know traditional systems work great, but to think nothing new can be developed is silly. Imagine if anything else you have purchased recently would have stopped being developed 25 years ago.

Imagine general aviation without Vans and the experimental community. Ist time we put the same enthusiasm into engine development as we have put into airframes and avionics. Oh, I am....

grubbat said:

OFAT (scientific method) is 19th /20th century stuff long abandoned by top quality experts. Today it's a DOE (design of experiment) set up as a fractional factorial or if money and time allows, a full factorial experiment. Either way, you learn so much more besides which variables are significant, but also if there are interactions between and within variables. All this while in a shorter timeframe and lower cost. (Interactions is really good info, and pretty common. OFAT will make you go broke before you get enough information about interactions)

Either way, all this publicity should be good for all parties assuming we can fail to reject the null hypothesis. (Null would be EFII is better, . )

Cj

Click to expand...

I am not sure what you are saying here at all. Let's say we make a camshaft change and an intake manifold change simultaneously. The cam change gains us 5% hp and the intake loses 5%. We assume neither did any good so we discard and move on to something else- lost opportunity to learn. By changing only one variable, we can determine the relative change and build from there. Time consuming, yes, but still the way I and most others do it on the flow bench or dyno.

Wind tunnel and flight testing still done this way too to my knowledge. Hardly obsolete.

Comparison

Kyle Boatright said:

What wasn't "straight" about the story in Kitplanes?

I find it hard to challenge the results of a test where the rules were clear and everyone involved accepted the conditions beforehand.

Click to expand...

From the way I read this thread, it appears that due to a testing equipment limitation the EFII system was not able to be properly plumbed. The manufacturer is specific about the need to have the return fuel returned to the fuel tank with a minimum seperation of four inches from the pick up port. This happens in the fuel tank before the fuel enters the fuel pump. the purpose of this limitation is to remove air bubbles and allow unheated fuel into the system. Neither a fuel pump nor an injector can be expected to perform effectively without a clean cool fuel supply.

EFII testing

At least we are back onto a useful conversation about this!

What no one expected at the Barrett test was that the majority of the time would be spent trying to make the Bendix injection and the mags work. Every part of these systems was either replaced or overhauled (mags) during the effort. And believe me it was trying and tedious to all.

In the end, there wasn't time to properly hook up the EFII system or to tune it. This is not a disaster by any means - welcome to testing. Processes rarely work perfectly on the first run.

After the fact, my request was to complete the test properly and continue the original goal of getting the Barrett facility set up to run modern engine controls. This is still my goal. It was unfortunate that a magazine article was thrown together to cover the results of an incomplete test - this didn't help builders or the magazine.

The bright side is that the same test was completed a couple days later at Titan on their Superflow dyno and the results were as expected. A nice smooth running engine that makes a little more hp. Not much of a surprise to those who operate modern vehicles.

LOTS more dyno results will be floating around soon as more testing completes.

Many thanks to all of our customers who have helped us get this far. We couldn't be working with a better group of people.

Robert

I really hated to see this article in Kitplanes. Just starting to read it I could see there were too many variables introduced to come to any scientifically valid conclusions. It didn't do the EFII products or the Bendix any favors as they were not able to set the test bed up to factory specs.
It sure would be nice to repeat the test when everything is properly setup.
That said I have the ignition part of the EFII. It starts a lot easier, especially when hot. Runs really smooth (of course the Barrett IO-540 always runs really smooth) and doesn't have to be rebuilt every 500 hours.

OFAT and DOE

Ross,
I don't want to hijack the thread nor do I want to offend you or your OFAT testing method. I apologize for both in advance.

If you take a step back and look at the testing process, the example you gave illustrated two variables. A camshaft change and an intake manifold change. However, in real life, those are not the only two variables. They may be the only two variables that we wish to acknowledge, but they are certainly not the only two variables. If i were to ponder for the next 30 seconds on this, I would add the following variables: Air temp, operator, fuel temp, run time, cam temp, oil temp, gasket thickness, valve spring rate, valve spring thickness, cam duration, cam lift, cam hardness, intake air temp, exhaust temp, intake hose length, intake hose diameter, etc. And I am only scratching the surface. The fact is that there are literially tons of input variables that can affect the output. So, the question is, which input variable is significant. Also, which input variable has an interaction going on? Also, how significant is the interaction? A person who is used to doing OFAT would choke at this point because there is simply too many variables. So the OFAT experiment has to limit stuff and make assumptions to eliminate a lot of these variables. Eliminating these variables reduces the knowledge you gain and your degrees of freedom. Not good. There are things that can be done that can reduce the input variables down to the critical few but a DOE is very comfortable with multiple variables.
Research MOEN on his book on Design of Experiments. Its a good read, if you like engineering stuff and this sort of thing.

Let me close with this example. In the IC world, the quality of the chips were never getting to the level that they needed. Several DOE experiments were conducted over time and one of the critical input variables identified was, well, the weather. WELL, everybody knows you can't control the weather so the old-timers laughed at the idea that weather, since it was a critical factor, should somehow be controlled. hahaha, Who thinks they can control the weather....hahahah. Well, someone did. They build a large building with strict temp controls and started mfg IC's and the reliability level when off the charts. It is true that some variables are uncontrollable, and so robustness has to be built into the design so that this variable does not have any significant impact to our desire output. But, hey, we learn something and become smarter and hopefully make a more competitive product.

Knowledge of input variables, no matter how many there are, and the determination of which are significant, which should be controlled, and the interactions between each is stuff that the smart folks are doing today. They are learning more at a lower cost, and at a faster rate with DOE's.

I doubt that anyone will abandon OFAT experimentation and progress to DOE based on my feedback. However, I trust that folks who are making innovation and progress in our aviation world, will at least open an ear and see what others have already learned in the Quality world. I for one am thrilled at the innovation we have seen not only on VAF, but in the experimental sector.

Believe it or not, it is this sector that is now leading the industry.
cj

rv6ejguy said:

I am not sure what you are saying here at all. Let's say we make a camshaft change and an intake manifold change simultaneously. The cam change gains us 5% hp and the intake loses 5%. We assume neither did any good so we discard and move on to something else. By changing one variable, we can determine the relative change and build from there. Time consuming, yes, but still the way I and most others do it on the flow bench or dyno.

Wind tunnel and flight testing still done this way too to my knowledge. Hardly obsolete.

Click to expand...

Barrett IO540-X with Dual EFII Ignition

I can attest to the smoothness of the engine with the Dual EFII ignition and how quickly the engine starts with this system. I too have a Barrett built 540 that was on their dyno and outfitted with EFII Ignition system. On that occasion a couple of issues got in the way of a successful full power run but it's my understanding it had to do with the loading of the dyno and the mapping of the timing curve, hence the timing should have been disabled. Allen and Robert discussed this and obviously got it figured out since John's engine was tested at full power.

I can't speak to the injection portion of the kit but as mentioned before with Dual ECUs you are already half way there.

Gotta go with Ross on this one.............

grubbat said:

Ross,-----snip----- the example you gave illustrated two variables. A camshaft change and an intake manifold change. However, in real life, those are not the only two variables. They may be the only two variables that we wish to acknowledge, but they are certainly not the only two variables. If i were to ponder for the next 30 seconds on this, I would add the following variables: Air temp, operator, fuel temp, run time, cam temp, oil temp, gasket thickness, valve spring rate, valve spring thickness, cam duration, cam lift, cam hardness, intake air temp, exhaust temp, intake hose length, intake hose diameter, etc.

Click to expand...

Ross did not talk about changing all the possible variables you mention, only two variables to illustrate how making multiple changes at the same time can created useless results.

All the other variables you mention would not be a factor--(ie, valve spring rate/thickness--as the springs are not changed) in his example as they are not effected with a cam or manifold change, or are factors that can be controlled/corrected for in the actual testing--(air or fuel temp etc). Or, as in the cam lift/duration---well that is what has been changed in the test Ross is using as an example.

Anyhow, long winded way to say despite the number of variables, to see what the result of changing any one of them is, you should change only that thing, then test against baseline. Then change something else and test.

In the example given where one change helps, and the other hurts for a net zero change----how will you ever know what each one of them did???

And, yes cam and intake changes are quite dependent on each other and often will not give optimum results unless changed together as a "system".

grubbat said:

Ross,
I don't want to hijack the thread nor do I want to offend you or your OFAT testing method. I apologize for both in advance.

If you take a step back and look at the testing process, the example you gave illustrated two variables. A camshaft change and an intake manifold change. However, in real life, those are not the only two variables. They may be the only two variables that we wish to acknowledge, but they are certainly not the only two variables. If i were to ponder for the next 30 seconds on this, I would add the following variables: Air temp, operator, fuel temp, run time, cam temp, oil temp, gasket thickness, valve spring rate, valve spring thickness, cam duration, cam lift, cam hardness, intake air temp, exhaust temp, intake hose length, intake hose diameter, etc. And I am only scratching the surface. The fact is that there are literially tons of input variables that can affect the output. So, the question is, which input variable is significant. Also, which input variable has an interaction going on? Also, how significant is the interaction? A person who is used to doing OFAT would choke at this point because there is simply too many variables. So the OFAT experiment has to limit stuff and make assumptions to eliminate a lot of these variables. Eliminating these variables reduces the knowledge you gain and your degrees of freedom. Not good. There are things that can be done that can reduce the input variables down to the critical few but a DOE is very comfortable with multiple variables.
Research MOEN on his book on Design of Experiments. Its a good read, if you like engineering stuff and this sort of thing.

Let me close with this example. In the IC world, the quality of the chips were never getting to the level that they needed. Several DOE experiments were conducted over time and one of the critical input variables identified was, well, the weather. WELL, everybody knows you can't control the weather so the old-timers laughed at the idea that weather, since it was a critical factor, should somehow be controlled. hahaha, Who thinks they can control the weather....hahahah. Well, someone did. They build a large building with strict temp controls and started mfg IC's and the reliability level when off the charts. It is true that some variables are uncontrollable, and so robustness has to be built into the design so that this variable does not have any significant impact to our desire output. But, hey, we learn something and become smarter and hopefully make a more competitive product.

Knowledge of input variables, no matter how many there are, and the determination of which are significant, which should be controlled, and the interactions between each is stuff that the smart folks are doing today. They are learning more at a lower cost, and at a faster rate with DOE's.

I doubt that anyone will abandon OFAT experimentation and progress to DOE based on my feedback. However, I trust that folks who are making innovation and progress in our aviation world, will at least open an ear and see what others have already learned in the Quality world. I for one am thrilled at the innovation we have seen not only on VAF, but in the experimental sector.

Believe it or not, it is this sector that is now leading the industry.
cj

Click to expand...

Mike pretty well sums things that many of the things you listed here would not change hp output in the first place. I'll step in and add a few things based on 30+ years of doing this stuff for a living. First, QC and process control are vastly different things than dyno work. Secondly, I've learned that despite 30 years in the field, the interrelated things in engine performance are generally too complex to simplify and predict in an engine development program and you will often be wrong with a prediction. This is why we do actual testing- because we are always going to learn something.

The dyno room purposely controls the test environment and applies correction factors for almost all known and significant factors which would affect the results such as barometric pressure, IAT and humidity. A good operator will also do several baseline pulls to get a reliable average and he'll do all subsequent pulls at similar CHTs and oil temps to reduce experimental error to a minimum. This is how the industry does it and it's the best way to get the best accuracy. I think you'd find another 10,000 dyno operators worldwide who wouldn't agree with you on your view here and I doubt if all those guys are wrong...

Accurate baseline
One change
Observe result
Compare
Next change
Repeat

True

Hi Mike,
I was using the other variables just as an example. The point I was trying to make is that we think we know what the variables are and sometimes we do. However, a OFAT experiment greatly limits your knowledge gained because we just do not have the time and money to investigate all the corners of the frame in question. In the past, designers / innovators just had to choose through experience or past knowledge what they thought might be the significant drivers. Sometimes they were right. Sometimes, after a lot of experiments, time, and money, they learned something. Changing one factor at a time now a-days is just simply too costly. OFAT is still taught in some schools because it is easily understood. However, the top tier schools, NASCAR folks, and top quality institutions are utilizing/teaching Design of Experiments (DOE).

I was teaching this stuff back in 2000. I had a couple of great Quality mentors from Allied Signal that showed me the light back then. At the time, we were only a handful of folks (in the world) who knew about AND actually practiced DOE's. It's really can be a useful tool.

In real life, interaction within and between variables is real and very common and sometimes nonlinear. Try figuring out a non-lineral interaction with an OFAT experimentation. Gives me heart burn just thinking about it.

As far as how do you know which factor or variable is significant, which one isn't, which one is interacting with another one, and what variable that you thought was significant when in reality isn't so much, well, that's why the DOE was born. Software such as Minitab makes it easier than slide rule.

cj

P.s. Ross, you are correct that most people do not use it. However, the smart ones do. Of course, when you are talking about running a multi-million dollar engine in a test cell, time is money. (GE did it, Allied did, and others also. To compete, you have to.)

Mike S said:

Ross did not talk about changing all the possible variables you mention, only two variables to illustrate how making multiple changes at the same time can created useless results.

All the other variables you mention would not be a factor--(ie, valve spring rate/thickness--as the springs are not changed) in his example as they are not effected with a cam or manifold change, or are factors that can be controlled/corrected for in the actual testing--(air or fuel temp etc). Or, as in the cam lift/duration---well that is what has been changed in the test Ross is using as an example.

Anyhow, long winded way to say despite the number of variables, to see what the result of changing any one of them is, you should change only that thing, then test against baseline. Then change something else and test.

In the example given where one change helps, and the other hurts for a net zero change----how will you ever know what each one of them did???

And, yes cam and intake changes are quite dependent on each other and often will not give optimum results unless changed together as a "system".

Click to expand...

Did I miss the horse power number in this thread?

grubbat said:

Hi Mike,
I was using the other variables just as an example. The point I was trying to make is that we think we know what the variables are and sometimes we do. However, a OFAT experiment greatly limits your knowledge gained because we just do not have the time and money to investigate all the corners of the frame in question. In the past, designers / innovators just had to choose through experience or past knowledge what they thought might be the significant drivers. Sometimes they were right. Sometimes, after a lot of experiments, time, and money, they learned something. Changing one factor at a time now a-days is just simply too costly. OFAT is still taught in some schools because it is easily understood. However, the top tier schools, NASCAR folks, and top quality institutions are utilizing/teaching Design of Experiments (DOE).

I was teaching this stuff back in 2000. I had a couple of great Quality mentors from Allied Signal that showed me the light back then. At the time, we were only a handful of folks (in the world) who knew about AND actually practiced DOE's. It's really can be a useful tool.

In real life, interaction within and between variables is real and very common and sometimes nonlinear. Try figuring out a non-lineral interaction with an OFAT experimentation. Gives me heart burn just thinking about it.

As far as how do you know which factor or variable is significant, which one isn't, which one is interacting with another one, and what variable that you thought was significant when in reality isn't so much, well, that's why the DOE was born. Software such as Minitab makes it easier than slide rule.

cj

P.s. Ross, you are correct that most people do not use it. However, the smart ones do. Of course, when you are talking about running a multi-million dollar engine in a test cell, time is money. (GE did it, Allied did, and others also. To compete, you have to.)

Click to expand...

Engine development projects are time consuming by nature and dyno testing can become a tedious but rewarding learning experience as part of that. In this case, the point was to quantify the result of a SINGLE change. Your theory simply has no merit as a time saver in this instance.

We already well know what the external variables are for about the last 100 years. They are controlled for the most part. What we don't know is the effect of the mechanical changes we make to the system.

We have computer tools to predict lots of things these days- dyno in a PC, CFD to design aircraft and gas turbine engines etc. These are big time savers but in the end, we still validate the design through actual testing. In the case of big projects like new aircraft or gas turbines, this still involves thousands of hours. Simulation simply isn't a substitute for real world testing and it's not accepted by the powers that be like the FAA. We still see failures in the most heavily engineered devices like the PW F135 and PW1500G engines recently.

For something like the tests we are talking about in this thread, it is way way cheaper to simply put the engine on the dyno, baseline, make the single change to the EFII system and observe/ report the results. Your posts, while food for thought perhaps in other fields and endeavors, don't make much sense in this case where a few hours would have given a solid answer.

tadsargent said:

Did I miss the horse power number in this thread?

Click to expand...

I think it was lost forever...

tadsargent said:

Did I miss the horse power number in this thread?

Click to expand...

Tad, it was 276 vs 265. All this argy-bargy and dog-ate-my-homework over a whopping 4%.

Never mind that I also reported 194hp @16.3 GPH on mags and Bendix, vs 190hp @14.8 GPH with the EFii systems, both at 24/2400. That's 2% hp for 10% less fuel, at a high cruise power. If you could claim that, would you make a fuss and draw attention to the other?

rv6ejguy said:

For something like the tests we are talking about in this thread, it is way way cheaper to simply put the engine on the dyno, baseline, make the single change to the EFII system and observe/ report the results.

Click to expand...

Sounds familiar.

Dan, those HP and fuel flows, they would have to be ROP. BSFC around 0.498 for the Bendix and 0.46 for the EFII.

As it was being run ROP there could be any variation applied on the rich side. So you can make a ROP HP rating vary by tiny amounts but with considerable fuel flow differences. Not that I need to tell you that of course. But many reading this may not pick up on that.

So what data did you gather at best BSFC at 24/2400 for arguments sake? (or any other setting). I would expect that the EI would achieve a good number. I get something like 0.397 with my stock so it would make for interesting viewing.

I hope you had time to collect it. Would have been a fun day testing.

Be nice to see the ambient and the CHTs for both cases. Not that it is a direct substitute for peak ICP and ThetaPP but it would give a guide as to what stress is being applied in achieving the results.

Bluelabel said:

The results:
The big one that we were all watching was of course the WOT HP reading. ........All of the above leads me to believe that our test results got scewed, We?ll never know for sure, unless we do it again. (Which we will?..stay tuned)

Click to expand...

Thanks for the report, John.

I am no expert on the design of EFII but have faith in Robert Paisley and Allen Barrett. They are credible and knowledgeable.

My experience with EFII and the Subby engine was good except for the sub system failures of the engine and inadequate cooling. If I were younger and richer I would be perusing with the 0360 what you are doing with the IO-540. It is exciting, smooth and fun to experience the effect of EFII and an engine.

Again, thanks for reporting your adventure in this area.

RV10inOz said:

Dan, those HP and fuel flows, they would have to be ROP. BSFC around 0.498 for the Bendix and 0.46 for the EFII.

Click to expand...

The previously posted values had a calculated BSFC of 0.483, with the appropriate correction for fuel density. It's on the chart in the article.

So you can make a ROP HP rating vary by tiny amounts but with considerable fuel flow differences. Not that I need to tell you that of course. But many reading this may not pick up on that.

Click to expand...

Right. They shouldn't have any trouble picking up on it, as the article states "The fourth and fifth tests were more subjective. We would have Barrett set 24 inches/2400 and 19.5 inches/2400 to simulate cruise conditions, then search for the lowest possible fuel flow without significant torque loss?"

The article is available to all subscribers as a PDF download, so I'm sure anyone who wishes to read for themselves can obtain a copy sooner or later.

Thanks Dan,

I will keep my eye out for a copy down here. Unless you can email me the article you wrote. It would make a good read.

DanH said:

Tad, it was 276 vs 265. All this argy-bargy and dog-ate-my-homework over a whopping 4%.

Never mind that I also reported 194hp @16.3 GPH on mags and Bendix, vs 190hp @14.8 GPH with the EFii systems, both at 24/2400. That's 2% hp for 10% less fuel, at a high cruise power. If you could claim that, would you make a fuss and draw attention to the other?



Sounds familiar.

Click to expand...

Perhaps the frustration stems from different, more favorable (for the EFI managed engines) results that came out of a similar test, on a correctly set up dyno, 2 days later. Results that you were aware of before the story went to print, but chose to run anyway?

I mean, I don't have any affiliation with Protek, other than knowing Robert from CCB. I simply looked at their system, compared to all the other EI systems I've worked on or installed, and determined that the EFII system is the only one worthy of a dual EI installation from a hardware standpoint.

For an up and coming company, to have their first major dyno shootout go like that, and for a major industry magazine to publish 'results' without a follow up on a dyno that is properly set up might be frustrating. Especially since HP sells systems, not MPGs.

osxuser said:

Perhaps the frustration stems from different, more favorable (for the EFI managed engines) results that came out of a similar test, on a correctly set up dyno, 2 days later. Results that you were aware of before the story went to print, but chose to run anyway?

Click to expand...

As usual, things may not be quite what they seem.

Sunday afternoon, June 29th, Robert and I were on the phone almost an hour:



Among other requests, Robert asked that I call Kevin Eldredge and consider their results. I agreed.

I called Kevin on Monday. We discussed a number of points. The actual conversation is off the record. However, it breaks no confidence to tell you I asked Kevin for a complete dyno printout comparable to the Barrett dyno logs.

I sent a follow-up email the same afternoon:

From: Dan Horton
Sent: Monday, June 30, 2014 12:42 PM
To: [email protected]
Subject: Dyno Data - EFII

Kevin,

Thanks for taking my call. Again, I apologize for the intrusion, but Robert insisted that I contact you, and I promised.

As discussed, any data you may choose to supply is off the record until agreed otherwise. Published or not, it would be interested to examine the raw dyno record for the same engine, run with conventional and then EFII systems.

In any case, please log my email and cell phone, as we’ll no doubt have fun things to talk about in the future.

On Wednesday, July 2nd, at 8:34 PM I responded to a note from Robert:

As promised, I spoke with Kevin about midday on Monday, and requested a raw dyno download for the same engine, run with conventional systems and then EFii systems. He agreed to send along some data, which will be off the record until agreed otherwise. As of tonight, I’ve not yet received it.

On Thursday, July 3rd, I followed up with Kevin by cell phone:



And this by email on Tuesday, July 8, at 1:44 PM:

Hi Robert,

Kevin and I agreed Titan dyno data is off-the-record. Although I am very interested, as it stands now I can’t publish it, or even refer to it, without a new agreement with Kevin.

The current information is very limited. For example, there is no indication if these are brake or corrected HP values, no IAT, baro, or humidity, and no comparison CHT’s, EGTs, BSFC, and fuel flows. No dates either. Is the AFP/Lightspeed data from some previous time? How much has the engine run since then? If it was flown (like for the prop vibration survey), I would expect it to make more power as it loosened up.

Think you can get a complete information, and Kevin’s blessing? Editorial deadline is the 10th.

In the end, the deadline was extended a month. I never received the Titan dyno log. Robert did send a single Excel graph showing a HP gain of less than 1% at 2700 RPM, hardly a game changer. As noted, I couldn't use it without Kevin's permission, which I did not receive, and anyway, could not have printed it without supporting information similar to the Barrett data.

DanH said:

In the end, the deadline was extended a month. I never received the Titan dyno log. Robert did send a single Excel graph showing a HP gain of less than 1% at 2700 RPM, hardly a game changer. As noted, I couldn't use it without Kevin's permission, which I did not receive, and anyway, could not have printed it without supporting information similar to the Barrett data.

Click to expand...

So what you are saying is, a 4% loss in peak HP from stock mags and injection compared to a 1% gain over a LIGHTSPEED equipped engine wasn't enough for you to wait for another test that everyone else involved thought was necessary? It just seems irresponsible to print data you know isn't complete, especially since the reasons for the low numbers during your tests were obvious and known going in, lacking a dyno with a fuel return line.

This is defiantly a novel in the making, fiction, non-fiction. Striving for better engine management is like the carrot hanging in front of the donkey on a stick with a string, theory, to move forward. Dyno pulls don't always go as planned. Data crashes, etc. or new ideas that need more tweaking. Results, contribute to evolution and knowledge.

osxuser said:

So what you are saying is, a 4% loss in peak HP from stock mags and injection compared to a 1% gain over a LIGHTSPEED equipped engine wasn't enough for you to wait for another test that everyone else involved thought was necessary? It just seems irresponsible to print data you know isn't complete, especially since the reasons for the low numbers during your tests were obvious and known going in, lacking a dyno with a fuel return line.

Click to expand...

Like the results or not, the process seemed pretty cut and dried, and the results of the testing were published objectively. If the EFII installation wasn't optimal, whose fault was that? I'd say it was the EFII suppliers' problem.

If you want your product to do well in comparative testing, you have to show up prepared, which includes bringing whatever bits you need to plumb a fuel return or make any other adaptations are necessary for your installation. Otherwise, you end up on message boards making excuses about why your product underperformed.

Kyle Boatright said:

Like the results or not, the process seemed pretty cut and dried, and the results of the testing were published objectively. If the EFII installation wasn't optimal, whose fault was that? I'd say it was the EFII suppliers' problem.

If you want your product to do well in comparative testing, you have to show up prepared, which includes bringing whatever bits you need to plumb a fuel return or make any other adaptations are necessary for your installation. Otherwise, you end up on message boards making excuses about why your product underperformed.

Click to expand...

Kyle,
Not sure what's with your negativity, why you just want to say "those were the rules, they lost" or if you're just determined to find a way to validate what's in your plane... There were no rules, it wasn't a competition. It was an opportunity to do some testing and I footed the bill.....
The answer to your statement is in my thread....did you actually read it? Had Robert not spent an entire day just helping to get the Bendix system running, he(we, all of us) would have had time to modify the fuel set up and make it correct... As it went, we didn't. It's nobodies "fault" but that's what happened.
I wrote this threat just to fill everybody in on the whole story so that, combined with the Kitplanes article, you'd get the whole picture...
I DID NOT intend to spur on this type of ongoing unproductive bickering... Questions...I support, helpful info tidbits, great!...
Opinionated finger pointing, I cannot support....
Come on people.... You can do better, this stuff is supposed to FUN!!

Bluelabel said:

Had Robert not spent an entire day just helping to get the Bendix system running, he (we, all of us) would have had time to modify the fuel set up and make it correct... As it went, we didn't. It's nobodies "fault" but that's what happened.

Click to expand...

That's ridiculous.

Paisley never checked the dyno room setup, despite being there Monday. Monty Barrett pointed out the lack of a Floscan to Paisley at 11:10 AM on Tuesday, give or take a few seconds....and you were there.

Constructive Feedback

This thread invited constructive feedback, which has been provided. There's considerable useful information here.

This experiment's lessons:

1) Superior due diligence is a prerequisite for meaningful results.
2) Show the press when the results are consistent & repeatable!

Issues

DanH said:

That's ridiculous.

Paisley never checked the dyno room setup, despite being there Monday. Monty Barrett pointed out the lack of a Floscan to Paisley at 11:10 AM on Tuesday, give or take a few seconds....and you were there.

Click to expand...

Dan,
I'm not sure what your problem is....I personally am getting really sick of it.... For once, just leave it alone. We didn't have enough time to correct the problem.... Period end of story. You really want to split hairs about whether there WAS OR WASN'T enough time to re-plumb, and fix the set up??? But you did validate my point...2 DAYS TO SET UP AND TUNE THE FIRST SYSTEM, A FEW HOURS TO INSTALL AND RUN THE SECOND....
One reason I posted this thread is to make up for the fact that the article made us all look like idiots... Things weren't set up correctly, the results were flawed, you knew it, you printed it anyway.... As has been pointed out by many.
All I wanted was to see my engine run on MY system before I took it home.... This has turned into a total S$&@ show....

An Objective Look

No matter how one looks at the Barrett testing, the EFII system as tested was not fully configured with the required fuel return line. It sounds like most of the test time was spent trying to make the baseline engine configuration work (mechanical injection and magnetos) and very little time was left to set-up and test the EFII system. I don?t see how this situation represents EFII (Robert Paisley) not being prepared, diligent, or working hard to make the testing as successful as possible.

In an attempt to keep the dyno fuel flow reading working, Robert installed the EFII system without a fuel return line. In other words, the team elected to proceed despite the sub-optimal EFII test configuration. They ran with what they had; not what they wanted, because the facility was not prepared to handle the new set-up.

I don?t see how conclusions can be drawn about fully-configured EFII system performance except that the EFII system was able to run successfully with a very slight decrease in horsepower and a noticeable increase in fuel efficiency despite the sub-optimal test configuration. Hopefully, Robert and Allen Barrett will be able to re-test in the future with the proper setup.

It should be noted that two days after the Barrett testing, the Titan testing, in the proper configuration with a fuel return line (cooler fuel), showed an increase in horsepower.

Bill Palmer said:

No matter how one looks at the Barrett testing, the EFII system as tested was not fully configured with the required fuel return line. It sounds like most of the test time was spent trying to make the baseline engine configuration work (mechanical injection and magnetos) and very little time was left to set-up and test the EFII system. I don?t see how this situation represents EFII (Robert Paisley) not being prepared, diligent, or working hard to make the testing as successful as possible.

In an attempt to keep the dyno fuel flow reading working, Robert installed the EFII system without a fuel return line. In other words, the team elected to proceed despite the sub-optimal EFII test configuration. They ran with what they had; not what they wanted, because the facility was not prepared to handle the new set-up.

I don?t see how conclusions can be drawn about fully-configured EFII system performance except that the EFII system was able to run successfully with a very slight decrease in horsepower and a noticeable increase in fuel efficiency despite the sub-optimal test configuration. Hopefully, Robert and Allen Barrett will be able to re-test in the future with the proper setup.

It should be noted that two days after the Barrett testing, the Titan testing, in the proper configuration with a fuel return line (cooler fuel), showed an increase in horsepower.

Click to expand...

Thanks Bill,
I think that sums it up.
Yes, the hope is to do it again..at Barrett. Since I go right near there frequently and I really like Rhonda and Allen (Just great people, what can I say), I've offered to help facilitate getting them fully set up for regular dyno runs and maybe even do another test just out of sheer curiosity...

John

Bluelabel said:

Dan, I'm not sure what your problem is...

Click to expand...

I can help you with that!

The problem is your statements regarding schedules and available time.

You previously wrote:

Bluelabel said:

The "plan", as best I know, was for the engine to be running on Mags/Mech Monday morning, do the runs, collect data, then Monday afternoon switch over to the EFII set up, then have Tuesday to tune the system, train Barrett and us, then Wed break it down to take home. Well, as you can read, that didn't happen. It turned more into a reality show type deadline racing the clock. The EFII system literally got slapped on, ran for about 5 minutes before the runs to take data were done. Hence the improper set up.

Click to expand...

The shop's standard run-in was scheduled for Monday. Allen's first priority was to ensure the core engine was satisfactory.

Set up and run-in was not really an all-day affair. There is no reason why you would know about the whole crew spending some of Monday morning breezing with a visiting celebrity pilot, or taking a nice lunch at the Mexican place, or knocking off at the usual time to have a lovely dinner downtown (Thank you Rhonda!). In between, we turned wrenches.

As noted in the article, the Bendix and Slick components were used stuff from the storage room. We had to try another flow divider, swap a pair of miss-connected fuel lines, and send the mags up the street for overhaul. Most of the run-in was complete when we quit for dinner. Allen came in early Tuesday and finished it up before any of the rest of us got there.

The reason there was no plan to do the comparison runs on Monday is rather iron-clad; Monty spends a big chunk of every Monday, Wednesday, and Friday at a medical clinic. It's not optional.

Moving to Tuesday, take a look at the dyno log. It's set for CST, not CDT, so add one hour to each for local time. The Bendix/Slick runs were finished at 9:32 AM. The rest of the day was all EFii, and the blow-by-blow account in the article is dead accurate.

One reason I posted this thread is to make up for the fact that the article made us all look like idiots...

Click to expand...

No, it doesn't. Each reader will form his own opinion when he obtains a copy, but I know my intent. It's merely an narrative of events, as they happened. No one is crucified or blamed. There is no use of "winner" or "loser", no end zone dance, no spiking the ball. Mr Paisley took responsibility for what he later felt was an error. His explanation is given due weight, and it is clear that the outcome could be different on another day. You personally are portrayed as a nice young man with great in-laws.

Things weren't set up correctly, the results were flawed, you knew it, you printed it anyway.

Click to expand...

The Barrett dyno room has always included a tank return line. What it didn't have was a Floscan to measure return quantity. It was Mr. Paisley who elected to tee back to the pump inlet, but it should be noted that we could have just as easily connected to the tank return, initially or after the first run. All we would have lacked is a fuel flow indication; torque, EGT, and all else would have remained operational. It would have immediately proved or disproved the bubble theory. Instead you were instructed to make a timing change. Not a good call, but such is the nature of run-what-you-brung.

I use the word "theory" for a reason. None of us know factually if bubbles existed, or if they would make a difference. Subsequent runs at another facility have been vigorously waved about as proof, but as noted previously, no one would produce the complete data set.

The ground rules for the story were laid out many months ago. It was going to print, regardless of outcome, and would relate what actually happened, not what one might wish had happened, or believes should have happened. There's no conspiracy of the Luddite League, no malignant Oz behind the curtain. In the end it's just a story about gearheads having fun. I do hope everyone will read it and decide for themselves.

My problem with the whole thing has more to do with responsible journalism than Dan H's, my own, Robert P.'s or anyone elses OPINION on the the system itself.

Dan H, you have made it clear through multiple jabs in other threads on this forum that you believe the EFII system is sub par to others, fine.

In real world settings, and in the later Dyno test everyone is seeing more power over mags/mech injection or carb. This is from speed tests in the real world. That the Titan dyno run had to remain proprietary in unfortunate for Protek from the standpoint of your article.

From the standpoint of publishing data from incomplete tests, that no one was happy with the circumstances of (except maybe you? Although, I would like to think that you would have liked to see a fully functioning dyno test as designed as well). I'm surprised Kitplanes even published it, and I'm disappointed that you submitted it as finished work.

If the mags/Mech Injection had been forced to the same timeline, it would have posted incredible efficiency, burning no gas at all, but disappointing power numbers, 0HP and 0LB/FT of torque.

This thread has run its course. Closing it.