Undocumented Microchip Errata Strike Again

Two years ago, I found an issue with the open drain performance of one of the I/O lines of a PIC24FJ32GB002 microcontroller.  I got it confirmed from Microchip, but they never actually put out an errata on that behavior.  This time, I have found an even larger problem, that I am sure will rise to the level of an official errata.

I was trying to use a PIC24FJ64GC010 microcontroller in a design for a client.  The design has a capacitive sensing keypad that has worked well in the past with a slightly different Microchip microcontroller.  I moved to the PIC24FJ64GC010 because it had some other features that were necessary for the design.  Unfortunately, migrating to the new micro would prove to be problematical.  The capacitive keypad was implemented as shown in the schematic below.  In order to get the best performance, each button was implemented on its own.  In addition, the CTMU output current was routed through the internal op-amp as a voltage follower so that the noise immunity of the keypad would be greater.  Little did I know that dragons lurked in this approach.

image

 

Problems arose when I started shaking out the PCB.  For the life of me I could not get some I/O lines to output a current pulse.  The scope capture below shows the problem.  The yellow trace shows the output of the op-amp.  Every time the current from the I/O goes up, the current on the op-amp should ramp up.  The current is turned off and then a different I/O is selected.  Every time a button is sampled, the micro would toggle an I/O as shown with the green trace of the scope capture.  As can be seen from the scope capture, only 8 current ramps are shown.  However, there are 12 toggles on showing that the micro was trying to output current with the CTMU.

image

 

At this point, I put on my troubleshooting cap and went to work, because when problem like this arise it is almost always the engineers fault.  I individually addressed the I/Os shown in the schematic in digital mode – I could read inputs and set outputs as desired on ALL I/O lines.  I could also read analog inputs on all of the I/O lines (which due to the new A/D architecture is a feat in itself and will be detailed in an upcoming blog).  I could individually turn on and off the internal pull up resistors.  However, I could not make 4 of the I/O lines output CTMU pulses.  I even lifted individual I/O pins off of the PCB to ensure that the problem was not somehow related to the PCB layout.

I spent some time trying to cajole the Microchip application engineers that there was a problem and could they help me.  After fending off their initial dismissive replies (ie – “try using our sample code”, “it works here”, “have you used our demo board”, “you are overdriving the outputs, they should not clip”, etc), after 2 weeks, I was finally able to convince them to look at the problem for real.  I had to send them my hardware. and from there it was easy for them to confirm the problem.  Another engineer here was able to initially figure out the pattern that allowed the diagnosis, which is fairly odd:

“The CTMU will NOT output on any A/D that is even and less than 15”.  For example, RB4(AN4) will not output CTMU pulses while RB1(AN1) will.  Unfortunately for me, I had randomly selected four of the offending analog I/O lines.  This was a setback to the schedule and my stress level.  However, at least I know I am not crazy and I did everything I could to get the part to work.  In order to get the design to work, I needed to reroute the board and leave off these bad I/Os.

Unfortunately, I do not know if this is a family-wide problem, or a problem with any of the new advanced analog micros, or just related to the PIC24FJ64GC010.  It will be interesting to see what the errata says.  Microchip put out a new errata sheet on 4-20-14 (about 3 weeks after they confirmed the problem), but this issue is not listed.

Measuring small currents with an oscilloscope

When designing systems with microcontrollers, it is often necessary to make them low power and to ensure they draw minimal current.  Unfortunately, common desktop instruments do not allow for easy verification of these currents.  A DMM can really only show DC currents and can’t capture transients.  An oscilloscope typically can only measure down on the order of 2mV per division range.  These limitations pose a problem for microcontroller systems that typically have to wake up periodically to check things and then go back to sleep:  the current waveform is too active for the DMM to measure and too small for the oscilloscope to resolve.  A simple and obvious method around this is to amplify the current measurement so that a scope can reliably show the current waveform.  This blog entry describes one, easy-to-implement method for accomplishing this.

The figure below shows the schematic for the amplification circuit using an instrumentation amplifier.  A brief description of the circuit follows. 

current sense amplifier.

  • The instrumentation amplifier is the Linear Technologies LT1920. This is a basic instrumentation amp (IA) that comes in a DIP package so it is easy to prototype and even breadboard with. It is readily available from Digi Key.
  • The gain of the circuit is set to 1000. With the expected minimum current of the DUT(on the order of 100uA) and a gain of 1000, the output will be about 100mV. This is an easy voltage to see with a scope. In addition the maximum current is about 5mA, so a gain of 1000 gives an output voltage about 5V. Again, this is easy to see with a scope.
  • The gain bandwidth product of the IA is large enough to show the transitions of the circuit.
  • The circuit is run off of +-10V so there is enough headroom to output the approximate +5V output. Dual supplies are necessary because the minimum allowed voltage input is V- + 1.5V.
  • The 10k provides a load so that output of the IA has a return path.
  • Pin 5 is tied to ground so that the output is referenced to ground.
  • Outside of the circuit are the Device Under Test (DUT) power supply, the DUT, and the current sense resistor. The current sense resistor is sized to ensure that the current can be adequately measured and that it does not affect the DUT performance. In this case a 1-ohm resistor is used because it meets these requirements and allows for easier math (see below).
  • The amplifier circuit was only bread-boarded so the actual accuracy is not dialed in, but the results are close enough to show the idea.

A typical low-power micro spends most of the time in a low-power state an wakes up periodically to check conditions.  The average current can be modeled as shown in the equation below.

clip_image002

To get the total average current the amplifier can be used to show the current on an oscilloscope and then the equation can be filled in.  Below is a scope capture of a typical micro current consumption while in a sleep condition.  The asleep/awake behavior can be seen. For approximately 80% of the time the current is low – the voltage is approximately 250mV which is equivalent to 250uA.  For the remaining 20% of the time, the micro is waking up and then for a real short time it is at full power as it performs its actions before it goes back to sleep.

image

By zooming in on the higher current portions the following times and currents can be found for the DUT.

Micro State Duration Current (Measured Voltage/1000)
Asleep 813ms 250uA
Waking Up 185ms 925uA
Active 2ms 5.5mA

Using these measured values, the average current equation can be filled out as shown below.

clip_image002[7]

So in this case, the average current for the micro system is 385uA. 

This method can be modified to measure smaller currents by increasing the current measurement resistor and/or increasing the gain of the IA.  Also, more accurate measurements can be made by placing this circuit on a PCB.  Also other IA ICs can be used to get better performance.  Hopefully there are some ideas that you use to help troubleshooting in the future.

The travails of fixing an old design

 

data_erase

 

Recently I had the opportunity to fix a design I did way back in 2005.   It turned out to be a bit more trouble than I anticipated.  In the past I have talked about the need for good record-keeping and how that gets harder  and harder to do as the technology in question gets older and older.  Fortunately for me we had been conscientious enough to retain enough of our old-technology to allow me to fix the problem . . . . barely.

[Read more…]

National Instruments DAQ Attack

daq

They say no plan survives contact with the enemy.  I think that pearl of wisdom applies to engineering.  In my experience the enemy takes the form of everything with wires as well as software running on anything with wires.
It certainly applies to a project we’re currently working on.  We can’t go into too much project detail, as the design is part of a contract we’ve been hired to complete. But we can cover some of the broad concepts that describe the battle we just survived.

[Read more…]

Helping the USFS fight fires

The day after the Thanksgiving break, I had the opportunity to give a training to the US Forest Service at the Air National Guard base in Oxnard California.  During the winter, the USFS performs maintenance and upgrades to their aerial fire fighting equipment.  In this case it was for their MAFFS program.  Even more specifically, their MAFFSII program.  About ten years ago, Solutions Cubed developed the electronic controller for the fire fighting system.  It reads inputs such as drop amount and flow rate and then controls the output so that the right amount of retardant is dropped for the right amount of time.  We were sub-contracted to the main company that produced this system.  Unfortunately, during the economic downturn, they went bankrupt and the USFS is now scrambling to cover the support issues with the system and its continued operation.  That is where the Solutions Cubed training comes in.

There are about ten mechanics that service all of the MAFFS systems, but they were not around during the initial design and development of the system.  In addition, they are not super techno-savvy.  Also, because the MAFFSII system is loaded into Air National Guard C-130 airplanes, there is a layer of military rules and regulations on top of the USFS layer.  So the training and resource materials we provided had to be very straight-forward and super easy to follow.  I was able to accomplish both goals, but just barely.  When I started the two hour talk, it soon became apparent, I was approaching the training from a too-technical direction.  I was able to change things up on the fly and things went smoothly from there.

Solutions Cubed redesigned some interface software for the controller, so we demonstrated its use and detailed all of the functions of the controller.  The mechanics and the USFS representatives were receptive to the training and appreciated the contents.  After the classroom portion of the training we were able to try out the new software on an actual system that had been removed from a C-130 and was on a trailer.  The photo shows the size of the system and its complexity.  After we hooked up the software, we went through a couple of procedures in the manual.  We found a couple of problems with the documentation that Solutions Cubed will have to change, but overall it was a good training.  I look forward to seeing the planes flying and fighting fires again.

 

2012-11-26 12.54.09

Plotter nemesis defeated

I have a love-hate relationship (as do most folks in our office) with our HP 450C D-sheet plotter.  On the positive side of the ledger:  it prints out D-size sheets in color.  This proves valuable for schematics – having the entire circuit on one legible page makes understanding and troubleshooting designs much easier than having the schematic on a screen or paging through a bunch of 8 1/2” X 11 “ pieces of paper.  On the negative side:  it uses 12-year old ink-jet technology, so the ink pots dry out, unless a couple of full color pages are printed a week; loading a roll of paper, while theoretically easy, tends to become a half-hour adventure in “hoping the plotter accepts the paper this time”; sometimes it does not print certain symbols on the schematic; it is 12 years old, so it is obsolete and HP does not seem too keen on supporting it.  All that aside, the plotter is a workhorse in our company.  Because plotters are such a low volume item, the price of a new replacement is on the order of $2500 – pretty big pill to swallow when the plotter “mostly works”.

Unfortunately, last Thursday, the plotter gremlins decided to strike.  I was plotting a schematic for a new test circuit when the plotter made the most ungodly noise and stopped working altogether.  All three of the error indicators were blinking an ominous orange.  Upon further exploration, I was able to see that the main drive belt appeared shredded.  Much to my chagrin, there is no easy way to replace this.  Luckily I was able to find the actual service manual on line and disassemble the plotter.  Because of the way the plotter is assembled, this project would have been much more difficult without the manual.  There are a bunch of “hidden” screws and odd orders of operation.

IMG_0707

After some teeth-gnashing and hair pulling, I removed the offending belt.  Obviously, there was no way to salvage it.  I think the belt was the original – so the 12 year old rubber was probably due to fail.

IMG_0706

I ordered a new belt through Amazon.  It showed up the next day (amazing!).  I was able to get the plotter back together on Friday – it all looked good.  But I was unable to load the paper before I had to leave – my first try ended in heartache.  This morning I tried again and 20 minutes later, I had the plotter back up and running.  Now I should be able to get back to designing cool stuff, and the rest of the office can plot in peace.

IMG_0708

Toothbrush saves the space station

The headline is probably overwrought, but essentially astronauts on the International Space Station needed to replace a power unit on the exterior of the space station.  They tried to complete this replacement a week ago, but were foiled by metal shavings in a bolt hole that made it impossible to drive a bolt.  So after eight hours in space, the astronauts returned to the space station to come up with a way to overcome the problem.  The main catch being that they couldn’t run down to Home Depot to pick up some parts – they had to make do with what was on-board.  In a scene that I like to imagine was exactly like the one from the movie Apollo 13, the international space station astronauts along with engineers, technicians, scientists, and assorted- interested parties on the ground came up with a solution.

There are a couple of things I like about this.  First is the teamwork and communication involved that got the problem fixed.  Secondly, like I have mentioned about keeping good records, everyone know exactly what was on the space station.  They were then able to figure out how to use those items in not-intended manners to fix the problem.  With these tools, smart and motivated people were able to solve the problem.  From reading the articles, I don’t think that the toothbrush actually saved space station – from what I can tell they used it to clean the metal shavings out of the bolt hole – but the troubleshooting was impressive.

As an engineer, I need to remember to keep this open mindset when I approach new problems and non-working issues.  Maybe I can use a tool in a different way than I am using it now to overcome my obstacle an move on.  It seems that the space program is providing quite a bit of food for thought over the the past few months.  I hope it continues.

Troubleshooting for the US Forest Service

IMG_0533

I recently discussed the necessity of keeping and having good records and data so that troubleshooting and problem-solving in the future could be performed.  I had no idea that these ideas would come home to roost so quickly!  I spent the last couple of days on the tarmac helping the US Forest Service and the Air National Guard fix one of their fire-fighting air tankers.  It really drove home how important good information is to quality engineering.

About ten years ago Solutions Cubed completed controller for a modular fire-fighting system that would fit into a C-130 airplane.  The client delivered the products to the US Forest Service, but then went out of business.  The Forest Service has been maintaining the units for the last couple of years with no vendor support.  Unfortunately, when the vendor went out of business, the Forest Service also lost all access to the schematics, complete wiring diagrams, parts lists, etc.  Needless to say, this makes maintenance difficult.  Recently they found a problem in the electrical system that they could not troubleshoot on their own and were able to find the Solutions Cubed website and contact us for help.

Four of us (a US Forest Service engineer, two mechanics, and I) spent a day tracing the problem and found the culprit – a shorted potentiometer.  We were also able to reconstruct how it shorted (someone pressure washed it while it was powered!).  That evening, they replaced the pot (on an airplane, nothing is replaced fast – there is too much at stake if something is messed up).  The next morning we were able to test the fix and everything was back up to snuff.  Another couple of hours of functional tests and the plane was cleared to fly.  This is great, because this particular airplane is being used to fight the Chips, Reading, Ponderosa, Rush, and other fires all around where Solutions Cubed is located.  So there is some self-interest in getting this fixed!

It is great that we were able to fix the problem, but it would have been much easier to diagnose if we had the correct documents.  Below is video of the system operating on the ground.