Two peripherals attached to Q68 fried at the same time. Coincidence ?

Nagging hardware related question? Post here!
Post Reply
FrancoisLanciault
Trump Card
Posts: 167
Joined: Mon Aug 08, 2011 11:08 pm

Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by FrancoisLanciault »

Hi,

As the title says, my Q68 setup sustained multiple casualties that appeared to take place at the same time.

The Q68 itself still work great. But I want to have expert opinions as to what might have happened and precautions I should take in the future.

Here is the setup:

Q68 powered by a HP-6623A lab regulated power supply. The Q68 is attached to a LCD screen and to a keyboard/trackball combo.

The HP-6623A also powers the LCD (12V) attached to its output #1
The Q68 is power by output #2 (5V)

The keyboard/trackball is attached to the Q68 through a KVM switch. (But the screen is directly attached i.e. not through the KVM switch)

I have been using the Q68 in this setup for a few weeks without problem.

A few days ago, after powering up everything, the Q68 boot process stops with the "No valid SDHC card found" message. A bit surprised (first time I see this) I power off the Q68, unplug/plug the card, retry and I get the same message. Try again after cleaning the contacts, same problem. Then I noticed the Q68 draw 0.450 Amps, it usually draws less than 0.200 Amps. With the card removed, the power consumption goes back to <200mA. The card was the one I received with the Q68.

As I wasn't sure if the problem was the Q68 or the card, I connected the card to a SD card reader and to my Mac, hopping the issue was with the card. As a matter of fact, the Mac didn't saw any new volume. After trying a few minutes, I removed the card from the reader and it is really HOT, like "burning your finger" hot.

I was quite surprised as I had no idea a SD card could become so hot. After reading on the internet, I realized this is an known issue with some SD card, some of them even catching fire. I guess this explain why the Q68 was drawing so much power, but probably even more current went through the card when it was plugged to the Mac.

I procured a new card and went through the process of re-installing SMSQE and WIN containers. (That is the only good point about the story. Now I know how to do this and I have learned a few things in the process). I lost all my Q68 configuration (no backup, I know, I know), but at least I did not loose any code as I backup my code on all my QLs).

I don't think "freaking hot SD cards" is a common problem, so I guess I was unlucky, but all in all not to big of a problem; I will do an even better SMSQ/E/application setup and this time I will do backups...

The new card booted without error messages and soon I was presented with the 3 windows and flashing cursor... only to find out that the keyboard did not work anymore. But the trackball worked. Removed the keyboard from the KVM switch: same problem. Try the keyboard to another computer: not working. Open the keyboard and checked the cable/plug continuity from the PCB to the pins: no issue.

Now, having a self destroying SD card, I can accept that. It happens from time to time it seems. Having a working keyboard that stop working, I can accept that also. But having the two happening at, it seems, the same time, I find it strange.

I can see four possibilities:

1 - Something went wrong with the power supply and it fried both components. (I did something differently this particular time, see below)
2 - The card decided to die and brought the keyboard with it. Why ? Current too high ????
3 - Something went wrong with the keyboard and the card died because of it. Very unlikely. But who knows.
4 - Pure coincidence, also very unlikely.

Now more details about the powering up procedure I used.

The lab power supply can operate in two modes
- Constant voltage, the current will adapt to the power requirement of the circuit.
- Constant current, the current is limited to a certain value, so the voltage adapt so V=RI is still true.

By default, for safety, the HP-6623A have a preset maximum current of 0.080Amps and a voltage of 0 volts

What I usually do when I use this power supply is:
- I first disable the output
- Then I key in the maximum current I want to allow. In the case of the Q68 I usually enter 2 Amps. This is high enough so the PSU won't operate in constant current mode.
- Then I key in the voltage, i.e. 5 volts.
- Then I re-enable the output.

This is, in my opinion, the safe way to do it. And it has been working for weeks.

But this particular time, I was careless, (though it should not have been a problem)

- Powered up the PSU
- I did not disable the output
- I keyed in the maximum voltage first i.e. 5 volts

The fact that the output was limited to the default 0.080 Amps, the applied voltage end up to be around 2.7 Volts. So the Q68 stayed silent. When I realized the problem, I set the maximum current to 2 Amps and the Q68 was brought to life... with the "No valid SDHC card found" message. You know the rest.

Now even though I did not followed my rigorous power up scheme, I find it hard to believe that 2.7 volts at 0.080Amps can destroy electronics designed to works at 5 volts. But I am no expert, this is why I ask.

As I said, the Q68 now works perfectly with another different keyboard and the new card. I did not tried yet to go trough the KVM switch (can that be the problem ??) I did not tried all Q68 features (i.e. serial port)

I did bought another keyboard combo, identical to the one that fried. But I am uneasy to plug it to the Q68, even though it is probably not the culprit. (This particular keyboard is quite expensive and difficult to come by, so I don't want to blow another one)

What is your views on this ? Reason 1, 2, 3 or 4 ?
Was I very lucky my Q68 survived ?

Peter will know the electrical path between SD card and keyboard and if one can bring down the other.

Or am I just very unlucky ??

Thank you for your time.

François


Derek_Stewart
Font of All Knowledge
Posts: 3928
Joined: Mon Dec 20, 2010 11:40 am
Location: Sunny Runcorn, Cheshire, UK

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by Derek_Stewart »

Hi François,

I am sorry you having problems with your Q68 board.

I will put some freshly made Q68 boards on overnight test to see if I can re-produce your high temperature problem with the SD Card.

Have you checked KVM Switch on another computer.

I have used 2 Amp power supplies on the Q68 board with no problems.

Can you check the Lab Power supply is correctly calibrated.

The Q68 SD Card reader sockets are supplied with 3.3V 200mA through a voltage regulator for both card reader sockets. It looks like this is working as you would not be able to read a fresh card. Any higher voltage to the socket would of blown the voltage regulator and both sockets would not be working.

Maybe the SD Card is is faulty or the MAC computer caused an issue with the card. Can you check the MAC computer to see if a new card is hot or cool.


Regards,

Derek
User avatar
Peter
QL Wafer Drive
Posts: 1953
Joined: Sat Jan 22, 2011 8:47 am

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by Peter »

The 5V of the power supply connector goes directly to the keyboard, no Q68 components involved except the PCB traces. So if the keyboard was damaged, I'd have no idea what the Q68 could have contributed.

What exactly can happen to a complex mix of electronics if a current-limited supply is attached that creates an instable wrong voltage is not easily predictable. Among other issues, voltage regulators can get into oscillation, power-up timing requirements between the different internal supply voltages will be violated, and distances between the internal supply voltages will be violated.

I can not say it will cause damages (although that is quite likely in case of the SD card), but I can also not say it will not. It is simply not allowed.

It seems a repeating issue that people connect unsuitable power supplies and then have problems.

Only use good quality regulated 5V supplies. Voltage must always be constant. It is not acceptable that other voltages are applied, also not smaller voltages or current limits.


User avatar
tofro
Font of All Knowledge
Posts: 2685
Joined: Sun Feb 13, 2011 10:53 pm
Location: SW Germany

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by tofro »

Even if it might sound a bit silly: A lab PSU is maybe not the most suitable thing to power a computer. Set to "Constant Voltage", these PSUs are built to maintain the output voltage regardless of what current you draw from them - they will just push in copious amounts of energy to maintain their output voltage - A "normal" PSU will give in much earlier and drop the voltage.

What I think may have happened:
  1. SD card went bust (due to whatever reason) and started to draw significant current. Sh*t happens.
  2. Voltage dropped, the HP PSU increased it to stay within the limit you set
  3. You removed the card, HP PSU may not have been fast enough to adapt, so created a short over-voltage condition.
  4. Voltage peak destroyed the keyboard
This assumes you pulled the SD card while the power was on - This might very well have resulted in a peak high enough to destroy the keyboard. I could imagine the same thing can also happen when the card was still in there, and the PSU came within its limits.

I would strongly recommend you use a "weaker" PSU - I guess with the same SD card defect, the Q68 would just have crashed because with the high current drawn by the card the voltage would have gone too low for the Q68 to operate, but your keyboard might have survived.

Tobias


ʎɐqǝ ɯoɹɟ ǝq oʇ ƃuᴉoƃ ʇou sᴉ pɹɐoqʎǝʞ ʇxǝu ʎɯ 'ɹɐǝp ɥO
User avatar
Peter
QL Wafer Drive
Posts: 1953
Joined: Sat Jan 22, 2011 8:47 am

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by Peter »

tofro wrote:[*]SD card went bust (due to whatever reason) and started to draw significant current. Sh*t happens.
If I understood correctly, François powered the overall system including SD card by sort of a "80 mA current supply" - this has a good chance to produce wildly oscillating internal supply voltages and ruin the SD card. The Q68 electronics itself is likely to be more robust.

Side note: The Q68 provides separate voltage regulators and buffer capacities for the SD card sockets, reducing the impact of hot-swapping on the rest of the system.

To me it looks like the headline could be "Two peripherals attached to wrong power supply fried at the same time. Coincidence ?"


User avatar
Peter
QL Wafer Drive
Posts: 1953
Joined: Sat Jan 22, 2011 8:47 am

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by Peter »

FrancoisLanciault wrote:But this particular time, I was careless, (though it should not have been a problem)

- Powered up the PSU
- I did not disable the output
- I keyed in the maximum voltage first i.e. 5 volts
To be more clear: This is a problem, and has the potential to damage sensitive circuitry, especially with several further internal power supplies that are derived from the PSU output.

A suitable power supply must be able to provide a turn-on current which is higher than the average current during operation. The supply voltage must ramp up monotonically, and reach it's stable state in less than a few hundred miliseconds.

Providing a current source (as seems to be the case here) is worse than, for example, just a too low voltage source. Simplified example:

3.3V regulator for the SD card starts up as input voltage from PSU rises -> SD Card supply rising
SD Card supply rising -> SD Card tries to start up
SD Card starting -> needs more current
More current won't be supplied -> PSU reduces voltage
Voltage going down -> Startup disrupted and less current needed
Less current needed, but PSU trying to keep current constant -> voltage from PSU rises again
Repeat...

In effect, the SD card supply can start oscillating, and it is not even clear what kind of oscillation that will be or how long it will last - as there are other voltage regulators in the system and other components that run into irregular startup conditions as well. Furthermore, the cores of FPGA and SDHC card run at even smaller voltages, which are derived from those instable supplies. The SDHC card usually contains an internal LDO for that. With bad luck, the result can be a total mess of wrong, instable supply voltages, violating both the need for a monotonic rise and the need to rise within a reasonably short time. Furthermore, interdependences between the different internal voltages can be violated. E.g. FPGA using multiple supplies require a defined "power-up sequence".

I have myself actually damaged electronics by accidentally letting a lab power supply run into inadequate current limit at power-up. So this is not just theory. DON'T DO IT.


User avatar
Dave
SandySuperQDave
Posts: 2765
Joined: Sat Jan 22, 2011 6:52 am
Location: Austin, TX
Contact:

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by Dave »

The SD cards are fed power by a XC6215B332NR-G. This is a 200mA part with a 6V upper limit and a very low drop out voltage of only 320mV. It is followed by a 10uF cap. It seems perfectly suited to the job of supplying two SD cards, even if two are internally working at the same time.

Other regulators are an LM1117-3.3 (800mA) to power video, an XC6215B112 (200mA) to provide 1.2V for the FPGA core. Again, all well capped on the downward side.

I used the way this power supply system has been designed as a mini case study for designing the power system inside the Issue 8 board. I made one change. I added a pair of caps near the 5V input to prevent ringing. I never observed the ringing using a 5V 1.5A wall wart, but I did see it using my generic bench power supply when in current limit mode set to 500mA. This is not a fault of the Q68, but a characteristic of current limit setting on bench power supplies.

What happened was the voltage would start to rise and the lower regulator (XC6215B122, 1.2V) starts settling down and charging its caps. Then the two LMP1117-3.3s both start working at the same time and settle down to their assigned output voltages as their downline caps charge. This inrush current hits the power supply current limiter and it starts to respond by bouncing the voltage, which on my crappy bench supply would be between about 2.3 and about 6.8v. The ringing frequency was 22Hz and it lasted for about a second before settling down. It was repeatable.

What the Q68 sees is a delay on reset/power good for the FPGA, video is given a little noise for a few ms (but this isn't visible as video hasn't even started yet). The 47u cap on the Q68 wasn't low ESR, and the calculated impedance during this period was 154 ohms.

The SD card's regulator flips out and detects input voltage below threshold and shuts off, then restarts.It bounces between 2 and 5 times with *my* bench supply.

This does not happen at all with my little 5V 1.5A retired Vonage VoIP box adapter. It's a voltage regulated supply and comes up to 5.08 volts nice and smoothly, allowing all the systems to come up in turn within the Q68. My bench supply, in voltage regulated mode, also comes up smoothly.

So, the failure report of SD card and keyboard tells me the problem was with the power being fed into the Q68, not the Q68 itself. If the bench supply is anything like mine, it could have started ringing for any number of reasons.

I also went through my entire box of wall warts (I don't throw them away, they might be useful) and can generally comment that while some of them are very good, others are very poor. The ones designed to provide 5V for USB were the worst, putting out anything from 4.5v to 8.2v, with lots of ripple. So I do not use wall warts that were designed to *charge* things to ever *power* things. The worst wall wart was a little USB-C one that under low loads (under 100mA) would put out 8.2V!

The only thing I read above I could disagree with: the 200mA regulator for the SD cards wouldn't blow. It reacts very quickly to over current and reduces output. It reacts VERY quickly to input voltage falling below threshold (about 3.62v). I would put a 1u cap before it like the datasheet says, particularly because SD cards put some quite high frequency noise onto their power lines and tend to demand all or nothing. The 47u electrolytic just a couple of cm away covers it pretty well.

So, don't use bench supplies in current limit mode and when using a wall wart, select one that's designed to power devices, not charge them. The regulation is a whole lot better.


FrancoisLanciault
Trump Card
Posts: 167
Joined: Mon Aug 08, 2011 11:08 pm

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by FrancoisLanciault »

Thanks Peter and all.

I guess this is a good lesson learned for me. Always make sure the power supply cannot fall into limited current mode. I won't make the same mistake twice. Just loosing a keyboard is not too bad. Could have been worse.

I have re-connected my new keyboard combo through the KVM switch and everything works.

Now that the problem have been identified, I can move along with my project!

François


User avatar
Peter
QL Wafer Drive
Posts: 1953
Joined: Sat Jan 22, 2011 8:47 am

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by Peter »

Dave wrote:What the Q68 sees is a delay on reset/power good for the FPGA, video is given a little noise for a few ms (but this isn't visible as video hasn't even started yet). The 47u cap on the Q68 wasn't low ESR, and the calculated impedance during this period was 154 ohms.
Off topic remark: The damping is intended, although I think impedance is less that the calculated figure. Low ESR would result in a very small, but visible noise on video output.


User avatar
Dave
SandySuperQDave
Posts: 2765
Joined: Sat Jan 22, 2011 6:52 am
Location: Austin, TX
Contact:

Re: Two peripherals attached to Q68 fried at the same time. Coincidence ?

Post by Dave »

Peter wrote:Off topic remark: The damping is intended, although I think impedance is less that the calculated figure. Low ESR would result in a very small, but visible noise on video output.
When the supply was bouncing, the cap acted a little like a choke. It was really nicely balanced.

It's a good system.


Post Reply