Wednesday, December 6, 2017

Powering GPU mining rigs


Since I started mining ethereum almost two years ago, I have found that power distribution is important not just for equipment safety, but also for system stability.  When I started mining I thought my rigs should be fine as long as I used a robust server PSU to power the GPUs, with heavy 16 or 18AWG cables.  After frying one motherboard and more than a couple ATX PSUs, I've learned a lot of careful design and testing is required.

Using Dell, IBM, or HP server power supplies for mining rigs is not a new idea, so I won't go into too much detail about them.  I do recommend making an interlock connector so the server PSU turns on at the same time as the motherboard.  I also recommend only connecting the server PSU to power the GPU PCIe power connectors, as they are isolated from the 12V supply for the motherboard.  If you try to power ribbon risers, the 12V from the ATX and server PSUs will be interconnected and can lead to feedback problems.  Server PSUs are very robust and unlikely to be harmed, but I have killed a cheap 450W ATX PSU this way.  If you use USB risers, they are isolated from the motherboard's 12V supply, and therefore can be safely powered from the server PSU.

In the photo above, you might notice the grounding wire connecting all the cards, which then connects to a server PSU.  I recently added this to the rig after measuring higher current flowing through two of the ground wires connected to the 6-pin PCIe power plugs.  As I mentioned in my post about GPU PCIe power connections, there are only two ground pins, with the third ground wire being connected to the sense pin.  With two ground pins and three power pins, the ground wires carry 50% more current than the 12V wires.  Although the ground wires weren't heating up from the extra current, the connector was.  Adding the ground bypass wire reduced the connector temperature to a reasonable level.

For ATX PSUs, I've used a few of the EVGA 500B, and do not recommend them.  While even my cheap old 300W power supplies use 18AWG wire for the hard drive power connectors, the SATA and molex power cables on the 500B are only 20AWG.  Powering more than one or two risers with a 20AWG cable is a recipe for trouble.  I burned the 12V hard drive power wire on two 500B supplies before I realized this.  I recently purchased a Rosewill 500W 80plus gold PSU that was on sale at Newegg, and it is much better than the EVGA 500B.  The Rosewill uses 18AWG wire in the hard drive cables, and it also has a 12V sense wire in the ATX power connector.  This allows it to compensate for the voltage drop in the cable from the PSU to the motherboard.  The sense wire is the thinner yellow wire in the photo below.

Speaking of voltage drop, I recommend checking the voltage at the PCIe power connector to ensure it is close to 12V.  Most of my cards do not have a back plate, so I can use a multi-meter to measure at the 12V pins of the the power connector where they are soldered to the GPU PCB.  I also recommend checking the temperature of power connectors since good quality low-resistance connectors are just as important as heavy gauge wires.  Warm connectors are OK, but if they so hot that you can't hold your fingers to them, that's a problem.

My last recommendation is for people in North America (and some other places) where 120V AC power is the norm.  Wire up the outlets for your mining rigs for 240 instead of 120.  Power supplies are slightly more efficient at 240V, and will draw half as much current compared to 120V.  Lower current draw means less line loss going to the power supply and therefore less heat generated in power cords and plugs.  Properly designed AC power cables and plugs should never overheat below 10-15 Amps, however I have seen melted and burned connectors at barely over 10A of steady current draw.


10 comments:

  1. Don't know if I am bad at searching through the internet, but this is the only clear statement that I found which was clearly and concisely establishes and is relevant to my problem.

    "If you use USB risers, they are isolated from the motherboard's 12V supply, and therefore can be safely powered from the server PSU."

    There is however a huge divergence of opinion about powering risers with a secondary PSU. Majority state with anecdotal evidence that it will cause your motherboard to go up in smoke.

    On this note, Is there a way to test, (short test or something) to make sure the risers I have, have their data (from the motherboard) and power (from a psu not powering the motherboard) isolated from each other?

    You have a very insightful and detailed blog

    ReplyDelete
    Replies
    1. I did a short test, and only the ground seemed to be common between the power and data connections on the powered (USB) risers.

      Meaning the ground of my Server psu and Atx psu is going to be shorted via the pcie slot on the motherboard.

      I put both Psus into running states and measured the voltage across the grounds of both PSUs. The potential difference varied from 0.03V to 0.14V. If these grounds are to be shorted, current will flow. Maybe that is what's causing smoky motherboards.

      Delete
    2. There should not be a voltage across grounds on two PSU's that is connected to the same grounded wallsocket. Should there?

      Delete
    3. Kore, the ground voltage from the PSU to the wall will be slightly above 0V, in proportion to the current. For example if the ground wire resistance is 10 milliOhms, with 10A of current, the voltage will be 0.1V (10 * .01).

      Delete
    4. Kore, there are two separate grounds being discussed: the DC ground return from the power supply output and the safety ground at the wall socket. The safety ground should be connected to the metal case of the power supply and the chassis. The DC ground may or may not be tied to the safety ground; you can check (with the power supply off) by measuring the resistance between the two.

      Delete
    5. Unknown, David, I'm not an expert in ATX power supply designs, but connecting DC ground (GND) to wall socket safety ground (a.k.a Earth ground) makes no sense in the design. Usually there's complete galvanic isolation of the DC output part from the grid through a transformer (with current sensing feedback through an optocoupler, which is another form of galvanic isolation). Which means there will be a potential difference between GND and Earth when power supply operates and this will probably trigger ground safety protection in the breaker in your house.

      When playing around with electronics, you might eventually touch the GND of your device's power supply and not get electrocuted (unless you live on 1st floor and it's a wet ground and/or your feet are wet, meaning you're not galvanically isolated from Earth anymore). This simply means that you're making your body's potential equal to the potential of the GND pin, but since you're galvanically isolated from Earth and also, you're not completing the circuit between "+12V" - there's no current flowing.

      Which leads us to the second point - you can connect GND of anything with anything if there's no current/circuit and there's galvanic isolation in place from Earth ground.

      If you connect several power supplies in parallel - like you do when you power your cards with a dedicated power supply(ies) and main ATX power supply, there's no problem connecting their GNDs, the problem is in making them work together.

      See https://www.cui.com/blog/power-supplies-in-series-or-parallel-for-increased-power for a basic explanation (they also have great additinal resources at the bottom of the page).

      Delete
    6. I suggest you read the cui.com link again: "the output voltages of ground referenced power supplies cannot be stacked on the outputs of other supplies." It's exactly because the DC ground is connected to earth that you have disconnect it when you stack 2 12V PSUs to generate 24V.
      All you have to do to prove this for yourself is take a multimeter on continuity mode and touch DC ground to the ground connector on the AC input.

      Delete
  2. How do you feel about grounding different GPUs connected to the same motherboard but using two different PSU. Like with a Asus B250 motherboard. With that board you could have up to 3 PSU. All together driving up to 19 GPUs. 7,6 and 6 gpu's pr PSU. Would it be a problem to ground this together? or are they grounded already through the motherboard connection? So then it would probably only take load of the ground through the motherboard...?

    ReplyDelete
    Replies
    1. I've only used motherboards with up to 6 PCIe slots using a single ATX main power connection.

      Delete