Ive been looking into doing this with an M720q(?) for faster local internet. What all did you have to do to get that in? One of my lenovo tinies is on the same spec level as yours (more ram but still). Mainly looking into doing this to get better performance for iscsi. My server has 2 10gb fiber but the tiny has gbe so im bottlenecked by that so it works but is slower than i want. Mainly because the price of high storage (12-30tb) ssd are insane and can’t justify it for a few. Id love to see more physical photos of the setup of it!
Oh, there isn't much to do. You need the riser, the part number is 01AJ940. Here is an example: https://www.ebay.com/itm/255866390773?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=kHxHJD3pR1e&sssrc=2349624&ssuid=INhwoPcqTkq&var=&widget_ver=artemis&media=COPY
The main problem is that the bracket isn't standard. If you have access to a 3d printer you can find brackets for several cards on thingiverse. Here is an example: https://www.thingiverse.com/thing:5869388
That is pretty much the whole deal. You can also install a 2.5g card in the wifi slot if you are happy with that level of performance.
daww shucks, guess I need to upload the STL file with a RJ45 spot as well.
I am also still working on my modular rack system but that is slightly delayed due to a bad case of windows proprietary software.
https://imgur.com/a/QOJph7Z
Ha! Hilarious. I picked up a 2.5g card to see if I could fit it, and I've been putting off putting the RJ45 spot in exactly that location cause my 3d modeling skills are... subpar.
Would love if you are willing to upload the updated one!
https://www.printables.com/model/413074-m720qm920p-baffle-for-cx312b-pro-connect-x-3-card
https://www.thingiverse.com/thing:5869388
Uploaded the STL to both sites, the version with the RJ45 is not as robust because there just is not as much room, it also lacks the same airflow. That said if you use a M3 screw to "tap" threads into the holes for the RJ45 board it should not split.
Also note that you need to cut back the little tab that the back screw goes into on the top of the case otherwise it wont close. I used a dremel to cut mine off, you can do it any way you want, but it has to be done to get all 3 additional network ports installed.
If anybody needs any of these baffles in the UK, I have had some professionally printed and include the brass nut inserts already done.
Full attribution for original design has been given to infinitevalence.
This may just help people without a 3D printer and brass nuts!
The professional print quality is superb!
[Ebay Listing - Lenovo m720q m920q m920x p330 Baffle for Mellanox CX312B-PRO 10gb sfp+ Card](https://www.ebay.co.uk/itm/235307069876?hash=item36c964f1b4:g:7O0AAOSwZBxlW2ce&amdata=enc%3AAQAIAAAAwFUSU5I1uja6rurVPnY57rn1PRQ9g%2FcV75SqyEroj6ukZ8hsLh0rRTOopd%2BIr2EogwEFBI8su98CSAkeLLDgX7J%2Fqr0iJBLwQFnDoRKPrllLNKkZHGruEImD5RNKwwz60jctaD9PyxTpFwomVPinS1fgHUU1WUENp4gWUTV1NCx%2BiLX8BTWfF22q0qaSo6U8%2Bwe%2Br6cSKn%2FMbVWLzWUbIzwbKuajH3grbniGR1vSfwWeKnIprTx3ph2Ulqe4aCL7Og%3D%3D%7Ctkp%3ABk9SR67a-9X9Yg)
Interesting that you flipped the RJ45. I was going to stack it board to board, and put a longer screw through both boards into a heat-set TI. Your solution seems less sketch.
I plan to once I get done with my testing. I need to make some more tweaks and finish my custom modular hot swap bay.
Right now it can't support the drive weight.
But it should be 10in rack compatible for the tiny mini micro nodes.
Cage is an old OEM icydock from chenbro.
Awesome! I do have a 3D printer i can use if needed. For the card instead of the wifi can i go any faster or is that the limit on that pcie bus I assume isnt the same x number as the other one.
No one currently makes a card better than 2.5g. The slot is only 2 lanes of pcie i believe, so that might be an issue, but also 10g is just a lot of performance to ask from that form factor. Here is the one I bought to play around with: https://www.ebay.com/itm/115598730980?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=3jahFM-7QUu&sssrc=2349624&ssuid=INhwoPcqTkq&var=&widget_ver=artemis&media=COPY
I have hooked it up and made sure it worked, but haven't performance tested it in any way.
That’s pretty cool and makes a lot of sense that 2.5g might be the upper limit due to that slot. For the form factor I got to be honest these are pretty powerful compared to their bigger brothers ive noticed too. 10g is a lot to ask of it though but with the processor at least being within 5? years of production its at least going to do more than some of the other ones.
When I mentioned form factor, I was talking about the 30mm m.2 card that fits in the wifi slot, not the machine itself. As I mention in the main post, I got 19gb/sec (x2) when testing the CX354, so I don't think the machine itself will have any trouble with 10gbe.
Oh good to know! Theoretically you could run both off 10gb and aggregate that for either failover or can it be setup in LACP or whatever its called to have each one bring 10gbe to combine them for more?
>Yeah, I imagine it would handle bonding dual 10gb ports without issue. Or pfsense/opnsense running as a firewall for up to 10gb Internet connections, etc.
You make a good point. I have 10gb fiber incoming and I am able to utilize all of it with the UDMSE but I am honestly looking for a reason to jump ship of their Routing. Don't get me wrong it works pretty ok for some home use but when im running 2 VPNS. (1 wireguard one S2S to my parents) plus 5 VLANS ontop of the regular LAN it starts to slow down a bit. (150 devices on home network at one given time max). Their switching I have not had many issues with besides the simple reset.
Hey there... I'm the "somebody" who started that thread about my dual SFP+ adapter with twin 10GBASE-T transceivers running hot. I didn't specify it in my original post, but my adapter is a Mellanox CX312A, which is interesting because you identified that model as running especially hot.
After I posted that question I ran my Lenovo M720Q with the cover on for about a week, slinging 1.7Gbps from my ISP to my LAN without incident but I could tell the transceivers were running at or over the 70C maximum rating. Suddenly one evening I lost all internet and figured out one of my SFP modules was locked up. When I pulled it out it literally scorched my fingers. The other one was much cooler. Luckily after letting it sit for half an hour it worked fine again.
After experiencing this I dug up an old 4" 120VAC PC fan which runs very quiet and just placed it over the rear of the Lenovo and the exposed part of the transceivers. I don't have a baffle mounted so the fan also forces some air into the opening and over the CX312A heat sink. Since doing that the transceivers, which always felt really hot even before one of them locked up, now barely feel warm. Likewise, if I run for a while and then pop the cover off the box the Mellanox heat sink is very comfortable to the touch.
I was concerned about the Mellanox adapter and transceivers pumping so much heat into the enclosure that the other components would run too hot, but since adding the janky active cooling upgrade my six CPU cores are running at 46/47C and my SSD is chillin' at 37C. I'm confident that my additional cooling hack is going to be sufficient to keep the adapter and SFP modules running smoothly.
BTW, you mentioned that there is a CLI tool to read the chip temperature on ConnectX-3 boards. I looked for something like that but didn't find anything. Do you have a link?
Anyway, thanks for doing all this research. I'm sure it will be useful to many others than just me.
The CLI tool came in the "Mellanox Firmware Tools" package. I had to download it to reconfigure the CX354A card from Infiniband to Ethernet, and it had a bunch of other stuff in it. The batch file mget\_temp.bat was what you are looking for.
It was kind of a pain to find the firmware tools marked as for ConnectX-3 cards, as they have mostly removed them from the site. I don't know if the newer tools work, but they probably do? I'd try and get you a direct link, but I'm getting access denied from [mellanox.com](https://mellanox.com) and [nvidia.com](https://nvidia.com) right now. Not sure if it is a problem on their end or mine. I'll try and remember to revisit tomorrow morning.
My M720Q is so quiet I have trouble hearing the fan even with my ear next to the case. Yes, half-height 2.5GBe adapter cards will work too. I also replaced the WiFi card with a 2.5GBe M.2 RG45 adapter and that works perfectly as well. I currently have four RG45 copper sockets available... one 1GBe/100MBe built-in one on the mainboard, two 10GBe/5GBe/2.5GBe/1GBe/100MBe SFP+ transceivers installed in my Mellanox adapter, and one 2.5GBe/1GBe/100MBe socket from the M.2 adapter.
The Mellanox is [this one](https://www.ebay.com/itm/394334880480). The SFP+ transceivers are [this one](https://www.aliexpress.com/item/1005003029257307.html) (I got them with a coupon for $41 CDN each). I also needed [this riser](https://www.aliexpress.com/item/1005004237346189.html) to plug in the Mellanox. Finally I used [this 2.5GBe M.2 adapter](https://www.amazon.ca/dp/B0B6RJ1MCK) to replace the WiFi card. You can get that same card on eBay for about $25 USD.
I was going to get someone to 3D print a bracket for me, but haven't got around to it yet. It works ok the way it is. I'll probably do something about it eventually.
Did you source all the same parts I did?
I have no experience with the ConnectX-4 but it wouldn't surprise me if it's a bit more efficient that my ConnectX-3 card. It's still going to run pretty hot though. If you use Base-T (copper) transceivers they run very hot too. Fiber and DAC transceivers run a lot cooler than the copper ones.
What would happen if you just used the SFP+ with a fiber link to another SFP+ on a switch (e.g Aruba 2500)? Would be an acceptable temp? I read fiber uses a lot less wattage.
So putting this all together (thanks for the data/post!) what NIC would you recommend?
Sounds like maybe the CX354? Or general avoidance of RJ45 transceivers overall?
I know you said you wouldn’t recommend the 82599ES, but also that the CX312A ran quite hot. I hear the CX312B might be better thermally?
I use the Rev 2.1 versions of the Supermicro card, mostly because the shorter length allows me to sneak an mSATA boot drive in. I know it’s running hot, but never really figured it was THAT hot. I’m also using DACs where possible…
Very interesting, I’m digging all these Tiny related posts.
tl;dr - CX312B.
Based on what I found, I would get a CX312B if I was buying one. They all run around the same price on ebay, so that is the one I would choose. It is also the most compact (at the ports), so there is room to squeeze in a 2.5g rj45 port hanging off the wifi m.2 slot, which is a bonus.
My CX312A ran quite hot, but it seems suspect since all the other Mellanox cards from the same generation were fine. If someone already had one laying around, I would recommend they try it before replacing it. I think mine is just running hotter than usual.
Certainly one of my take-aways was that the CX354 ran at a fine temperature, so if I had switching for 40gbe available, I would run that without concern.
The SuperMicro cards are definitely the most compact inside the case, so if you wanted to add a blower fan or something inside the case, they leave the most room for experimentation in that area.
Finally, yes, I would avoid RJ45 transceivers without additional cooling. If I had an Intel x540 or x550 rj45 base-T card laying around, I definitely would have tried it. I would guess that the purpose built cards run cooler than a transceiver based solutions?
Thanks for the reply!
I have some CX354s with SFP+ adapters but I may sell those since I also don’t have 40GbE switching. My 82599ES cards seem ok, but will keep and eye out for CX312Bs for cheap.
Thanks again!
A few tips:
1. Use DACs, because optics get very hot and consume relatively much
2. Use 10Gbit cards with active cooling, because they get way too hot for such an enclosure
I'm running a Intel X520-DA2 10Gb inside a M920x tiny with an i7 8700 and 32gb of ram running xcp-ng. Do you know how to get the temps from the card ?
I tried but only getting the transceiver temps :/
That card is using the Intel 82599 as well, very similar to the SuperMicro card I tested. I found a post from Intel Embedded tech confirming that the chip doesn't have an internal temp sensor. It's possible for them to put a temp sensor on the card external to the chip, but I would think that is unusual, so I expect there isn't a way to get the temp from the card. I don't know for certain either way though.
Getting transceiver temps still seems better than nothing though, how hot are they running? The operating temperature for that card is 0C-55C, just like the supermicro. [https://www.intel.com/content/www/us/en/support/articles/000032893/ethernet-products.html](https://www.intel.com/content/www/us/en/support/articles/000032893/ethernet-products.html)
10GBASE-T transceivers can run pretty hot. Sometimes it is even advised not to place them to adjacent SFP+ cages.
Just for example:
MikroTik S+RJ10 10GBASE-T consumes 2.7 W,
MikroTik S+85DLC03D Optical MM (300 m) consumes 0.8 W
Yes. That was part of my motivation for doing this experiment. I showed a +10W usage between a DAC cable and 2 RJ45 transceivers w/Cat6a when measuring the entire machine. I was using Wiitek brand, but 2.7 seems low based on what I recorded.
Even still though, I'd be surprised if the transceivers ran hotter than the chip itself.
Here is the deal. I stressed the SuperMicro card pretty hard. I ran it at full throughput on both ports for over an hour with the cover off the m920q, and it didn't fail. When I tried it with the rj45 transceivers, it still ran for 12-14 minutes before it failed. Even after it failed, I just pulled the transceivers and let it cool down, then rebooted the machine and it worked fine again.
So, if I was purchasing a card to put in one of these machines, I would definitely not choose an Intel 82599 based card. But, if I already had one on hand, I'd probably just run it until it died and it would probably outlast me. Though I definitely wouldn't run it with RJ45 transceivers.
If you want a good reason to replace it, the spec for the card tops at 55C and the spec for the 82599 chip tops at 70C. When I checked the SuperMicro card, the \*heatsink\* was 160f (71C) at idle, so no question it was out of spec.
Depending on the clearance and proximity to the CPU fan shroud, you could possibly 3D Print a custom fan shroud to divert some air flow towards the NIC heat sink.
Those blower fans move some good air.
tbh there is not that much room. I was going to work on a 40mm blower fan mod and shroud at some point. Need to find out what the voltage is for the CPU fans and figure out how to add the blower to it so it will spin up with chassis temperature.
Ive been looking into doing this with an M720q(?) for faster local internet. What all did you have to do to get that in? One of my lenovo tinies is on the same spec level as yours (more ram but still). Mainly looking into doing this to get better performance for iscsi. My server has 2 10gb fiber but the tiny has gbe so im bottlenecked by that so it works but is slower than i want. Mainly because the price of high storage (12-30tb) ssd are insane and can’t justify it for a few. Id love to see more physical photos of the setup of it!
Oh, there isn't much to do. You need the riser, the part number is 01AJ940. Here is an example: https://www.ebay.com/itm/255866390773?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=kHxHJD3pR1e&sssrc=2349624&ssuid=INhwoPcqTkq&var=&widget_ver=artemis&media=COPY The main problem is that the bracket isn't standard. If you have access to a 3d printer you can find brackets for several cards on thingiverse. Here is an example: https://www.thingiverse.com/thing:5869388 That is pretty much the whole deal. You can also install a 2.5g card in the wifi slot if you are happy with that level of performance.
daww shucks, guess I need to upload the STL file with a RJ45 spot as well. I am also still working on my modular rack system but that is slightly delayed due to a bad case of windows proprietary software. https://imgur.com/a/QOJph7Z
Ha! Hilarious. I picked up a 2.5g card to see if I could fit it, and I've been putting off putting the RJ45 spot in exactly that location cause my 3d modeling skills are... subpar. Would love if you are willing to upload the updated one!
https://www.printables.com/model/413074-m720qm920p-baffle-for-cx312b-pro-connect-x-3-card https://www.thingiverse.com/thing:5869388 Uploaded the STL to both sites, the version with the RJ45 is not as robust because there just is not as much room, it also lacks the same airflow. That said if you use a M3 screw to "tap" threads into the holes for the RJ45 board it should not split. Also note that you need to cut back the little tab that the back screw goes into on the top of the case otherwise it wont close. I used a dremel to cut mine off, you can do it any way you want, but it has to be done to get all 3 additional network ports installed.
Thanks for that! Excited to give it a try.
If anybody needs any of these baffles in the UK, I have had some professionally printed and include the brass nut inserts already done. Full attribution for original design has been given to infinitevalence. This may just help people without a 3D printer and brass nuts! The professional print quality is superb! [Ebay Listing - Lenovo m720q m920q m920x p330 Baffle for Mellanox CX312B-PRO 10gb sfp+ Card](https://www.ebay.co.uk/itm/235307069876?hash=item36c964f1b4:g:7O0AAOSwZBxlW2ce&amdata=enc%3AAQAIAAAAwFUSU5I1uja6rurVPnY57rn1PRQ9g%2FcV75SqyEroj6ukZ8hsLh0rRTOopd%2BIr2EogwEFBI8su98CSAkeLLDgX7J%2Fqr0iJBLwQFnDoRKPrllLNKkZHGruEImD5RNKwwz60jctaD9PyxTpFwomVPinS1fgHUU1WUENp4gWUTV1NCx%2BiLX8BTWfF22q0qaSo6U8%2Bwe%2Br6cSKn%2FMbVWLzWUbIzwbKuajH3grbniGR1vSfwWeKnIprTx3ph2Ulqe4aCL7Og%3D%3D%7Ctkp%3ABk9SR67a-9X9Yg)
Interesting that you flipped the RJ45. I was going to stack it board to board, and put a longer screw through both boards into a heat-set TI. Your solution seems less sketch.
Dude, you really need to share this. What's the HDD cage?
I plan to once I get done with my testing. I need to make some more tweaks and finish my custom modular hot swap bay. Right now it can't support the drive weight. But it should be 10in rack compatible for the tiny mini micro nodes. Cage is an old OEM icydock from chenbro.
Awesome! I do have a 3D printer i can use if needed. For the card instead of the wifi can i go any faster or is that the limit on that pcie bus I assume isnt the same x number as the other one.
No one currently makes a card better than 2.5g. The slot is only 2 lanes of pcie i believe, so that might be an issue, but also 10g is just a lot of performance to ask from that form factor. Here is the one I bought to play around with: https://www.ebay.com/itm/115598730980?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=3jahFM-7QUu&sssrc=2349624&ssuid=INhwoPcqTkq&var=&widget_ver=artemis&media=COPY I have hooked it up and made sure it worked, but haven't performance tested it in any way.
That’s pretty cool and makes a lot of sense that 2.5g might be the upper limit due to that slot. For the form factor I got to be honest these are pretty powerful compared to their bigger brothers ive noticed too. 10g is a lot to ask of it though but with the processor at least being within 5? years of production its at least going to do more than some of the other ones.
When I mentioned form factor, I was talking about the 30mm m.2 card that fits in the wifi slot, not the machine itself. As I mention in the main post, I got 19gb/sec (x2) when testing the CX354, so I don't think the machine itself will have any trouble with 10gbe.
Oh good to know! Theoretically you could run both off 10gb and aggregate that for either failover or can it be setup in LACP or whatever its called to have each one bring 10gbe to combine them for more?
>Yeah, I imagine it would handle bonding dual 10gb ports without issue. Or pfsense/opnsense running as a firewall for up to 10gb Internet connections, etc.
You make a good point. I have 10gb fiber incoming and I am able to utilize all of it with the UDMSE but I am honestly looking for a reason to jump ship of their Routing. Don't get me wrong it works pretty ok for some home use but when im running 2 VPNS. (1 wireguard one S2S to my parents) plus 5 VLANS ontop of the regular LAN it starts to slow down a bit. (150 devices on home network at one given time max). Their switching I have not had many issues with besides the simple reset.
Hey there... I'm the "somebody" who started that thread about my dual SFP+ adapter with twin 10GBASE-T transceivers running hot. I didn't specify it in my original post, but my adapter is a Mellanox CX312A, which is interesting because you identified that model as running especially hot. After I posted that question I ran my Lenovo M720Q with the cover on for about a week, slinging 1.7Gbps from my ISP to my LAN without incident but I could tell the transceivers were running at or over the 70C maximum rating. Suddenly one evening I lost all internet and figured out one of my SFP modules was locked up. When I pulled it out it literally scorched my fingers. The other one was much cooler. Luckily after letting it sit for half an hour it worked fine again. After experiencing this I dug up an old 4" 120VAC PC fan which runs very quiet and just placed it over the rear of the Lenovo and the exposed part of the transceivers. I don't have a baffle mounted so the fan also forces some air into the opening and over the CX312A heat sink. Since doing that the transceivers, which always felt really hot even before one of them locked up, now barely feel warm. Likewise, if I run for a while and then pop the cover off the box the Mellanox heat sink is very comfortable to the touch. I was concerned about the Mellanox adapter and transceivers pumping so much heat into the enclosure that the other components would run too hot, but since adding the janky active cooling upgrade my six CPU cores are running at 46/47C and my SSD is chillin' at 37C. I'm confident that my additional cooling hack is going to be sufficient to keep the adapter and SFP modules running smoothly. BTW, you mentioned that there is a CLI tool to read the chip temperature on ConnectX-3 boards. I looked for something like that but didn't find anything. Do you have a link? Anyway, thanks for doing all this research. I'm sure it will be useful to many others than just me.
The CLI tool came in the "Mellanox Firmware Tools" package. I had to download it to reconfigure the CX354A card from Infiniband to Ethernet, and it had a bunch of other stuff in it. The batch file mget\_temp.bat was what you are looking for. It was kind of a pain to find the firmware tools marked as for ConnectX-3 cards, as they have mostly removed them from the site. I don't know if the newer tools work, but they probably do? I'd try and get you a direct link, but I'm getting access denied from [mellanox.com](https://mellanox.com) and [nvidia.com](https://nvidia.com) right now. Not sure if it is a problem on their end or mine. I'll try and remember to revisit tomorrow morning.
Thanks.
I think the download you want is in here: https://network.nvidia.com/support/firmware/connectx3en/
Thanks. Seems to be just the firmware image. I can't see any tools there.
My bad. I think this is the tool download: https://network.nvidia.com/products/adapter-software/firmware-tools/
How noisy the 720q gets with this cards ? any idea of 2.5 cards that might work there too ?
My M720Q is so quiet I have trouble hearing the fan even with my ear next to the case. Yes, half-height 2.5GBe adapter cards will work too. I also replaced the WiFi card with a 2.5GBe M.2 RG45 adapter and that works perfectly as well. I currently have four RG45 copper sockets available... one 1GBe/100MBe built-in one on the mainboard, two 10GBe/5GBe/2.5GBe/1GBe/100MBe SFP+ transceivers installed in my Mellanox adapter, and one 2.5GBe/1GBe/100MBe socket from the M.2 adapter.
can you share models of cards ?
The Mellanox is [this one](https://www.ebay.com/itm/394334880480). The SFP+ transceivers are [this one](https://www.aliexpress.com/item/1005003029257307.html) (I got them with a coupon for $41 CDN each). I also needed [this riser](https://www.aliexpress.com/item/1005004237346189.html) to plug in the Mellanox. Finally I used [this 2.5GBe M.2 adapter](https://www.amazon.ca/dp/B0B6RJ1MCK) to replace the WiFi card. You can get that same card on eBay for about $25 USD.
Thanks, got it all and it works perfect. did you find a way to mount the 2.5 M2 port somehow ?
I was going to get someone to 3D print a bracket for me, but haven't got around to it yet. It works ok the way it is. I'll probably do something about it eventually. Did you source all the same parts I did?
Yeah. Everything. Thanks for the tips. Just different SFPs.
I am thinking about the Mellanox ConnectX-4 LC. Would you exptect that one to run cooler?
I have no experience with the ConnectX-4 but it wouldn't surprise me if it's a bit more efficient that my ConnectX-3 card. It's still going to run pretty hot though. If you use Base-T (copper) transceivers they run very hot too. Fiber and DAC transceivers run a lot cooler than the copper ones.
>ConnectX-4 LC did you end up trying this? considering it, myself.
Not yet, still waiting for a cheap tiny to show up in my eBay feed first 🙃😂
does it have to be the 920 or would the 720 work? do you have a minimum CPU in mind? sometimes deals come across my path.
What would happen if you just used the SFP+ with a fiber link to another SFP+ on a switch (e.g Aruba 2500)? Would be an acceptable temp? I read fiber uses a lot less wattage.
Fiber transceivers run a **lot** cooler than Base-T (RJ45). The coolest of all is DAC which has no or almost no electronics inside.
Always cool to see people running BOINC, join us at /r/BOINC4Science
So putting this all together (thanks for the data/post!) what NIC would you recommend? Sounds like maybe the CX354? Or general avoidance of RJ45 transceivers overall? I know you said you wouldn’t recommend the 82599ES, but also that the CX312A ran quite hot. I hear the CX312B might be better thermally? I use the Rev 2.1 versions of the Supermicro card, mostly because the shorter length allows me to sneak an mSATA boot drive in. I know it’s running hot, but never really figured it was THAT hot. I’m also using DACs where possible… Very interesting, I’m digging all these Tiny related posts.
tl;dr - CX312B. Based on what I found, I would get a CX312B if I was buying one. They all run around the same price on ebay, so that is the one I would choose. It is also the most compact (at the ports), so there is room to squeeze in a 2.5g rj45 port hanging off the wifi m.2 slot, which is a bonus. My CX312A ran quite hot, but it seems suspect since all the other Mellanox cards from the same generation were fine. If someone already had one laying around, I would recommend they try it before replacing it. I think mine is just running hotter than usual. Certainly one of my take-aways was that the CX354 ran at a fine temperature, so if I had switching for 40gbe available, I would run that without concern. The SuperMicro cards are definitely the most compact inside the case, so if you wanted to add a blower fan or something inside the case, they leave the most room for experimentation in that area. Finally, yes, I would avoid RJ45 transceivers without additional cooling. If I had an Intel x540 or x550 rj45 base-T card laying around, I definitely would have tried it. I would guess that the purpose built cards run cooler than a transceiver based solutions?
Thanks for the reply! I have some CX354s with SFP+ adapters but I may sell those since I also don’t have 40GbE switching. My 82599ES cards seem ok, but will keep and eye out for CX312Bs for cheap. Thanks again!
Do you mean that you are running a msata drive and couldn't fit a 2.5" drive?
A few tips: 1. Use DACs, because optics get very hot and consume relatively much 2. Use 10Gbit cards with active cooling, because they get way too hot for such an enclosure
I guess a more general question: Should latency and temps not go down when using fiber instead of RJ45 or DAC?
I'm running a Intel X520-DA2 10Gb inside a M920x tiny with an i7 8700 and 32gb of ram running xcp-ng. Do you know how to get the temps from the card ? I tried but only getting the transceiver temps :/
That card is using the Intel 82599 as well, very similar to the SuperMicro card I tested. I found a post from Intel Embedded tech confirming that the chip doesn't have an internal temp sensor. It's possible for them to put a temp sensor on the card external to the chip, but I would think that is unusual, so I expect there isn't a way to get the temp from the card. I don't know for certain either way though. Getting transceiver temps still seems better than nothing though, how hot are they running? The operating temperature for that card is 0C-55C, just like the supermicro. [https://www.intel.com/content/www/us/en/support/articles/000032893/ethernet-products.html](https://www.intel.com/content/www/us/en/support/articles/000032893/ethernet-products.html)
transceiver is running between 50 and 55C on idle. I haven't stress test it yet but seems te be working.
Yeah, that seems hot, but not hot enough to worry too much? I would be very surprised if the transceiver is running hotter than the chip though.
10GBASE-T transceivers can run pretty hot. Sometimes it is even advised not to place them to adjacent SFP+ cages. Just for example: MikroTik S+RJ10 10GBASE-T consumes 2.7 W, MikroTik S+85DLC03D Optical MM (300 m) consumes 0.8 W
Yes. That was part of my motivation for doing this experiment. I showed a +10W usage between a DAC cable and 2 RJ45 transceivers w/Cat6a when measuring the entire machine. I was using Wiitek brand, but 2.7 seems low based on what I recorded. Even still though, I'd be surprised if the transceivers ran hotter than the chip itself.
Here is the deal. I stressed the SuperMicro card pretty hard. I ran it at full throughput on both ports for over an hour with the cover off the m920q, and it didn't fail. When I tried it with the rj45 transceivers, it still ran for 12-14 minutes before it failed. Even after it failed, I just pulled the transceivers and let it cool down, then rebooted the machine and it worked fine again. So, if I was purchasing a card to put in one of these machines, I would definitely not choose an Intel 82599 based card. But, if I already had one on hand, I'd probably just run it until it died and it would probably outlast me. Though I definitely wouldn't run it with RJ45 transceivers. If you want a good reason to replace it, the spec for the card tops at 55C and the spec for the 82599 chip tops at 70C. When I checked the SuperMicro card, the \*heatsink\* was 160f (71C) at idle, so no question it was out of spec.
Depending on the clearance and proximity to the CPU fan shroud, you could possibly 3D Print a custom fan shroud to divert some air flow towards the NIC heat sink. Those blower fans move some good air.
tbh there is not that much room. I was going to work on a 40mm blower fan mod and shroud at some point. Need to find out what the voltage is for the CPU fans and figure out how to add the blower to it so it will spin up with chassis temperature.
Hey, it's a little off-topic but I'm trying to get M920q for home lab but I'm not sure if it has an automatic power on - in case of a power loss.
It does. In needs to be set in bios. Default setting is last state.