Disk storage performance trick to achieve almost speed of RAM using RAID 100 alike and 2 PERC Controllers.
Server Tweaks for compensate RAM using RAID - Making Swap your friend.
Disk storage performance trick to achieve almost speed of RAM using RAID 100 alike and 2 PERC Controllers.
In: Memory Management, OS Internals, ServersThe problem with exponential growth of data and memory consumption in Machine Learning, especially building models is somewhat problematic, or at least it will be in near future. Most of Industry Grade servers, such as Dell PowerEdge’s that I’m using supports up to maximum 1TB of Ram. With about 500G of data growth daily, we can expect industry grade servers soon are not going to be enough for running ML model build node feeding these TPU's requiring a fresh dataset every now and then built fast.
Let's see what we saw in real life. The "Swap is your friend this time" Approach:
- 2 X PERC H740P handling 8x 1TB EVO SSD’s each
- RAID 1 on the low layer for redundancy and persistence.
- RAID 0 on top, with software raid bridge on top of two controllers.
How do we stand with SWAP usage backed up by that RAID
Here we are with IOWait of ML model build node (in production)
I think i can totally live with this level of iowait. 1TB of RAM-alike Storage is definitely to serve for a long run. Moreover, no more re-syncing and failing to allocate enough memory.
Node is up, swap like a crazy, yet IOWait performs close to non-blocking. So far, everything looks and works as expected to my surprise, with customer grade SSD's,
that are not even SAS, just regular SATA with which PERC is backwards compatible.
This is one of approaches to address the increased amount of data. From there, there's a custom built "HAProxy" performing a split of traffic between multiple nodes based on the
block number.
Who would ever say SWAP could be your friend? And yes, you can scale this up with multiple layers on top of each other.
Join the talk
Share your toughts on the subject or whatever you would like to know.