From crypto mining to fog computing
Cudo started as a cryptocurrency mining platform, providing a desktop application that made it easier for users to mine the most profitable coins according to their hardware, the network and other factors like exchange rates. The original desktop app has evolved into a dedicated operating system used by professional crypto miners rather than hobbyists using their home machine. Now the company is expanding into fog computing, a cloud computing platform that allows hardware owners to host workloads when they have extra capacity, allowing workload owners to run on any ISP that is participating by running on Cudo Cloud.
Cudo’s clients tend to be companies with idle hardware sitting around, like gaming cafes, large data centers and dedicated mining farms. As they move more towards fog computing they expect an increasing number of customers to be data center owners.
“We knew we were going to hit the limits of it pretty quickly,” explained Richard Poole, lead architect at Cudo Ventures about their homegrown solution for users to monitor their infrastructure. “We just needed to release something sooner rather than later.” The monitoring system was MySQL-based, using sharded tables and date partitions to let them drop data that was over 30 days old. “We knew we were going to hit the limits of what our MySQL server could offer pretty quickly.”
One of the biggest problems with the solution was that there was no ability to aggregate any of the time series data, as well as very limited data retention — one minute resolution retained for an hour, 10 minute resolution for seven days and one hour resolution for 30 days. “The really strictly defined resolution windows made the time series data not that useful for actually debugging issues,” Poole said. The total inability to aggregate any of the time series data also made it impossible to get an overview of health and cost metrics on the entire fleet, something customers were interested in doing but not able to.
Then there was the internal monitoring. “We were running pretty blind on a lot of the services,” Poole said. “We had the baseline Kubernetes metrics being exposed and we checked those only when there was an operational issue.” The internal monitoring was managed with Prometheus. This setup made it practically impossible to act proactively. One issue specific to cryptotrading, however, is that sometimes the cryptocurrency networks have to fork. Until someone noticed the fork, the node would stop synchronizing with the network and users would be unable to withdraw from the platform. With Cudo’s Prometheus-based monitoring system, the team would often first discover the fork when users started complaining.