IDC predicts that public cloud will account for almost 32 percent of IT infrastructure spending in 2020, while private cloud will take a 20-percent slice of the budget pie, totaling more than half of all infrastructure spending for the first time. The rest will go to traditional data centers.
By using a mix of on-premises, private cloud and public cloud services, organizations gain the maximum amount of business flexibility from cloud architectures. This helps maximize budget efficiency and performance goals at the same time. In a nutshell, hybrid cloud is a cloud computing environment that uses a mix of on-premises, private cloud and public cloud services, with orchestration between the two platforms.
Organizations are learning as they go about the pluses and minuses of deploying a hybrid cloud approach. In this article, we will look at design elements you can use to ensure your hybrid cloud delivers the performance, flexibility and scalability you need.
The Secret Sauce: Scale-Out NAS
Storage must be addressed for the hybrid cloud approach, and a scale-out NAS is the answer. Since hybrid cloud architectures are relatively new to the market — and even newer in full-scale deployment — many organizations are unaware of the importance of consistency in a scale-out NAS. Many environments are eventually consistent, meaning that files that you write to one node are not immediately accessible from other nodes. This can be caused by not having a proper implementation of the protocols, or not tight enough integration with the virtual file system. The opposite of that is being strictly consistent: files are accessible from all nodes at the same time. Compliant protocol implementations and tight integration with the virtual file system is a good recipe for success.
There are three layers that a state-of-the-art hybrid cloud architecture using a scale-out NAS approach should be based on. Each server in the cluster will run a software stack based on these layers.
The persistent storage layer comes first. This layer is based on an object store, which provides advantages like extreme scalability. However, the layer must be strictly consistent in itself. The second layer is the virtual file system, the heart of any scale-out NAS. It is in this layer that features like caching, locking, tiering, quota and snapshots are handled. The third layer contains the protocols like SMB and NFS but also integration points for hypervisors, for example.
It is very important to keep the architecture symmetrical and clean. If you manage to accomplish that, many future architectural challenges will be much easier to solve.
That first layer merits further examination. Because the storage layer is based on an object store, we can now easily scale our storage solution. With a clean and symmetrical architecture, we can reach exabytes of data and trillions of files.
One of the storage layer’s jobs is to ensure redundancy, so a fast and effective self-healing mechanism is needed. To keep the data footprint low in the data center, the storage layer needs to support different file encodings. Some are good for performance and some for reducing the footprint.
The Role of Metadata
In a virtual file system, metadata are pieces of information that describe the structure of the file system. Metadata is a crucial piece of the virtual file system. For example, one metadata file can contain information about what files and folders are contained in a single folder in the file system. That means that we will have one metadata file for each folder in our virtual file system. As the virtual file system grows, we will get more and more metadata files.
It is a practice among some users to centrally store their metadata. For smaller set-ups that might be a good solution, but here we are talking about scale-out. So, let’s look at where not to store metadata. Storing metadata in a single server can cause poor scalability, poor performance and poor availability. Since our storage layer is based on an object store, a better place to store all our metadata is in the object store – particularly when we are talking about large quantities of metadata. This will ensure good scalability, good performance and good availability.
The Role of Cache
Caching devices are a necessary ingredient to boost performance in software-defined storage solutions. From a storage solution perspective, both speed and size matter as well as price, and finding the sweet spot is important. For an SDS solution, it is also important to protect the data at a higher level by replicating the data to another node before destaging the data to the storage layer.
It becomes increasingly important to support multiple file systems and domains as the storage solution grows in both capacity and features, particularly in virtual or cloud environments. Supporting multiple file systems is also very important. Different applications and use cases prefer different protocols. And sometimes, it is also necessary to be able to access the same data across different protocols.
Hyper-Convergence and the Virtual World
Hypervisors must be accommodated in cloud settings. Therefore, the scale-out NAS needs to be able to run as hyper-converged as well. Being software-defined makes sense here.
In a scenario such as this where there is a flat architecture with no external storage systems, the scale-out NAS must be able to run as a virtual machine and make use of the hypervisor host’s physical resources. The guest virtual machine’s (VM) own images and data will be stored in the virtual file system that the scale-out NAS provides. The guest VMs can use this file system to share files between them, making it perfect for VDI environments as well.
Also in this scenario, it is important to support many protocols. This is because in a virtual environment, there are many different applications running, having different needs for protocols. By supporting many protocols, we keep the architecture flat and have the ability to share data between applications that speak different protocols, to some extent.
A very flexible and useful storage solution results, one that:
- is software-defined;
- supports both fast and energy-efficient hardware;
- has an architecture that allows us to start small and scale up;
- supports bare-metal as well as virtual environments; and
- has support for all major protocols.
The Importance of Sharing
In today’s geographically distributed companies, each site has its own independent file system. A likely scenario is that different offices have a need for both a private area and an area that they share with other branches. So, only parts of the file system will be shared with others.
Choosing a portion of a file system to let others mount at any given point in the other file systems provides the flexibility needed to scale the file system outside the four walls of the office, making sure that the synchronization is made at the file system level in order to have a consistent view of the file system across sites. Being able to specify different file encodings at different sites is useful, for example, if one site is used as a backup target.
The Future of Storage
Taken together, these elements constitute a new kind of hybrid cloud system that delivers clean, efficient and linear scaling up to exabytes of data. One of its greatest benefits is the single file system that spans all servers, as it provides multiple entry points and thus removes possible performance bottlenecks. This next-generation solution is capable of flexible scale-out while offering high performance through its support for flash. Data center expansion becomes easier and more affordable as you adjust storage capacity to meet current and future demands.
About the Author
Stefan Bernbo is the founder and CEO of Compuverde. For 20 years, Stefan has designed and built numerous enterprise scale data storage solutions designed to be cost-effective for storing huge data sets. From 2004 to 2010, Stefan worked within this field for Storegate, the wide-reaching internet-based storage solution for consumer and business markets, with the highest possible availability and scalability requirements. Previously, Stefan has worked with system and software architecture on several projects with Swedish giant Ericsson, the world-leading provider of telecommunications equipment and services to mobile and fixed network operators.