12/16/2023 0 Comments Redshift serverless costArguably though, the exploratory work may not be a good fit in an environment that dev/test is also performed. Highly utilised environments where development, test and exploratory work is performed with real data and with little idle time during working hours is better served from Provisioned Redshift, with the cluster started and paused at the start/end of the day. Our own work with clients has shown that significant savings could be seen when using Serverless over Provisioned Redshift in development environments (where usage was sporadic and experimental in nature) and in production environments where product pipelines ran in a fairly predictable manner, supplemented by small volumes of ad hoc usage (to give a pattern of high spikes in demand and plenty of idle time). There is nothing to prevent some workloads being run on Serverless capacity and others on Provisioned Redshift. Or, if the workloads require different levels of compute capacity they could be run in their own workgroups with each having the required level of base RPUs to suit performance/costs a la separate Redshift clusters, but without the setup/management/startup overhead. In the above scenario the extra demand from parallel workloads would be served by the built in fast auto scaling functionality. Another strategy could be to have separate clusters triggering different workloads. From experience, it can take time for the offloaded queries to begin to run.Īlternatively, more nodes could be requested (via elastic resizing) but again there is some delay before they are available. Provisioned Redshift has the option of Concurrency Scaling whereby once all internal queues have queries running, will offload queued queries to additional capacity outside of the cluster. However, perhaps workloads are triggered by upstream processes on an ad hoc basis and could run in parallel, demanding more resources than the default setup can cope with, or not run at all. The predictable nature of this workload could allow the cluster to be rightsized for better costs. Hence if your workload is a long running batch process that starts/ends around the same time each day, one approach could be to have a process that starts the (existing) cluster, runs the batch and then pauses the cluster to prevent any more running costs (storage costs are separate). In terms of the first point, recall that Provisioned Redshift will charge when the cluster is running, regardless of whether it is in use, whereas Serverless only charges when queries are running. Redshift Serverless may be cost effective for “spiky” or less predicable workloads or where fast scaling of resources is required (for example, a process requires a certain SLA that can be at risk when parallel workloads are inflight) Provisioned Redshift may be cost effective if it is sized correctly and utilisation is high and constant Generally though, the following guidelines can be used: Nor is anything published by AWS to state how each level of RPUs (Redshift Processing Units) compares to cluster node types. It should be noted first that there are no known plans for AWS to stop the provisioning of Redshift clusters and there is nothing stated in the documentation that Serverless will perform better or cheaper than any existing setup: it will depend on the number and types of workloads.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |