123 Dyno extends Heroku with the ability to autoscale via CPU load by simply installing the add-on and configuring. While [controllable response time autoscaling](/blog/drastically-improved-response-time-autoscaling)  is useful for a large number of applications especially older concurrent frameworks, ***CPU autoscaling ensures that you get the most out of the dynos you're paying for***.  This can drastically reduce your bill by only paying for compute power you use versus the response time your app returns in stock autoscaling. This is particularly true of event loop paradigms like Node.js and Tornado where the application can handle incredible amounts of requests on little compute power, response times usually bottlenecked elsewhere. 


The Node.js powered Strapi Headless CMS server this blog is running on can handle conservatively 800 req/s before starting to autoscale and max out CPU on a shared Standard-1x Dyno, better performance on dedicated Performance Dynos. All languages/frameworks can benefit from CPU autoscaling however as a way of autoscaling via compute power independently from response times which are dependent on many different factors.

Below is a Apache Benchmark test of Strapi at ~290 req/second on a single Standard-1x Dyno at 1000 concurrency along with the analytics dashboard of 123 Dyno below.

At first glance it might seem like response times increased drastically but actually that spike is due to file uploads using Strapi normally. Notice the `/upload` path in **Max Response Times**. In stock autoscaling this can trigger upscaling when its not necessary and throw off your daily M95 sample value, increasing your bill with enough common file uploads. 

***More important to note here is that reponse time is not related to used compute power.*** The lower left box is CPU utillization which is close to zero most of the time, peaks in autoscaling are from Apache Benchmark tests only.

Autoscaling via response time here would not only increase your bill but it would not make full use of the dyno compute power paid for. You can save *incredible* amounts of money by autoscaling by CPU instead of response times for a large number of compute scalable applications. 

If you have a non-compute bound application and want to autoscale by response times, [123 Dyno has an improved response time autoscaling solution too](/blog/drastically-improved-response-time-autoscaling).


123 Dyno provides the tools to take Heroku further, cut your dyno bill and autoscale efficiently, keeping you on Heroku because of the ease of use, expansive ecosystem and complete infrastructure management. Use the savings for features like Heroku's managed private spaces or other add-ons that can save you crucial development time on the platform. 


*Bonus: The Strapi instance used in this example can be installed via this ***[button click](https://github.com/123-Dyno/publishing-strapiv4-heroku)***.*


**Read More About 123 Dyno**

[The 123 Dyno 12x speed increase to Heroku.](/blog/123-dyno-autoscaling-speed-benchmarks)

[Drastically Improved Response Time Autoscaling.](/blog/drastically-improved-response-time-autoscaling)


large_barez-omer-YrKsjHRglWw-unsplash.webp

medium_barez-omer-YrKsjHRglWw-unsplash.webp

small_barez-omer-YrKsjHRglWw-unsplash.webp

thumbnail_barez-omer-YrKsjHRglWw-unsplash.webp

barez-omer-YrKsjHRglWw-unsplash.webp

123 Dyno extends Heroku with CPU autoscaling so you can get the most value out of every dyno. View a use case of autoscaling by Strapi CPU performance on Heroku.

CPU Autoscaling on Heroku with 123 Dyno

123 Dyno provides an instant 12x autoscaling speed upgrade by providing autoscaling intervals of 5 seconds compared to stock Heroku's 60 seconds. This is the fastest autoscaling response time available in the Heroku ecosystem. This is important because it means apps running 123 Dyno will autoscale faster and more responsively under load than autoscaling with anything else.


A 60 second interval means an apps dyno formation can wait up to 60 seconds before autoscaling takes place to accomodate load. This means that if a traffic burst hits your servers it can critically slow down services and page load time, dyno formation waiting 60 seconds before autoscaling to become response to users when they are most interested. 

Compared to waiting for a box to start up to 3 minutes on other cloud providers this is still fast, but 123 Dyno takes it to the ultimate and makes sure your app stays snappy no matter what is thrown at it. The add-on solves this problem by providing configurable autoscaling intervals down to 5 seconds, allowing your application to respond to changing load faster. Lets look at an example of this by comparing Heroku autoscaling to 123 Dyno autoscaling with the repo https://github.com/123-Dyno/howdy-express-benchmark that simulates response time load.

We can use 123 Dyno's Transparent Logs features to see autoscaling events along with response times. The blue box is the initial autoscaling event and the red box are response times that should trigger autoscaling.

The important thing to note here is that [Heroku M95 autoscaling](https://devcenter.heroku.com/articles/scaling#autoscaling) autoscales when the target is initially surpassed but does not autoscale further given increasing response times simulating increasing load, more on this in ***[123 Dyno's Consistent Response Time Autoscaling Solution](/blog/drastically-improved-response-time-autoscaling)***. When it does autoscales it does so at intervals at least 60 seconds apart, a long time to wait for a new server.


### How 123 Dyno Autoscaling Compares


----------


123 Dyno decreases autoscaling intervals to 5 seconds and uses the concept of strictly definable Goldilocks ranges to autoscale. Simply upscaling when response times rise above maximum defined range and downscaling when response times drop below minimum defined range.


See the same test for 123 Dyno below, gold indicating the application is in Goldilocks, blue indicating the intial autoscaling event, red incoming load and purple graceful downscaling as specified.


The initial speed difference between 123 Dyno's initial autoscaling event and Heroku is negligible, usually less than 10ms. More importantly, **123 Dyno autoscales with increasing load consistently fast every 5 seconds, while Heroku takes 60 seconds or longer.**

This is crucial for autoscaling your application under increasing load, if the server is getting hammered, 60 seconds can mean a lost customer. 123 Dyno superchargers your stack like no other solution to make this a non-issue.

### What else can you do with 123 Dyno?


----------

If response time autoscaling isn't a good fit for you application, 123 Dyno extends Heroku with CPU based autoscaling, memory based autoscaling and queue time autoscaling for both web and worker dynos with the same speed upgrade. 123 Dyno was designed to provide an autoscaling extension that provides everything you need to intelligently autoscale an application for any occasion, more use cases below.

[**Improved response time autoscaling that works consistently**](/blog/drastically-improved-response-time-autoscaling)

[**Optimize IO event loop services with CPU autoscaling**](/blog/cpu-autoscaling-on-heroku-with-123-dyno)

scaling_5af8aa556a_194450958a.svg

Comparing 123 Dyno speed to stock Heroku autoscaling speed. 123 Dyno autoscales 12 times as fast, with 5 second scaling intervals compared to other add-ons 15 second intervals of Heroku's stock 60.

123 Dyno and Heroku Autoscaling Speed Benchmarks

There is value in being able to define a desired response time range and to autoscale to keep an application in that metric range.

Heroku add-on 123 Dyno offers an alternative, definable, and consistent option to autoscale applications by their response times for maximum control with critical options like Path Blocklisting long running urls, choice of average, maximum, or percentile metric values, and choice of sample window. 

[Heroku stock autoscaling](https://blog.heroku.com/heroku-autoscaling) does its job but can be inconsistent, uncontrollable, and strangely defined due to the fact that it is depdendent on the past days M95 response time value. It does however exceed in simplicity, able to be setup in a few button clicks. Lets take a look at the below example for a Performance dynos set to autoscale with a 500ms target with a max number of dynos at 3, a mostly blank instance with no M95 history.


The standard autoscaling jumps to 2 when M95 threshold is exceeded but does not autoscale further given higher response time values simulating increased load. If response time autoscaling is returning up to 10x what the target is, the application should autoscale. This can be inconsistent due to the nature of M95 being calculated over the past day's values along with other factors. Downscaling also can take minutes even when no requests are being sent to a server to act as guard rails for potential future load. 

123 Dyno addresses this with the concept of strictly defined & configurable response time Goldilocks ranges, upscaling when response times exceed defined range and downscaling when response times fall below defined range with a 12x speed boost to accomodate burst traffic. The add-on allows you to define exactly what you need and have it happen. 123 Dyno understands that applications are nuanced and gives you the full toolkit to make autoscaling work for you.

See the configuration below for [5 second upscaling intervals](/blog/123-dyno-autoscaling-speed-benchmarks) on M95 between 1 and 3 dynos autoscale exactly to specification.

As you can see in the autoscaling logs and response times 123 Dyno autoscales exactly to spec when it leaves it's Goldilocks range every 5 seconds under increasing simulated response time load of 800ms. Afterwards it autoscales down every minute as specified given M95 of the past 2 minute window.

### Accurate Response Time Values: Pruning Paths From Autoscaling Calculation


----------


Accurate response time data is important for accurate, efficient autoscaling. Long running values triggering autoscaling when they shouldn't and altering response time calculation of M95 over the past sample window. 123 Dyno gives you the ability to prune URLs from autoscaling calculation along with configurable sample window.

See below the Strapi Headless CMS instance this blog is running on the, `/upload` path takes up to 32000ms to complete for large files. A day of uploading large files or other long running paths can either throw off M95 calculation or trigger autoscaling increasing your bill. 123 Dyno offers the analytics you see above to notice this and a URL Blocklist to prune out long running paths in your application, saving you money from inaccurate autoscale events.

### Response Time Autoscaling That Works

----------

Altoghether these options allow for accurate response time autoscaling that works consistently and is not offered anywhere else except with 123 Dyno on Heroku. Have your service well defined in a response time range and autoscale to remain in range.


### What else can I do with 123 Dyno?

----------

If response time autoscaling isn't a good fit for you application, 123 Dyno extends Heroku with all the options you need to handle load. Autoscale via CPU, memory and queue times for both web and worker dynos up to 12x faster. 123 Dyno was designed to provide an autoscaling extension that provides everything you need to intelligently autoscale an application for any occasion, more use cases below.


[Are your response times to variable, look into autoscaling by CPU.](/blog/cpu-autoscaling-on-heroku-with-123-dyno)

[Need a speed increase? Adjustable autoscaling intervals on 123 Dyno can provide an up to 12x speed boost.](/blog/123-dyno-autoscaling-speed-benchmarks)

large_Screenshot from 2023-01-07 18-34-40.png

medium_Screenshot from 2023-01-07 18-34-40.png

small_Screenshot from 2023-01-07 18-34-40.png

thumbnail_Screenshot from 2023-01-07 18-34-40.png

Network Timeline

A controllable improved response time autoscaling method you will not find anywhere else but on 123 Dyno and Heroku.

Drastically Improved Response Time Autoscaling

Rafael Ancheta

Server scaling is the process of increasing compute power to accomodate increasing traffic and load, autoscaling dynamically adjusts compute power so you only pay for it during times it's needed. This is an in-depth guide to autoscaling Heroku to maximum effciency with add-on 123 Dyno whether your new or old the topic. Together these provide more autoscaling options and faster autoscaling than another other cloud provider, achieving rates like 800+ req/second per single Standard Dyno.

Autoscaling is a nuanced task and requires awareness of multiple layers of your stack to handle load. Dynos can be scaled both vertically and horizontally to handle incoming load past what one dyno can serve speedily. Vertically a dyno can expand both in computer power and memory to run an application, with low cost Eco dynos for prototyping and higher end Performance dynos with dedicated CPU and memory.

Aside from vertical scaling "making a server bigger", Heroku also comes with the now more standard method of load balancing requests across multiple computers "lets spin up a bunch of servers and distribute load across all of them". This is accomplished with autoscaling via Performance dynos with Heroku's 95th percentile (M95) method.

It autoscales up when server response times become too high and autoscales back down when load returns to normal around every 60 seconds. All that's needed is a few button clicks, a response time target and Heroku will then autoscale to keep the application at that response time target conservatively.

#### Extending and Optimizing Heroku Autoscaling with Add-On 123 Dyno 

While Heroku's M95 method is easy to setup and works for the general web application, it leaves much to be desired for those who want more options in varying situations or optimized efficient autoscaling that fits the application.

123 Dyno extends Heroku with scaling via standard dynos, CPU, memory, queue times, and response times along with options for a 12x boost to autoscaling speed and pruned URL paths. In contrast to a single target value, 123 Dyno uses the concept of a Golidlocks range, autoscaling up when metrics exceed upper threshold and downscaling when metrics drop below lower threshold.

#### 1. Choosing an Autoscaling Metric
Applications respond to load differently, some may be CPU bound while others are bound to memory. Choosing the correct autoscaling metric is crucial to ensuring that your stack is autoscaling efficiently to handle load. Autoscaling via a less optimal method can lead to slow server formation or piling resource costs on your bill. 123 Dyno provides a wide array of metrics to fit any web or worker dyno.

**Response Times - 95th Percentile, Average, and Maximum**

123 Dyno comes default with M95 autoscaling found in Heroku for the general application before any fine tuning. This method autoscales when M95 exits threshold values, this cuts out most statistically outlying response times and autoscales by the speed at which an application is responding to incoming load.

Autoscaling via average response time is available for applications that either have high varience in response time ranges (eg. file manipulation of different sized files), as well as maximum response time for applications that criticially need to autoscale to keep response times low (eg. live editing applications).


**CPU Load - Average and Maximum**

Response time autoscaling is dependent on a variety of different factors from overloaded downstream resources to cleared caches. For those seeking cost performance or maximum compute power for each dyno, CPU load/utilization can be used to autoscale. This method autoscales via the amount of CPU utilized instead of response time range.

A typically use case is for non-blocking IO applications like Node.js and Tornado that mostly wait on other services, the event loop architecture in these can achieve higher throughput at relatively low CPU cost. Bottlenecks typically exist in downstream resources if response times start to rise and throughput is low. Autoscaling by CPU ensures that the Dyno application layer of a stack scales both indepedently from other parts of the stack and at maximum efficiency.

Heroku measures CPU usage via a metric called load which closely corresponds to available CPU cores on underlying machines, Standard Dynos max out at about 4 load with 4 CPU cores, and Performance Dynos similarly at 8 load. 123 Dyno is able to be configured to autoscale to achieve maximum CPU usage unlike any other Heroku add-on.

**Queue Times - Average and Maximum**

123 Dyno queue times are representative of the Heroku `connect` time, the amount of time a request remains waiting before being processed by the dyno server. Higher queue times indicate that the downstream server, dyno handling the requests may be overloaded and cannot yet accomodate the incoming request. This style of autoscaling works well for frameworks and services that operate under the paradigm of concurrent connections and threaded processes. This includes frameworks like Flask behind Gunicorn and Ruby on Rails with easy to read syncronous code.

**Memory Utilization**

Memory utilization is the percentage amount of used memory divided by the total memory allowed for your Dyno plan. Dynos begin at 512MB of memory and increase to 14GB at the highest range.  For applications and workers that are memory dependent this can be an excellent way of autoscaling an application. In particular its possible to autoscale a worker or web Dyno handling large files or otherwise and avoid out of memory (OOM) errors occuring to overages.

OOM errors can be avoided using memory based autoscaling in conjuction with other configurations like settings a servers max accepted file size, compressing images or files, and passing images into queues to be processed by memory autoscalable workers.

#### 2. Pruning URL Paths
Response time base autoscaling is highly dependent on good response time data to know when to autoscale. 123 Dyno provides the option to prune out long running paths from autoscaling calculation to ensure that your application only autoscales when under sufficient load. For example, a server's file `/upload` path can throw off metrics as it is typically longer even though the server isn't under load, usually just waiting on a downstream file service. This can cause autoscaling to scale when an application shouldn't increasing your bill. URL blocklisting gives you the ability to cut this out from metric calculation, a feature that can save you tons.

#### 3. Adjusting Autoscaling Speed
123 Dyno provides the option of adjustable autoscaling speed up to 12x faster with intervals at 5, 10, 15, 30 & 60 seconds for both upscaling and downscaling. This is important for applications with burst traffic as the default Heroku 60s autoscaling interval can leave your application overloaded and slowed down until the next interval. This allows for complete speed customization as well as options to balance robustness versus efficiency.

By default 123 Dyno comes standard with 60s intervals between autoscaling events for both upscaling and downscaling, these can be adjusted independenlty to fit your application. Upscaling speed can be adjusted to down to 5 seconds for a 12x speed boost ensuring server's can adapt to changing and increasing load faster, keeping your application snappy at all times. Downscaling in tandem can be slowed down to ensure that server resources stay online during periods of high traffic.

#### 3. Monitoring Response to Load and Adjusting

The above ways to choose an autoscaling method are good guidelines however monitoring your application in practice is still necessary to ensure that resources are being used to the maximum extent possible, and reconfiguring if they are not.

You may find during load testing or production that an application may fit one method over the other especially as it changes to meet future demands. Your application may be CPU bound but later evolve into a memory dependent service for instance.

123 Dyno provides monitoring tools for all Dyno metrics along with path specific response times to quickly and easily find the metric that most directly correlates to required resource usage. Viewing this will allow you to both find what metric to autoscale by as well as a general idea of what normal application boundaries that metric is remaining in. You may find, for example, that an application's response times do not correlate to the CPU usage and decide to autoscale by CPU.

Further, you may find that your application layer of Dynos isn't being stressed at all via CPU or memory yet response times are still highly variable. This can be a sign of downstream resources that must be scaled up like databases, caches, or api services. In cases like these take a glance at other resources usage and scale accordingly.

#### 4. Alerting and Production Monitoring

After initial setup you may want to monitor your dynos to ensure that they are provisioned to meet all incoming load. 123 Dyno comes default with load alerting for web dynos that exceed 4000ms M95 response times as well as CPU at 3.5 load or 87.5% utilization for Dynos up to Standard 2x. These alerts will provide you a sense of when resources are being stressed to review analytics and adjust dyno formations as necessary. 

#### In Conclusion
Autoscaling an application is a nuanced task that requires a holistic view of your system, experience and continuous monitoring to ensure that your application is fit to the task. Heroku and 123 Dyno make this easy.

Often times it may appear that Heroku is costly but many services like Strapi for instance can achieve conservatively ~800 req/s on single Standard Dynos and likely much more for varying use cases. 123 Dyno provides the toolset, analytics, monitoring and alerting to be able to do this easily in PaaS fashion. It installs in a button click with no dependencies and allows intelligent handling of any load thrown at your application.

More Articles:

[Read more about autoscaling by CPU.](/blog/cpu-autoscaling-on-heroku-with-123-dyno)

[Read more about 123 Dyno's 12x speed boost.](/blog/123-dyno-autoscaling-speed-benchmarks)

large_ian-taylor-h7b1SUEMPIs-unsplash.jpg

medium_ian-taylor-h7b1SUEMPIs-unsplash.jpg

small_ian-taylor-h7b1SUEMPIs-unsplash.jpg

thumbnail_ian-taylor-h7b1SUEMPIs-unsplash.jpg

ian-taylor-h7b1SUEMPIs-unsplash.jpg

Learn how to analyze server load to configure Heroku with add-on 123 Dyno. Autoscale via CPU, memory, queue, and response times intelligently.

Blog

CPU Autoscaling on Heroku with 123 Dyno

123 Dyno and Heroku Autoscaling Speed Benchmarks

Drastically Improved Response Time Autoscaling

An In-Depth Guide to Autoscaling on Heroku with 123 Dyno

Loading...