At Troy Web, we're big fans of the Laravel PHP Web Framework. Its vast feature set, extreme flexibility, and expressive object oriented design make development enjoyable and allows us to deliver high-quality products in a timely fashion. At the end of July the Troy Web Laravel team ventured to New York City for the framework's annual LaraconUS conference. The presenter lineup consisted of top notch developers and creators from the Laravel community and beyond.
Troy Web Team Laravel at Laracon 2017
The presentations on the first day the conference concluded with a talk from the framework's creator Taylor Otwell covering some of the features arriving with the upcoming release of Laravel 5.5 and to introduce Laravel Horizon, a new package for managing, monitoring, and configuring queues.
In case you're not familiar with Laravel's queues feature, it offers the ability to delegate CPU intensive and timely operations such as network call to external services, sending emails or PDF generation to a queueable class that can be dispatched to a queue worker for later processing. Queue workers are instances of the Laravel application that runs on a separate long running process with the sole responsibility of processing these tasks one at a time in the order they are received.
I find support for this simple but effective form of asynchronous programming to be one of Laravel's most powerful and unique features. At Troy Web, we leverage queues quite often to keep response times snappy and throttle server load. Admittedly though, when I was first introduced to queues I found them to be one of the more complex and intimidating aspects of what was, for the most part, a very beginner friendly framework.
Horizon aims to simplify the configuration and set up of queue workers and provides a graphical dashboard for managing and monitoring all things queue related. Best of all Taylor announced that Horizon will be available for free and will be open sourced on GitHub (What a generous guy!).
Today we'll be setting up a fresh Laravel demo project with the currently available Horizon beta build so we can try it out for ourselves. I'll be using the Laravel Homestead developer environment as it comes with all that is needed to get started. At the time of writing Laravel 5.5 is still unreleased but we'll use the latest developer build for our Horizon playground. We can create a fresh Laravel project that uses the latest 5.5 developer build using Composer. ``` vagrant@homestead:~/Code$ composer create-project laravel/laravel=5.5 horizon-playground dev-develop Installing laravel/laravel (dev-develop 7bcf7a5450000a13a9771991cbe7713ad7e03514) - Installing laravel/laravel (dev-develop develop): Cloning develop from cache Created project in horizon-playground > @php -r "file_exists('.env') || copy('.env.example', '.env');" Loading composer repositories with package information Updating dependencies (including require-dev) Package operations: 68 installs, 0 updates, 0 removals ... - Installing laravel/framework (dev-master a76b589): Loading from cache ... Writing lock file Generating autoload files > Illuminate\Foundation\ComposerScripts::postAutoloadDump > @php artisan package:discover Discovered Package: fideloper/proxy Discovered Package: laravel/tinker Package manifest generated successfully. Do you want to remove the existing VCS (.git, .svn..) history? [Y,n]? y > @php artisan key:generate Application key [base64:ZJm5O0ED+xg7D1LNZHQ49xjWr3yhLCOSHCdBM4C6J34=] set successfully. ``` Good to go! let's cd into the new project directory and confirm that our version is correct. ``` vagrant@homestead:~/Code$ cd horizon-playground/ && php artisan --version Laravel Framework 5.5-dev ``` Next, let's pull in the Horizon package using composer. ``` vagrant@homestead:~/Code/horizon-playground$ composer require laravel/horizon Using version ^0.1.0 for laravel/horizon ./composer.json has been updated Loading composer repositories with package information Updating dependencies (including require-dev) Package operations: 3 installs, 0 updates, 0 removals - Installing cakephp/chronos (1.1.2): Downloading (100%) - Installing predis/predis (v1.1.1): Downloading (100%) - Installing laravel/horizon (v0.1.0): Downloading (100%) predis/predis suggests installing ext-phpiredis (Allows faster serialization and deserialization of the Redis protocol) Writing lock file Generating optimized autoload files > Illuminate\Foundation\ComposerScripts::postAutoloadDump > @php artisan package:discover Discovered Package: fideloper/proxy Discovered Package: laravel/tinker Discovered Package: laravel/horizon Package manifest generated successfully. ``` Notice `Discovered Package: laravel/horizon`. If you worked with Laravel prior to version 5.5 you're probably accustomed to registering a Service Provider and maybe some Facades when you pull in a new package as part of the installation process. Laravel 5.5 ships with a new feature that allows package maintainers to specify what providers need to be registered within the package's composer.json file. Installation is then handled automatically by the framework when it is pulled into a project. Horizon is, of course, a package that supports this so the package installation is now complete, neat stuff!
We can publish Horizon's configuration file and front end assets to our project directory using the `horizon:assets` artisan command. ``` vagrant@homestead:~/Code/horizon-playground$ php artisan horizon:assets Copied Directory [/vendor/laravel/horizon/public/js] To [/public/vendor/horizon/js] Copied Directory [/vendor/laravel/horizon/public/css] To [/public/vendor/horizon/css] Copied Directory [/vendor/laravel/horizon/public/img] To [/public/vendor/horizon/img] Publishing complete. ``` Now if navigate to the /horizon route of our application in the browser we're presented with the flashy VueJS powered Horizon dashboard.
The status is inactive because we haven't yet started the Horizon process with artisan. Before we do that let's first jump into config/horizon.php and adjust some settings. We'll add a second queue to our local environments configuration, bump the processes count, and set our balance to the string 'auto' so we can try out Horizon's automatic queue load balancing features. ``` //.. 'environments' => [ 'local' => [ 'supervisor-1' => [ 'connection' => 'redis', 'queue' => ['default', 'a_second_queue'], 'balance' => 'auto', 'processes' => 10, 'tries' => 3, ], ], ], //.. ``` Note: At the time of writing Horizon only supports queues that utilize the Redis queue driver.
Prior to Horizon setting up and configuring Laravel's queue worker processes required a bit diving into the the realm of DevOps (though if you are lucky enough to be using Laravel Forge to manage your servers it greatly simplifies things). With Horizon we only need to worry about running a single artisan command and all the DevOps work will be handled for us based on the values in the horizon configuration file. Queue worker configurations will truly be a part of the project's codebase, brilliant! Let's start our queue workers using the `horizon` artisan command.
``` vagrant@homestead:~/Code/horizon-playground$ php artisan horizon Horizon started successfully. ```
Our dashboard is nearly instantly updated to reflect the new state of our queues. Now we need to make a job class for our queue to process. We can generate a new using artisan. ``` vagrant@homestead:~/Code/horizon-playground$ php artisan make:job LazyUserJob Job created successfully. ``` Note: Several of Laravel's other core components are queueable besides job classes, including notifications, mailables, and event listeners. When we jump into app/Jobs/LazyUserJob.php we see the generated class comes with a constructor and a handle method.
When we dispatch a job to a queue we create a new instance of the class passing any serializable data objects needed by the routine into the constructor allowing us to bind them as class properties. The job class is then serialized and saved at the end of the processing queue. When the job eventually reaches the front of the queue the job instance is recreated and the handle method will be called. The handle function serves as the entry point for the routines business logic.
Note: Any Eloquent models that are bound to the class will be re-fetched from the database when the job is unserialized for processing so you can always count on them having up to date data. ``` <?php namespace App\Jobs; use App\User; use Illuminate\Bus\Queueable; use Illuminate\Queue\SerializesModels; use Illuminate\Queue\InteractsWithQueue; use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Foundation\Bus\Dispatchable; class LazyUserJob implements ShouldQueue { use Dispatchable, InteractsWithQueue, Queueable, SerializesModels; protected $user; public function __construct(User $user) { // Job accepts a single User instance when it is dispatched. $this->user = $user; } public function handle() { // I should probably do something with the User // but I'd rather just sleep for a half second. usleep(500000); // And sometimes i'll just randomly fail. if(rand(1, 100) >= 50) throw new \Exception('Everything is horrible.'); } } ``` Jobs can be dispatched to a queue from pretty much anywhere within a Laravel application (Including from within another job that is running on a queue worker!). For this example, I'm going to drop a code snippet into the Laravel's Tinker REPL command line utility that is included will Laravel. The snippet will use Laravel's model factory feature to generate 500 dummy User models, save them to the database and then dispatch a LazyUserJob to the default queue for each user.
Be sure to first to run Laravel's default migrations to create the users table (`php artisan migrate`) and then set the `QUEUE_DRIVER` to redis within your projects .env file. ``` vagrant@homestead:~/Code/horizon-playground$ php artisan tinker Psy Shell v0.8.10 (PHP 7.1.7-1+ubuntu16.04.1+deb.sury.org+1 — cli) by Justin Hileman >>> factory(App\User::class, 500)->create()->each(function($user) { dispatch(new App\Jobs\LazyUserJob($user)); }) ```
Because we're using auto balancing option and our default queue has a back log of jobs to process spare workers from our second queue will be reallocated to help balance the load, awesome!
Failed Job Management
From the Failed page we can dive into the details and retry any job that has failed within the last week or so. Prior to Horizon this was typically handled via artisan commands and often required manually diving into the framework's failed_jobs table, definitely a big step up.
Monitoring Horizon supports the concept of "tagging" items that are placed onto the processing queue. A queued items tag(s) will be automatically assigned by Horizon using the Eloquent model(s) that the item has been passed or by reflecting on the item itself for example when working with queued notifications. Tags can also be assigned explicitly by adding a tags() method to the queueable class. The tags method should return an array of strings representing the desired explicit tag names. On the Monitoring page, we can watch for queued items that match specific tags in real time. Debugging queue related support issues from specific customers just got a whole lot easier!
Notifications
Within the Horizon configuration file, we can set a number of seconds for what should be considered a "long wait time" for each of our queues.
```
//.. app/config/horizon.php
'waits' => [
'redis:default' => 60,
],
//..
```
If Horizon notices a queue's estimated wait exceeds it's configured long wait time a notification can be dispatched by either SMS or Slack. A phone number and/or a Slack hook URL can be registered to receive these notifications via calls to `App\Horizon\Horizon::routeSmsNotificationsTo('your-phone-number-here');` and `App\Horizon\Horizon::routeSlackNotificationsTo('your-slack-webhook-url-here')` from within the boot() method of a Laravel Service Provider.
Note: Under the hood, these notifications are sent via the native notification component that ships with Laravel. Make sure you've set up all the necessary prerequisites for the Slack and Nexmo drivers before registering to receive notifications from that channel.
Metrics
As an added bonus Horizon includes a Metrics page where throughput and runtime statistics can be viewed for specific types of tasks or queue workers in nicely drawn charts.
These statistics are aggregated using the included `horizon:snapshot` command. You can make use of Laravel's task scheduling features to periodically run this command at whatever interval you like. app/Console/Kernel.php ``` //.. protected function schedule(Schedule $schedule) { $schedule->command('horizon:snapshot')->everyMinute(); } //.. ``` Deployment
If you want to use the Horizon dashboard outside of the local environment you'll need to first specify some sort of authentication strategy to protect the dashboard. This is done by registering a closure with the Horizon class (the boot method of a Laravel Service Provider would be a good place to do this). When the dashboard route is accessed this closure receives the incoming request instance and should return a boolean indicating whether access should be allowed or denied. Note: By default, it's not possible to leverage the built in Laravel user authentication system within the authentication closure because Horizon's routes do not sit behind the needed middleware. ``` //.. class AppServiceProvider extends ServiceProvider { /** * Bootstrap any application services. */ public function boot() { \Laravel\Horizon\Horizon::auth(function (Request $request) { // Whitelisted IP address return $request->ip() === '10.0.2.2'; }); } //.. ``` Next, a process monitor like supervisord will need to be configured to monitor and restart the `php artisan horizon` process in the event it is killed. Sample supervisor configuration - /etc/supervisor/conf.d/horizon.conf ``` [program:horizon-1] directory=/home/forge/example.com/ command=php artisan horizon autostart=true autorestart=true user=forge redirect_stderr=true stdout_logfile=/home/forge/.forge/horizon-1.txt ``` Finally, because each queue worker is a long running instance of the Laravel application the `php artisian horizon:terminate` command will need to be run as part of your deployment process. This command will gracefully kill all of the existing worker processes so that new ones can be created with the latest code changes. Note: You can use the included `horizon:pause` and `horizon:continue` artisan commands to pause/resume your queues while using a process monitor with Horizon.
Conclusion
It sounds like Laravel Horizon will exit beta alongside the release of Laravel 5.5 which is expected to be sometime towards the end of August. I'm sure Taylor and the community will make some adjustments and possibly add some new features in the time leading up to the official release but even as it stands right now, Laravel Horizon is pretty damn impressive. Team Laravel at Troy Web is definitely looking forward to putting it into production and we're excited for whatever future improvements are on the horizon (ayy!) for the framework.
Cheers, -Damian