No Hugging, No LearningOlaf Alders

Toronto Perl Mongers

Aug 25, 2016

@olafalders (Twitter) oalders (GitHub) OALDERS (PAUSE)

https://github.com/oalders/no-hugging-no-learning

The ProblemBuilding an app that can track and chart (almost)anything that is available via 3rd party APIs

No screen scraping

No ToS violations

Doing this in my limited spare time

The SolutionBuild an app based solely on what I already know:"No hugging, no learning"

Of the three programmer virtues (laziness,impatience and hubris), I would let laziness be myguiding light

I tried to self-impose a real lack of intellectualcuriosity

The Evolution

The App

Before

Some command line scripts as a proof of concept

This became a Dancer (1) app run via Plack

Pros

Dancer apps are easy to get up and running

Cons

I didn't really love the Dancer (1) syntax

Dancer2 was still quite new and the plugin supportwasn't there yet

After

I moved to a Mojolicous Lite app run via Plack

I then transitioned to a full Mojo app

Run via morbo in development

Run via hypnotoad in production

morbo

By default watches your �lesystem for changesand then reloads your app

hypnotoad

Zero downtime restarts

You don't need to provide your own init script

If you don't care about zero downtime or just wantthe prefork server, you have that option

perl script/myapp prefork

same con�g args as morbo/daemon

Did I learn?

I partly became familiar with Mojo via my day job,but I did have to do a fair bit of learning on myown.

I was not familiar with Hypnotoad , but luckily thatwas a pretty easy transition. The docs are reallygood.

Authentication

Before

Mozilla's Persona

Pros

Pretty easy to con�gure

All anyone really needs is an email address

Cons

Issues getting it to work of�ine

No widespread adoption

Felt like the �rst 80% was easy but the last 20%was eating into my time

After

OAuth via any of the available integrations

I needed this anyway in order to get user data, soit keeps things simpler to have Persona out of themix

With Mozilla no longer 100% behind Persona, itbecomes a less attractive solution

most non-technical users won't be familiar withit at this point anyway

SSL Everywhere

Looked brie�y into Let's Encrypt via Ansible

Ended up spending USD 10 for a RapidSSL cert

Relational Database

Before

Application data stored in MySQL database

Pros

I'm very familiar with MySQL

I prefer many MySQL tools, like the mysql commandline interface and phpMyAdmin

MySQL replication is dead easy to con�gure

Cons

"Foreign keys that point to the renamed table arenot automatically updated. In such cases, you mustdrop and re-create the foreign keys in order forthem to function properly."

After

3 Postgres databases

Minion

Application data

Test suite �xtures

Did I learn?

Since we use Postgres at $work , I didn't have tolearn a whole lot to make the switch.

The database does a litle bit more than the bareminimum -- I'm not taking advantage of all thatPostgres has to offer

No jsonb columns just yet

Job Management

Before

cron scripts

After

Minion

Did I learn?

Had basically zero knowledge of Minionimplementation

But, there's not much you need to learn in order toget up and running

Minimized a lot of really convoluted cron job logic

This probably saved me time in the long run

Minion How To

You don't need to have a full-blown Mojo app touse Minion

Just create a bare bones app to get started

Use this with your Catalyst, Dancer, [insertframework here] application

Let's look at how MetaCPAN uses Minion

Create an App

https://github.com/metacpan/metacpan-api/blob/master/lib/MetaCPAN/Queue.pm

Sets up a tiny Mojo app

Adds Minion tasks in the startup() method

You can see how we trap warnings and log failedjobs in the _gen_index_task_sub() method

Optionally Create Your MinionBackend Elsewhere

We've abstracted it out tohttps://github.com/metacpan/metacpan-api/blob/master/lib/MetaCPAN/Queue/Helper.pm

We do this so that we can use an SQLite databasewhen running under the test harness and a Postgres database in all other cases

Using SQLite for testing makes our Traviscon�guration much easier

This will also allow us more easily to run tests inparallel

Start Up Your Queue

In development, you can create a 4-5 line script tostart up your queue via morbo

https://github.com/metacpan/metacpan-api/blob/master/bin/queue.pl

In production we use Daemon::Control to managestarting and stopping a daemon

https://github.com/metacpan/metacpan-puppet/blob/master/modules/minion_queue/templates/init.pl.erb

Add Tasks

A task is essentially an anonymous sub which youcan later refer to by name

A job is an instance of a task

$minion->add_task( add_things => sub { # $job is a Minion::Job object my ($job, $first, $second) = @_; $job->finish($first + $second); });

More Complete Example

$minion->add_task( add_things => sub { my ($job, $first, $second) = @_; $job->finish({ message => 'Great!', total => $first + $second, }); }); my $id = $minion->enqueue(add_things => [1, 1]); my $result = $minion->job($id)->info->{result};

$result is now

{ message => 'Great!', total => 2 }

Storing Your Job Result

$job->finish;

$job->finish('Fantastic!');

$job->finish({ foo => 'bar' });

Later, you can �nd this arbitrary data in $job->result

Populate Your Queue

enqueue() means "add job to queue"

$app->minion->enqueue( my_task_name => ['foo', 'bar'] );

$self->minion->enqueue( track_service_resource => [{ service_resource_id => $service_resource->id }] => { priority => 1 }, );

priority ranges from 0-X, where X is a positiveinteger

jobs with priority 10 will run before jobs withpriority 9 etc

enqueue() options

attempts => 10 (# of times to attempt job)

delay => 3600 (delay this job for 3600 seconds)

parents => [$id1, $id2] (One or more existing jobsthis job depends on, and that need to havetransitioned to the state finished before it can beprocessed)

priority => 3 (see previous slide)

queue => 'dexter' (some arbitrary name, defaultsto default )

Minion::Job

https://metacpan.org/pod/Minion::Job

You can set up events to be �red on failed and finished states

Get information via $job->info

$job->is_finished

$job->remove

Much, much more

Inspect Your Queue

On the MetaCPAN Vagrant box:

$ vagrant ssh $ cd /home/vagrant/metacpan-api/ $ ./bin/run bin/queue.pl minion job -s { "active_jobs" => 0, "active_workers" => 0, "delayed_jobs" => 0, "failed_jobs" => 0, "finished_jobs" => 0, "inactive_jobs" => 0, "inactive_workers" => 1 }

Test Your Queue

https://github.com/metacpan/metacpan-api/blob/master/t/queue.t

use MetaCPAN::Queue; use Test::More; use Test::RequiresInternet ( 'cpan.metacpan.org' => 443 ); my $app = MetaCPAN::Queue->new; my $release = 'https://cpan.metacpan.org/authors/id/O/OA/OALDERS/HTML-Restrict-2.2.2.tar.gz' $app->minion->enqueue( index_release => [ '--latest', $release ] ); $app->minion->perform_jobs; done_testing();

Advanced: Behaviour on Events

https://github.com/jberger/Minion-Noti�er/blob/master/lib/Minion/Noti�er.pm#L39-L56

sub setup_worker { my $self = shift; my $dequeue = sub { my ($worker, $job) = @_; my $id = $job->id; $self->transport->send($id, 'dequeue'); $job->on(finished => sub { $self->transport->send($id, $job->on(failed => sub { $self->transport->send($id, }; $self->minion->on(worker => sub { my ($minion, $worker) = @_; $worker->on(dequeue => $dequeue); }); return $self }

Code Context

The preceding code is called like this:

https://github.com/jberger/Minion-Noti�er/blob/master/lib/Mojolicious/Plugin/Minion/Noti�er.pm#L37

Pros

We can replace a failure-prone, forking, longrunning process with a series of jobs

If a job fails in an unexpected way, it doesn'tprevent all the other jobs from running

We can check the status of all jobs at any point togauge how things are progressing

Jobs which fail can be retried at intervals

We can start an arbitrary number of worker apps

Using Postgres replication MetaCPAN can nowstart X workers on 3 different boxes, which givesus greater scaling when needed

Cons

Minion doesn't come with a handy daemonhandler like hypnotoad , but you can set this stuff allup yourself pretty easily

There aren't yet a lot of tools to visualize what'sgoing on with the queue, but again this can bedone pretty easily if you need it.

Database Migrations

Before

Database schema managed by Database::Migrator

Pros

Database::Migrator is what we use at $work

It's pretty simple

In addition to SQL, you can also run Perl scripts aspart of a migration

Cons

Database::Migrator needs to be subclassed, whichmeans you have to provide a fair bit offunctionality just to get up and running

After

Application and �xture databases managed via App::Sqitch

Minion has its own schema deployment logic

Pros

sqitch handles rollbacks as well as deployment

Optionally can use custom assertions to ensurethat deployments were successful

Cons

There's a lot more going on here, so you probablyhave to do a bit of reading before just jumping in

I had to use the CLI rather than the modulesbecause of some weirdness between Moo and Moose when I tried to use the modules directly

Did I learn?

Yes, I did have to learn sqitch from scratch

The author (David Wheeler) was extremely helpful

I was able to get it working with my �xtures

See Also

Mojo::Pg::Migrations

Mojo::mysql::Migrations

Mojo::SQLite::Migrations

Automation of Deployment

Before

The plan was to use Puppet

Pros

I was already familiar with Puppet

Cons

I was already familiar with Puppet

Even simple things seemed hard

Tired of dealing with hard to debug certi�cateissues

There may have been obvious and easy �xes tomy problems, but I couldn't easily �nd them

I'm not saying that Puppet isn't a wonderful tool,but it felt like a bigger hammer than I needed

After

Ansible

Pros

Excellent integration with Vagrant

No need for bootstrapping on the targetdeployment machines

It's easily installed via Homebrew on OS X

Has good handling of git submodule

User contributed plugins are trivial to install

Cons

I actually don't have a lot of complaints

Moving from v1 to v2 addressed the few issuesthat I ran into

Ansible in Vagrant�le

config.vm.provision "ansible" do |ansible| ansible.verbose = "v" ansible.playbook = "ansible/site.yml" ansible.sudo = true ansible.groups = { "development" => ["default"], "webservers" => ["default"], } end

Installing Packages

--- - gather_facts: no hosts: webservers tasks: - apt: 'pkg={{item}} state=installed' name: Install misc apt packages with_items: - curl - ntp - tree - unzip - apt: update_cache=yes cache_valid_time=3600 name: Run the equivalent of "apt-get update" - apt: upgrade=dist autoremove=true name: Update all packages to the latest version

Did I learn?

Yes, I had to learn all about Ansible , since I had neverused it before. In the meantime $work has alsoswitched from puppet to ansible , so it turned out tobe knowledge that I can apply there as well.

Keeping in mind how many hours I've spent battling Puppet on past projects, I think learning about Ansible was actually the better, faster choice.

Predictions Which Did Not Change

JS-Driven Charts

Highcharts turned out to be an excellent choice

Excellent docs

Responsive support

I'm also quite happy with the product itself

Twitter Bootstrap

Still a solid choice for a CSS framework

Travis CI

Private repository support is not within my budget(starts at USD 129/month)

Open Source option works excellently forhttps://github.com/oalders/wundercharts-plugin

Carton for Dependency Pinning

This suits my needs �ne at this point

https://metacpan.org/pod/Carmel is still listed asexperimental

Vagrant

Initially I just ran the app inside OS X directly

I realized pretty quickly that I wanted to have asandbox

I've even got a Vagrantfile for the plugin systemhttps://github.com/oalders/wundercharts-plugin/blob/master/Vagrant�le

So, anyone who wants to contribute can have afully operational, sandboxed system with all ofthe modules installed within a matter ofminutes

Code::TidyAll

I use this to tidy as much code as possible

It's enforced via a Git pre-commit hook so that Idon't have to think about it

LWP::ConsoleLogger

I use this a lot when debugging interactions with3rd party APIs

This only really works when you can get at the UAwhich a module is using

I wish more module authors would expose theiruser agents

The Mojo equivalent is setting MOJO_USERAGENT_DEBUG=1

DBIx::Class

Use foreign keys and the schema dumperautomatically sets up relationships for you

Use this trick to enable perltidy to tidy yourschema �les:https://metacpan.org/pod/release/ILMARI/DBIx-Class-Schema-Loader-0.07045/lib/DBIx/Class/Schema/Loader/Base.pm#�lter_generated_code

Mojolicious::Plugin::AssetPack

Compress and convert css, less, sass, javascriptand coffeescript �les

In development mode you get the individual �lesin your template (for easier debugging)

Strict Constructors

MooseX::StrictConstructor

MooX::StrictConstructor

MetaCPAN::Moose

etc

nginx

Ubuntu 14.04.5 LTS

Perl v5.18.2

Things I Hadn't Planned on Using

Wercker

I didn't know this existed

I'm not good about running my test suite afterevery change, so I realized I had to automate this

I didn't want to run my own Jenkins/TeamCity/etcservice

I've seen how much con�guration can go intothese things and I didn't have the time to messwith it

Free, concurrent builds for private repositories

Wercker (cont'd)

I didn't want to give it access to all of my privaterepositories

I created a BitBucket account (free privaterepositories)

Added a new remote to my repository"bitbucket"

Every time I push to the "bitbucket" remote thetests are run and I get an email about success orfailure

Protip: if weird errors happen, sometimes youhave to clear your Wercker cache

wrapbootstrap.com

I hadn't even thought about the UI when I startedthis project

I bought the "Inspina" theme for USD 18

This gave me solid options for both logged inand logged out UI

Was trivial to implement and immediately mademy terrible design much better

Since I was already using Bootstrap thereweren't a lot of changes to be made

Date Range Picker

http://www.daterangepicker.com

Easy to use and makes it easy to use sensibledefaults when selecting dates or ranges of dates

Open Source That Came Out ofThis

Mojolicious::Plugin::Web::Auth::Site::Spotify

Mojolicious::Plugin::Web::Auth::Site::Runkeeper

WebService::HealthGraph

Code::TidyAll::Plugin::YAML

WWW::Spotify (co-maint)

Patch to WebService::Instagram

I open sourced all of my plugins

WunderCharts Plugins

https://github.com/oalders/wundercharts-plugin

100% open source

I'm in a position to integrate user contributedpatches without open sourcing the entire codebase

For example, if we want to integrate FitBit

Need to createMojolicious::Plugin::Web::Auth::Site::FitBit

Need to send a pull request forWunderCharts::Plugin::FitBit

Third Party Integrations

I didn't have a �rm idea of how many I wanted

Got a mixture of OAuth, OAuth2 and no auth

OAuth

Twitter

OAuth 2

Facebook

Instagram

Spotify

GitHub

No authentication required

Hacker News

Don't Count Your Chickens

Instagram apps (unlike the other sites) have aformal review process

I didn't realize until after my integration was�nished that my use case may not qualify

I did get accepted at the end of the day, but it's abit of a coin toss

You need to show them the integration via ascreen cast before you get approval, so it'spossible to do the heavy lifting and still be denied

Success?

Among other things, I ended up having to learn (orlearn more about) the following technologies:

Ansible

BitBucket

Date Range Picker

hypnotoad

Minion

Mojo

Mojolicious::Plugin::AssetPack

Persona

sqitch

Wercker

[also various 3rd party APIs for web services]

Thanks!

To Joel Berger for proofreading the Mojo bits ofan earlier version of this talk

All remaining (or since introduced) errors are myown

The "Finished" Product

https://wundercharts.com