Wringing Performance out of Perl

Grant Street Group

• Began as a financial advisor group

Grant Street Group

• Discovered the Internet in 1997

Grant Street Group

• Online Auctions of Property Tax Liens• Web-Based billing system for tax collectors• Conversion of legacy tax-collector databases• Online license / vehicle tag renewals• Online payment processing• Auctions of all types of bonds• And lots, lots more!

Tax Lien Auctions

• Absolute feeding frenzy– Our bidders threatened to exhaust TIN numbers– 20 million bidders in 2011– More than 30 billion bids altogether– Average was a 500,000-way tie– About 2,000 auctions closing simultaneously

Tax Lien Auctions

• How do we award auctions performantly?– Random tie-breaking with Crypt::Random– Random row-ID plus MySQL = S L O W– Turns out we can do it much faster in Perl

Tax Lien Auctions

• Net result: auction closing takes 20 seconds– Breaking 2,000 ties, each 500,000-way– Stress-testing indicates can scale by 4x– The IRS definitely cannot scale by 4x

Property Tax Online Payments

• Florida residents can pay the property tax• Hosted, customized sites per county• Largest counties have ~1,000,000 parcels• Users are typical Florida residents

Property Tax Online Payments• Backend is MySQL and Sphinx• Lightning-fast searches with Perl– Mapping IDs to table, column, PK– Parsing SHOW STATUS LIKE ‘sphinx%’• Lots of useful metadata!

• Net results:– Sub-second turnaround times– 9 minute average time on site by payers– 4 minute average time on site overall

Customer Data Conversion

• Largest county in FL is a customer– Population ~2.4M people– Tax roll of ~900K parcels– History of ~5.6M bills across 6 years

• Full database is large (by our standards)– Data files are ~30-50GB– Full conversion is ~160 hours, using Perl– Might be ~8 hours using pure SQL

• Problem is we can’t use pure SQL– Ridiculous amounts of business logic– Utterly different data models

• We’re a Perl shop; Perl is our hammer

• Hugely parallel data conversion– Subdivide conversion into smaller steps– Build hash of dependencies between steps– Construct DAG of work units in MongoDB

• Distribute the actual work– Run lots of Perl worker processes– Workers grab ready work units– Perform the work unit sequentially

• The end result– Total conversion time ~3 hours with 80 workers– Nightly reloads now very practical– Able to resume incomplete loads

We’re Hiring Telecommuters

Wringing Performance out of Perl

Technology

Transcript of Wringing Performance out of Perl

Perl: A Short Introduction for Bioinformaticians · Perl: A Short Introduction for Bioinformaticians 2 What is Perl? Perl is an interpreted (scripting) language Perl is (almost) platform-independent

README - IBM · 2016. 7. 1. · openldap openssh openssh-clients ... pcre pdksh perl perl-DateManip perl-Filter perl-HTML-Parser perl-HTML-Tagset perl-lib perl-URI pinfo popt portmap

Perl family: 15 years of Perl 6 and Perl 5

Advanced Perl Parsing - Monash Universityusers.monash.edu/~damian/Perl/DPW/AdvancedPerlParsing.pdf · Advanced Perl Parsing Deutscher Perl-Workshop 3.0 Tutorial Damian Conway 28 February

Survey of Advanced Perl Topics Database access "OpSys"-like functions Signal Handling Inside-Out Objects.

How to Read a Financial Report: Wringing Vital Signs Out of … · 2015-09-22 · WRINGING VITAL SIGNS OUT OF Eighth Edition ... 3 Profi t Accounting 23 4 Profi t Isn’t Everything

Programming Perl - Perl Training · This is version 1.39 of Perl Training Australia’s "Programming Perl" training manual. Table of Contents 1. About Perl Training Australia ...

Dumping Perl 6 (French Perl Workshop)

Brno Perl Mongers 28.5.2015 - Perl family by mj41

Perl programming session 5 operators in perl

1 Introduction to Perl and Perl Syntax Learning Perl, Schwartz.

Perl. Perl notes 2 Perl Perl - Practical extraction report language –for text files –system management –combines C, SED, AWK, SH –interpreted –dynamic.

1 Perl Regular Expressions. Things Perl Can Do Easily with Regular Expression 2 Pattern matching Find out if a string contains some specific pattern.

Perl for System Administration - Perl Training Australia

Perl 4: Applied Perl - O’Reilly Mediaarchive.oreilly.com/oreillyschool/courses/Perl4/Perl 4 Applied Perl... · Perl 4: Applied Perl Lesson 1: Introduction: ... Lesson 5: Installing

Use Perl like Perl

Introduction to Perl - Nisanov · 2008-07-31 · Introduction to Perl Pinkhas Nisanov. Perl culture Perl - Practical Extraction and Report Language Perl 1.0 released December 18,

Modern Perl for Non-Perl Programmers

Фрагменты Perl 6 в Perl 5.10

CGI & Perl Programming Language What is CGI? Introduction of Perl Programming Language –Simple Peal program –Perl variables –Perl Operators –Flow Controls.