Katello on TorqueBox
-
Upload
lzap -
Category
Technology
-
view
553 -
download
1
description
Transcript of Katello on TorqueBox
KATELLO ON TORQUEBOXJava Loves Ruby
ABOUT MELukáš Zapletal
ABOUT ME@lzap
ABOUT ME@lzap_CZ @lzap80
THEME SELECTIONBlah blah blah:
- - - - - Sky Beige Simple Serif Night Default
KATELLO
TORQUEBOX
RUBY
JRUBY
JAVA
JAVA
CATS-FREE TALK
WHAT IS KATELLO
KATELLO IS
A OPEN-SOURCE
CONTENT AND SYSTEM
MANAGEMENT STACK
FOR DATACENTERS
AND CLOUD
IF YOU TAKE ...
AND CLOUD
FOR DATACENTERS
AND CLOUD
FOR DATACENTERS
AND CLOUD
ENOUGH FUN!What the cloud is?
NISTNational Institute of Standards and Technology
NIST DEFINITIONthe NIST definition of cloud computing
NIST DEFINITION
NIST DEFINITION
NIST DEFINITION
NIST DEFINITION
NIST DEFINITION
NIST DEFINITION
NIST DEFINITION
CLOUD DEFINITIONCloud computing is a model for enabling ubiquitous,
convenient, on-demand network access to a shared pool ofconfigurable computing resources.
CLOUD DEFINITIONIt can be rapidly provisioned and released with minimal
management effort or service provider interaction.
ESSENTIAL CHARACTERISTICSon-demand service
ESSENTIAL CHARACTERISTICSbroad network access
ESSENTIAL CHARACTERISTICSresource pooling
ESSENTIAL CHARACTERISTICSrapid elasticity
ESSENTIAL CHARACTERISTICSmeasured service
SERVICE MODELSSaaSPaaSIaas
DEPLOYMENT MODELSprivate cloudcommunity cloudhybrid cloudpublic cloud
WHAT YOU CAN DO WITH KATELLO?
RED HAT SUBSCRIPTIONwhat is it and how it works
SUBSCRIPTION MANAGEMENTimport Red Hat subscriptions from Portalcreate your own products and subscriptionsregister machines and consume themsee some statistics and graphs
CONTENT MANAGEMENTsync RPM content from CDNsync RPM content from other repositoriessync puppet content from Puppet Forgeseparate content into environments and content viewspromote contentconsume content using yum or puppetremote install/upgrade content
THERE IS FOREMAN
PROVISIONINGregister installation treesprepare provisioning templatesprovision bare-metal/virtual systemsmaintain registry of all systems
CONFIGURATION MANAGEMENTimport Puppet classes into Foreman databaseassign classes to hosts (existing or provisioned)assign parameters to classescollect info from Facter and Puppetcreate statistics and graphs
KATELLO UI
KATELLO UI
KATELLO UI
RED HAT PRODUCTSSubscription Asset Manager (SAM)CloudForms System Engine
WHAT IS TORQUEBOX AND JRUBY
JRUBYRuby 1.8/1.9 on JVMmature and stable projectJIT and AOTbidirectionalis in Fedora
TORQUEBOXapplication platform for Ruby on Rails, Sinatra...runs atop of JBoss ASoffers services like messaging, scheduling, cachingallows use of clustering, load-ballancing and HAuses standards where possible
WHY SHOULD WE CARE?why to port to JVM
WHY SHOULD WE CARE?memory :-)
WHY SHOULD WE CARE?performance (skipping for this talk)
WHY SHOULD WE CARE?
MEMORY IS THE ISSUEbefore that we need to cover threads
MRI RUBY 1.8green threads
MRI RUBY 1.9native threads with GIL
GLOBAL INTERPRETER LOCKany time one thread is running _Ruby_ codeno other thread can be running _Ruby_ code
GLOBAL INTERPRETER LOCK
GLOBAL INTERPRETER LOCKsignificant barrier to parallelismdoes _not_ limit I/O by the designbut many native rubygems also limits I/O
STATE OF THREADING IN MRI RUBYnot the best
FORKING SERVERS IN MRI RUBYthreads are not the only options for web concurencyprocess forking can do the thing tooLinux is good in forkingunfortunately MRI Ruby can't leverage COW memory
FORKING SERVERS IN MRI RUBYRuby Enterprise Edition solves this for 1.8Ruby 1.9 has many REE optimalizations (but not COW)Ruby 2.0 will finally deliver COW-friendly forking(bitmaps)
STATE OF FORKING IN MRI RUBYnot the best
DEPLOYMENT OPTIONS WITH RUBYThe Ruby community has always insisted that performance
is not an issue while constantly searching for higherperformance web servers and application stacks. -- Greg
Weber
DEPLOYMENT OPTIONS WITH RUBYevented programming (reactor pattern) brings some
parallelism
DEPLOYMENT OPTIONS WITH RUBYforking - phusion passenger, unicornevented - thin, goliath, vert.xthreaded - mongrel, torquebox
DEPLOYMENT OPTIONS WITH RUBYcombination of forking + eventedcombination of threaded + eventedcombination of threaded + forking
DEPLOYMENT OPTIONS WITH RUBYthe issue with evented servers (thin) is granularity
DEPLOYMENT OPTIONS WITH RUBYcontroller - sql* - render - response*controller - sql* - render - response*controller - sql* - render - response*
DEPLOYMENT OPTIONS WITH RUBYto unleash power of evented processing, you need to rewrite
your application (fibers, goliath, node.js, vert.x, asyncsinatra)
DEPLOYMENT OPTIONS WITH RUBYthere are not many options for threading setups
SO WHEN TO CONSIDER JRUBY?you have an app that is not build around evented patternyour app takes decent amount of memoryyour app also contains lots of I/O operations (SQL,messaging, REST calls)you want to scale up
WARNINGI did not cover Rubinius or REE which partially solves some
of these issues
BY THE WAYthe following languages have concurrency built in the
runtimeerlanghaskellgoogle go
AND WHAT'S JAVA
BEFORE YOU START
TRY WITH JRUBY FIRSTinstead of torquebox
SLOW STARTjruby start a little bit slower
OPTIMIZE JRUBY START# JAVA_OPTS="-client -Djruby.compile.mode=OFF" \bundle exec rails server
OPTIMIZE JRUBY START# JRUBY_OPTS="--1.9 -J-XX:+CMSClassUnloadingEnabled \-J-XX:+UseConcMarkSweepGC \-J-XX:MaxPermSize=256m -J-Xmx1800m" \bundle exec rails server
OPTIMIZE JRUBY START# jruby --ng-server &# bundle exec rails server
KATELLO START IN DEV$ time bundle exec rake environment
real 0m19.876suser 0m18.244ssys 0m0.764s
KATELLO START IN PROD$ time rake environment
real 0m13.322suser 0m9.979ssys 0m2.817s
RUBYGEMS ARE SLOWmultiple directories approachruby needs to walk the treemany stat/open calls with ENOENTbundler adds more dirsrvm/rbenv adds even more dirs
HOW RUBY HANDLES REQUIRE# strace rake environment 2>&1 | grep ENOENT...open("x/ldap_fluff-0.1.3/lib/singleton.rb", O_RDONLY) = -1 ENOENTopen("x/net-ldap-0.3.1/lib/singleton.rb", O_RDONLY) = -1 ENOENT open("x/jshintrb-0.2.1/lib/singleton.rb", O_RDONLY) = -1 ENOENT open("x/js-routes-0.6.2/lib/singleton.rb", O_RDONLY) = -1 ENOENTopen("x/jammit-0.6.5/lib/singleton.rb", O_RDONLY) = -1 ENOENTopen("x/yui-compressor-0.9.6/lib/singleton.rb", O_RDONLY) = -1 ENOENTopen("x/i18n_data-0.3.3/lib/singleton.rb", O_RDONLY) = -1 ENOENT...
HOW RUBY HANDLES REQUIREO(n^2)
HOW RUBY HANDLES REQUIREoptimized in ruby 2.0
HOW RUBY 2.0 HANDLES REQUIREO(n^2) - k
HOW RUBY HANDLES REQUIREenough theory!
RUBYGEMS IN KATELLO# bundle install | wc -l120
KATELLO STAT/OPEN MISSES IN PROD# strace bundle exec rake environment 2>&1 | grep ENOENT | wc -l4023
KATELLO STAT/OPEN MISSES IN DEV# strace bundle exec rake environment 2>&1 | grep ENOENT | wc -l172342
KATELLO STAT/OPEN MISSES IN DEV
RUBYGEMS ARE SLOWand it's not getting better
RUBYGEMS ARE SLOWavoid bundleravoid rvm/rbenvuse bundler_ext
PORTING ISSUES
BINARY FILESwriting to a binary file needs b-flag
BINARY FILESFile.open("thefile.bin", 'wb') do |f| f.write(stuff)end
ACTIVERECORDinstall proper gems
ACTIVERECORDif defined? JRUBY_VERSION gem 'activerecord-jdbc-adapter' gem 'jdbc-postgres', gem 'activerecord-jdbcpostgresql-adapter',else gem 'pg'end
ACTIVERECORDvarious versions (rails 3.0 vs new adapter)
ACTIVERECORDERROR undefined method collect' for "created_at DESC":String (NoMethodError).../activerecord-jdbc-adapter-1.2.6/lib/arjdbc/postgresql/adapter.rb:620:in distinct'...
OTHER ISSUESimproper rails namespace:-)
RUBY AND SYSTEMTAP
WHAT IS SYSTEMTAPfree software infrastructure to simplify the gathering of
information about the running Linux system
WHY SYSTEMTAP IS USEFULno need to modify your appno need to restart it
WHY SYSTEMTAP IS USEFULsteep learning curveC-like syntax
WHY SYSTEMTAP IS USEFULvery low-levelsupports high-level (JVM, Python, Ruby)
WHY SYSTEMTAP IS USEFULpart of RHEL and Fedorakernels are systemtap readyRuby extension part of RHEL 6.2 (RHSA-2011-1581)
WHY SYSTEMTAP IS USEFULproject documentation and wikiRHEL6 SystemTap Beginners Guide
INSTALL SYSTEMTAP# yum -y install \systemtap \systemtap-runtime \kernel-debuginfo-uname -r \kernel-debuginfo-common-uname -i -uname -r \kernel-devel-uname -r
UC1: HUNTING FILE CHANGE# touch /test
UC1: HUNTING FILE CHANGE# ls -i /test274
UC1: HUNTING FILE CHANGE# ll /dev/md-0brw-rw----. 1 root disk 253, 0 Apr 17 10:23 /dev/dm-0
UC1: HUNTING FILE CHANGE# cat filechange.stpglobal ATTR_MODE = 1probe kernel.function("setattr_copy")!, kernel.function("generic_setattr")!, kernel.function("inode_setattr") { dev_nr = $inode->i_sb->s_dev inode_nr = $inode->i_ino
if (dev_nr == MKDEV($1,$2) # major/minor device && inode_nr == $3 && $attr->ia_valid & ATTR_MODE) printf ("%d %s(%d) %s 0x%x/%u %o %d\n", gettimeofday_us(), execname(), pid(), probefunc(), dev_nr, inode_nr, $attr->ia_mode, uid())}
UC1: HUNTING FILE CHANGE# stap -v filechange.stp 253 0 274 &# chmod 600 /test1334676922011223 chmod(6157) generic_setattr 0xfd00000/274 100600 0
UC2: DOWN THE RUBY STACK# cat factorial.rbdef factorial n f = 1; for i in 1..n; f *= i; end; fendputs factorial 42
UC2: DOWN THE RUBY STACK# cat calls.stp probe ruby.function.entry{ printf("%s => %s.%s in %s:%d\n", thread_indent(1), classname, methodname, file, line);}probe ruby.function.return{ printf("%s <= %s.%s in %s:%d\n", thread_indent(-1), classname, methodname, file, line);}
UC2: DOWN THE RUBY STACK# stap calls.stp -c "ruby factorial.rb"1405006117752879898543142606244511569936384000000000 0 ruby(16160): => Module.method_added in factorial.rb:1 13 ruby(16160): <= Module.method_added in factorial.rb:1 0 ruby(16160): => Object.factorial in factorial.rb:5 25 ruby(16160): => Range.each in factorial.rb:2 61 ruby(16160): => Fixnum.* in factorial.rb:2 ... 705 ruby(16160): <= Bignum.* in factorial.rb:2 712 ruby(16160): <= Range.each in factorial.rb:2 718 ruby(16160): <= Object.factorial in factorial.rb:2 0 ruby(16160): => Object.puts in factorial.rb:5 20 ruby(16160): => Bignum.to_s in factorial.rb:5 38 ruby(16160): <= Bignum.to_s in factorial.rb:5 53 ruby(16160): => IO.write in factorial.rb:5 74 ruby(16160): <= IO.write in factorial.rb:5 81 ruby(16160): => IO.write in factorial.rb:5 99 ruby(16160): <= IO.write in factorial.rb:5 106 ruby(16160): <= Object.puts in factorial.rb:5
UC2: DOWN THE RUBY STACK# cat rubycount.stp global fn_calls;probe ruby.function.entry{ fn_calls[classname, methodname] <<< 1;}
probe end { foreach ([classname, methodname] in fn_calls- limit 30) { printf("%dx %s.%s\n", @count(fn_calls[classname, methodname]), classname, methodname); }
delete fn_calls;}
UC2: DOWN THE RUBY STACK# stap rubycount.stp -c "ruby factorial.rb"140500611775287989854314260624451156993638400000000021x Bignum.*21x Fixnum.*2x IO.write1x Module.method_added1x Range.each1x Bignum.to_s1x Object.puts1x Object.factorial
UC3: RUBY "TOP"# cat ./ruby-top-modified.stpglobal fn_calls[10240];probe ruby.function.entry { if (isinstr(file, "katello")) fn_calls[pid(), file, methodname, line] <<< 1;}probe timer.ms(4000) { ansi_clear_screen() printf("%6s %80s %6s %25s %6s\n", "PID", "FILENAME", "LINE", "FUNCTION", "CALLS") foreach ([pid,filename,funcname,lineno] in fn_calls- limit 15) { printf("%6d %80s %6d %25s %6d\n", pid, filename, lineno, funcname, @count(fn_calls[pid, filename, funcname, lineno])); } delete fn_calls;}
BUNDLER_EXT
BUNDLER_EXThttp://rubygems.org/gems/bundler_ext
https://github.com/aeolus-incubator/bundler_ext
BUNDLER_EXT# cat Gemfilegem 'rails', '3.0.10'gem 'json'gem 'rest-client', :require => 'rest_client'gem 'jammit', '>= 0.5.4'gem 'rails_warden', '>= 0.5.2'gem 'net-ldap'gem 'oauth'gem 'ldap_fluff'
BUNDLER_EXTif File.exist?(File.expand_path('../../Gemfile.in', __FILE__)) require 'bundler_ext' BundlerExt.system_require(File.expand_path('../../Gemfile.in', __FILE__), :group1, :group2, Rails.env)else Bundler.require :group1, :group2, Rails.envend
WE ARE DONE
CREDITSGreg Weber - http://blog.gregweber.info/posts/2011-06-16-high-performance-rb-part3Ilya Grigorik -http://www.igvita.com/2008/11/13/concurrency-is-a-myth-in-ruby/inc.com - finish line picand world-famouse memegenerator.net