perl sysadmin dba

IntroducaoSysadmin

DBAConsideracoes finais

Perl para Sysadmins e DBA’s

Lindolfo ”Lorn” Rodrigues

Lindolfo ”Lorn” Rodrigues Perl para Sysadmins e DBA’s

IntroducaoSysadmin

$ whoami

$ id$ uid=1000(lorn) gid=100(users)grupos=11(#sao-paulo.pm), 17(#slackware-br), 18(#catalyst),19(#moose),83(#perl6)

IntroducaoSysadmin

Agenda

Sysadmin

One-linersParsers de logAutomatizacoes

ETLData Cleaning

IntroducaoSysadmin

Agenda

Sysadmin

ETLData Cleaning

IntroducaoSysadmin

Agenda

Sysadmin

ETLData Cleaning

IntroducaoSysadmin

Agenda

Sysadmin

ETLData Cleaning

IntroducaoSysadmin

Agenda

Sysadmin

ETLData Cleaning

IntroducaoSysadmin

Agenda

Sysadmin

ETLData Cleaning

IntroducaoSysadmin

Agenda

Sysadmin

ETLData Cleaning

IntroducaoSysadmin

One-liners

Perl na linha de comando

Lembra muito sed/awk, com todo o poder da PCRE

sed: sed -i.bck -e ’s/foo/bar/g’ arquivo.txtperl: perl -i.bck -npe ’s/foo/bar/g’ arquivo.txt

awk: awk ’{print $1;}’ arquivo.txtperl: perl -npe ’print $F[0]’ arquivo.txt

Perl = Shell Script + awk + sed + ... + CPAN!o/

IntroducaoSysadmin

One-liners

IntroducaoSysadmin

One-liners

IntroducaoSysadmin

One-liners

IntroducaoSysadmin

One-liners

IntroducaoSysadmin

One-liners

IntroducaoSysadmin

One-liners

IntroducaoSysadmin

One-liners

find /var/www -iname ”*.html-exec perl -i.bck -npe’s/foo/bar/g’ ’’

perl -MDBD::mysql -e’DBD::mysql-¿connect(’dbname=yapc;Host=192.168.1.2’,’root’, ’yapc2009’)

IntroducaoSysadmin

One-liners

find /var/www -iname ”*.html-exec perl -i.bck -npe’s/foo/bar/g’ ’’

perl -MDBD::mysql -e’DBD::mysql-¿connect(’dbname=yapc;Host=192.168.1.2’,’root’, ’yapc2009’)

IntroducaoSysadmin

One-liners

dicas.txt ( canivete suico one-liners )

App::Rad!

IntroducaoSysadmin

One-liners

dicas.txt ( canivete suico one-liners )

App::Rad!

IntroducaoSysadmin

One-liners

1 use App : : Rad ' i n c l u d e ' ;2 App : : Rad−>run ;

IntroducaoSysadmin

One-liners

$ perl dicas.pl include ’arruma string’ -i.bck -npe’s/foo/bar/g’ arquivo.txt

$ perl dicas.pl

1 Usage : dicas . pl command [ arguments ]23 Available Commands :4 arruma_string

5 help show syntax and available commands

6 include include one−liner as a command

$ perl dicas.pl arruma string outro arquivo.txt

IntroducaoSysadmin

One-liners

$ perl dicas.pl

IntroducaoSysadmin

One-liners

$ perl dicas.pl

IntroducaoSysadmin

One-liners

IntroducaoSysadmin

Parsers

Como mandar o log do Apache para o Syslog ( tutorial noultimo slide )

Usar Spreadsheet::WriteCustomizar com cores, gerentes/chefes adoram cores

maillog

IntroducaoSysadmin

Parsers

maillog

IntroducaoSysadmin

Parsers

maillog

IntroducaoSysadmin

Parsers

maillog

IntroducaoSysadmin

Parsers

maillog

IntroducaoSysadmin

Pre-requisitos

Hash tables, o que sao e como pode ser usada?

1 my %idade = (2 ” r a u l s e i x a s ” => 10000 ,3 ” a r t h u r d e n t ” => 424 ) ;5 p r i n t $idade{ ' a r t h u r d e n t ' } ;6 # a d i c i o n a n d o i t e n s ao hash7 $idade{” j e r e m i a s ”}=27;8 # d e l e t a n d o i t e n s9 d e l e t e ( $idade{ ' r a u l s e i x a s ' }) ;

IntroducaoSysadmin

Pre-requisitos

Estrutura de um hash:

1 $VAR1 = {2 ' j e r e m i a s ' => 27 ,3 ' a r t h u r d e n t ' => 42 ,4 ' r a u l s e i x a s ' => −100005 } ;

IntroducaoSysadmin

Pega ”spider”

Pegando o ”invasor” do seu site

1 w h i l e (my $line = <STDIN>) {2 my ( $ip ) = s p l i t (/\ s / , $line ) ;3 $hash_count{$ip} = $hash_count{$ip} + 1 ;4 }56 p r i n t Dumper \%hash_count ;7 f o r e a c h my $line ( s o r t { $hash_count{$b} <=> $hash_count{←↩

$a} } k e y s8 %hash_count ) {9 my $acessos = $hash_count{$line } ;

10 p r i n t ” i p : $ l i n e \ t a c e s s o s :\ t $ a c e s s o s \n” ;11 }

IntroducaoSysadmin

Pega ”spider”

1 $VAR1 = {2 ' 1 2 8 . 9 8 . 1 2 0 . 8 4 ' => 2 ,3 ' 3 8 . 9 8 . 1 2 0 . 8 4 ' => 6 ,4 ' 1 1 8 . 9 8 . 1 2 0 . 8 4 ' => ' 1 '

5 } ;6 ip : 3 8 . 9 8 . 1 2 0 . 8 4 acessos : 67 ip : 1 2 8 . 9 8 . 1 2 0 . 8 4 acessos : 28 ip : 1 1 8 . 9 8 . 1 2 0 . 8 4 acessos : 1

IntroducaoSysadmin

Achando arquivo duplicados

1 my @targets = ( ' /home/ l o r n / f o o / ' ) ;2 my $tree = File : : Find : : Object−>new ({} , @targets ) ;3 my %hash_file = ( ) ;4 my $md5 = Digest : : MD5−>new ;5 w h i l e (my $r = $tree−>n e x t ( ) ) {6 i f ( −d $r ) {7 n e x t ;8 }9 open (my $file , '< ' , $r ) or d i e ”$ !\ n” ;

10 $md5−>addfile ( $file ) ;11 my $digest = $md5−>hexdigest ;12 c l o s e $file ;13 my @files = ( $r ) ;14 push ( @{$hash_file{$digest }} , $r ) ;15 }

IntroducaoSysadmin

1 f o r e a c h my $md5hex ( s o r t { $hash_file{$a} <=> $hash_file{←↩$b} } k e y s

2 %hash_file ) {3 i f ( s c a l a r ( @{$hash_file{$md5hex }}) > 1) {4 p r i n t ” A r q u i v o d u p l i c a d o !\ n” ;5 f o r e a c h my $file ( @{$hash_file{$md5hex }}) {6 p r i n t ” f i l e : $ f i l e \n” ;7 }8 }9 }

IntroducaoSysadmin

1 $VAR1 = {2 ' 4 b1a785a1dd6db7b5e4f86c7e56c2d1d ' =>3 [4 ' / U s e r s / l o r n / f o o / a r q u i v o 2 . t x t ' ,5 ' / U s e r s / l o r n / f o o / a r q u i v o 2 3 . t x t '

6 ] ,78 ' 3789 f a 4 8 9 8 d 7 b 7 f 1 5 1 3 c 8 a d 3 9 6 5 c 3 f 9 a ' =>9 [

10 ' / U s e r s / l o r n / f o o / a r q u i v o 1 . t x t ' ,11 ' / U s e r s / l o r n / f o o / a r q u i v o 1 1 . t x t ' ,12 ' / U s e r s / l o r n / f o o /tmp/ a r q u i v o 4 2 . t x t '

13 ]14 } ;

IntroducaoSysadmin

Automatizando tarefas

Qualquer programa com opcao de include, pode serautomatizado

Apache, Samba, Bind

IntroducaoSysadmin

Automatizando tarefas

Qualquer programa com opcao de include, pode serautomatizado

Apache, Samba, Bind

IntroducaoSysadmin

Extract, transform, loading

Nenhuma linguagem carrega dados no banco, mais rapido queo proprio banco

pg copy ( PostgreSQL ) , impdmp ( Oracle )

..mas voce pode modificar o que sera carregado antes de usaro proprio banco

Limpeza de caracteres ’?’dupla dinamic ord - chr

ord: descobre o ”id” do caracter estranhochr: recebe o ”id” do caracter e retorna o mesmomy $id estranho = ord(371);my $char estranho = chr($id estranho);my $texto sujo = s/$char estranho//g;

Validar os dados de entrada ( CNPJ, CPF, etc )

Business::BR::CNPJ Business::BR::CPF

IntroducaoSysadmin

Spreadsheet::Write tambem funciona bem com SELECT

... ou seja, nao precisa gerar um .csv e carregar no seu ”Excel”

IntroducaoSysadmin

Pequel ETL

IntroducaoSysadmin

PL/Perl - PL/PerlU

Palesta do David Fetter ( http://fetter.org )

IntroducaoSysadmin

Data Cleaning

Text::Levenshtein ( Edit Distance ou Levenshtein Distance )

rato - ralorodar - rodo

Algorithm::LCS

Algoritmo usado no diff de codigos

IntroducaoSysadmin

Data Cleaning

Algorithm::LCS

IntroducaoSysadmin

Data Cleaning

Algorithm::LCS

IntroducaoSysadmin

Data Cleaning

Algorithm::LCS

IntroducaoSysadmin

Data Cleaning

Algorithm::LCS

IntroducaoSysadmin

Bibliografia e coisas interessantes

Cultured Perl: Automating UNIX system administration with PerlCultured Perl: One-liners 101Sending Apache httpd Logs to Sysloghttp://www.lornlab.org/ palestra yapc2009/

IntroducaoSysadmin

Obrigado

Duvidas?

lorn at lornlab dot org

IntroducaoSysadmin

Obrigado

Duvidas?

lorn at lornlab dot org

perl sysadmin dba

Technology

Transcript of perl sysadmin dba