Don’t be a rookie forever.Be in command! (line)
Giuseppe Maxia
@datacharmerThis work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
$Sunday, August 21, 11
about me -Giuseppe Maxia§ a.k.a. The Data Charmer§ QA Director, Continuent, Inc§ Long time hacking with database features§ Formerly, database consultant, designer, coder.§ A passion for QA§ An even greater passion for open source§ ... and community§ Passionate blogger§ http://datacharmer.blogspot.com
Sunday, August 21, 11
What this tutorial is about§ Providing tools and tips for command line users;§ Learn a few tricks to make you achieve more and type
less.
Sunday, August 21, 11
What this tutorial is about§ Providing tools and tips for command line users;§ Learn a few tricks to make you achieve more and type
less.§ Making GUI enthusiasts green with envy;
bar
foo
baz
Sunday, August 21, 11
This tutorial§ A quick guide at the command line usage§ A practical case for database admins§ but ...§ Useful for all
Sunday, August 21, 11
Problem§ One document contains one date in 'mm/dd/yyyy'
format§ Need to convert the date into the 'yyyy-mm-dd' format
(or dd.mm.yyyy)
GUI USER SOLUTION§ open the document with a word processor§ change the date
§ Command line USER SOLUTION§ Probably the same
Sunday, August 21, 11
Problem§ One document contains many dates in 'mm/dd/yyyy'
format§ Need to convert the dates into the 'yyyy-mm-dd'
format (or dd.mm.yyyy)
GUI USER SOLUTION§ open the document with an advanced word processor§ hope it supports regular expressions
§ Command line USER SOLUTION§ Use vi or emacs. Do a search-and-replace operation§ Or use the next page solution
Sunday, August 21, 11
Problem§ Many documents contain many dates in 'mm/dd/yyyy'
format§ Need to convert the dates in all documents into the
'yyyy-mm-dd' format (or dd.mm.yyyy)
GUI USER SOLUTION§ Realizes that it will take days to do this manually§ Scream
§ Command Line USER SOLUTION§ uses sed or perl to do the replacement in one line
Sunday, August 21, 11
the command line
WHY8
?
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1• Windows NT
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000• Windows XP
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000• Windows XP• Windows Vista
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000• Windows XP• Windows Vista• Windows 7
Sunday, August 21, 11
An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-
9
• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000• Windows XP• Windows Vista• Windows 7
• Unix shell
• Linux
• Mac OSX
Sunday, August 21, 11
Why the command line• portability• efficiency• fun
10
Sunday, August 21, 11
the command line (for rookies)
NO PANIC
11
||||
Sunday, August 21, 11
Command line panic (for rookies)
$_Sunday, August 21, 11
Command line panic (for rookies)
$_NOW WHAT?
Sunday, August 21, 11
the command line (for rookies)
BASICS
13
Sunday, August 21, 11
The shell• It's a command interpreter• There are many flavors
> bash > zsh> ksh
14
Sunday, August 21, 11
MySQL command line client• called "mysql"
> Takes SQL commands> sends them to the server> gets results
• but it can do much more ...
15
Sunday, August 21, 11
other tools• Shell scripts• Aliases• Perl• awk• sort• …. and a lot of Unix little helper
16
Sunday, August 21, 11
other tools• Shell scripts• Aliases• Perl• awk• sort• …. and a lot of Unix little helper
16
which could replace everything else, of course
Sunday, August 21, 11
in case it wasn't clear (yeah, for rookies)
17
Sunday, August 21, 11
in case it wasn't clear (yeah, for rookies)
• command line actions are
17
Sunday, August 21, 11
in case it wasn't clear (yeah, for rookies)
• command line actions are> commands (the name of the application to invoke)
17
Sunday, August 21, 11
in case it wasn't clear (yeah, for rookies)
• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)
17
Sunday, August 21, 11
in case it wasn't clear (yeah, for rookies)
• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)> and the ENTER key
17
Sunday, August 21, 11
in case it wasn't clear (yeah, for rookies)
• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)> and the ENTER key
17
Enter
Sunday, August 21, 11
The Unix philosophy (1)• One tool for every feature
> make one tool that does one thing well> for many features, make many tools> make them work together (see next slide)
18
Sunday, August 21, 11
The Unix philosophy (2)• To make several tool cooperate:
> pipe the output of one tool to the input of another one.> for more actions, pipe more tools
$ sort < filename | uniq | nl
19
Sunday, August 21, 11
Unix pipes
20
Sunday, August 21, 11
Unix pipes
20
inputapplication <
Sunday, August 21, 11
Unix pipes
20
outputapplication >
inputapplication <
Sunday, August 21, 11
Unix pipes
20
outputapplication >
inputapplication <
outputapplication >input<
Sunday, August 21, 11
Unix pipes
20
outputapplication >
inputapplication <
outputapplication >input<
application | application
Sunday, August 21, 11
Unix pipes
20
outputapplication >
inputapplication <
outputapplication >input<
application | application
input output
Sunday, August 21, 11
Unix pipes = parallel processing
21
Sunday, August 21, 11
Unix pipes = parallel processing
21
application | application
Sunday, August 21, 11
Unix pipes = parallel processing
21
application | application
Sunday, August 21, 11
Unix pipes = parallel processing
21
application | application
output
Sunday, August 21, 11
Unix pipes = parallel processing
21
application | application
input output
Sunday, August 21, 11
Unix pipes = parallel processing
21
application | application
input output
Sunday, August 21, 11
Unix pipes = parallel processing
21
application | application
input output output
Sunday, August 21, 11
Unix pipes = parallel processing
21
application | application
input output output input
Sunday, August 21, 11
Who gets what• shell• application• database server• pager
22
Sunday, August 21, 11
who gets what
23
shell prompt $ mysql
mysql client
mysql server
Sunday, August 21, 11
any hostwho gets what
24
shellmysql client
mysql server
Sunday, August 21, 11
who gets what$ mysql -u username -ppassword
25
First the applicationthen the server
Sunday, August 21, 11
who gets what$ mysql -u username -ppassword \ -e "select VERSION()"
26
goes to the server
goes to the client, then the server
Sunday, August 21, 11
who gets whatmysql> helpmysql> pager lessmysql> select 1;
27
goes to the server
goes to the client onlygoes to the client only
Sunday, August 21, 11
the command line
readline28
Sunday, August 21, 11
readline• Common component
> shell> mysql client> many CLI applications
29
Sunday, August 21, 11
Main cursor movements
30
Ctrl A Ctrl E
Start/End of line
Meta B Meta F
Back/Forward one word
previous/next command
Ctrl L clear screen and take current line on top
Ctrl R Search history
Sunday, August 21, 11
Changing text
31
Meta U Uppercase word
Meta L Lowercase word
Ctrl D delete word
Ctrl K delete to end of line
Ctrl - undo
for more tips:$ man readline
Sunday, August 21, 11
Some conventions in this talk
$ commandat the shell
mysql> command at the mysql prompt
32
Sunday, August 21, 11
The command line friends
handytools33
#!
Sunday, August 21, 11
your best friends at the command line• man• ls• cd• find• grep• date• echo• pwd• mv• cp 34
Sunday, August 21, 11
man {command_name}• Tells you how to use a command
man ls
35
Sunday, August 21, 11
ls [directory]• lists contents of current [or given] directory
ls
ls /some/path
ls *.txt
36
Sunday, August 21, 11
cd [directory]• change directory to given path• takes to $HOME directory if no path is given
cd /some/path
cd ..
37
Sunday, August 21, 11
find {directory} [conditions]• finds files in the given directory that satisfy conditions
find $HOME/docs -name "*mouse*"
38
Sunday, August 21, 11
grep {pattern} filename• finds pattern matching rows in filename
grep Munich *.txt
grep "Munich\|Berlin" *.txt
grep "[0-9][0-9]:[0-9][0-9]" a.txt
39
Sunday, August 21, 11
date [layout]• displays the current date and time, with optional layout
dateTue Aug 16 08:33:04 CEST 2011
date '+%Y-%m-%d'2011-08-16
40
Sunday, August 21, 11
echo "message"• displays the given message and/or variables
echo "hi"hi
GREETINGS='Hello, there!'echo $GREETINGSHello, there!
41
Sunday, August 21, 11
pwd• displays the current path (Print Working Directory
pwd/home/gmax/test_dir
42
Sunday, August 21, 11
mv old_file_name new_file_name• moves a file from one place to another• renames a file
mv ugly.txt /some/other/pathmv *.txt /storage/texts
mv ugly.txt beautiful.txt
43
Sunday, August 21, 11
cp file_name destination• copies a file from one place to another• creates a copy of a file
cp ugly.txt /some/other/pathcp *.txt /storage/texts
cp ugly.txt beautiful.txt
44
Sunday, August 21, 11
other important friends at the command line
• rm• mkdir• rmdir• cat• sed• strings• wc• head• tail• more/less• sort
45
Sunday, August 21, 11
rm file_name• removes a file
rm ugly.txtrm *.txt
### DANGER!!!!rm -r * rm -rf *
46
Sunday, August 21, 11
mkdir/rmdir directory• creates or removes a directory
mkdir mydir
rmdir mydir
47
Sunday, August 21, 11
cat• shows the contents of a file
cat addresses.txt
48
Sunday, August 21, 11
sed [option] file • performs editing on a file
sed -e 's/Ship/Airplane/' travel.txt
49
Sunday, August 21, 11
strings filename• extracts printable text from a file
strings /bin/ls
50
Sunday, August 21, 11
wc filename• counts words (and characters and lines) in a file
wc .bashrc 72 164 2842 .bashrc# (lines, words, characters)
51
Sunday, August 21, 11
head/tail filename• display the first or the last lines of a file
head long_file.txthead -n 5 long_file.txt
tail long_file.txttail -n 1 long_file.txt
52
Sunday, August 21, 11
more/less filename• display a file with scrollable pages
# old: inefficientmore filename.txt
# new: preferredless filename.txt
53
Sunday, August 21, 11
sort filename• displays the ordered contents of a filename
sort unordered.txt
sort unordered.txt > sorted.txt
54
Sunday, August 21, 11
mixing languages
55
#!
Sunday, August 21, 11
This is a story about languages
Sunday, August 21, 11
you got to understand, Bernie, it's not personal.
It's just that you are obsolete
We use language to talk to computers
Sunday, August 21, 11
10 print "hello"20 goto 10We use
languages to talk to computers
Sunday, August 21, 11
I am a polyglot
Sunday, August 21, 11
An imperfect polyglot, actually
Sunday, August 21, 11
I speak Italian
Sunday, August 21, 11
I speak Italian
with a clear Sardinian accent
Sunday, August 21, 11
I speak English
Sunday, August 21, 11
I speak English
with a distinct Italian accent
Sunday, August 21, 11
I speak Perl
Sunday, August 21, 11
I speak Perl
with a hint of a C accent
Sunday, August 21, 11
I speak French and Spanish
Sunday, August 21, 11
I speak French and Spanishbut since I use them very little, they have melted with Italian and
EnglishSunday, August 21, 11
I speak Bash #!/bin/bash
Sunday, August 21, 11
I speak Bash #!/bin/bash
with a strong Perl accent
Sunday, August 21, 11
I speak fluent Italian
La donzelletta vien dallacampagna, in sul calar delsole, col suo fascio dell'erbae reca in mano un mazzolino di rose e viole, onde, siccome suole, ornare ella si appresta ...
Sunday, August 21, 11
However, Sardinian comes handy sometimes
Maissavidati frimisti! (*)
(*) May you never stopSunday, August 21, 11
I speak fluent English
When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.
Sunday, August 21, 11
I speak fluent English
When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.
hmm.wrong flag
Sunday, August 21, 11
I speak fluent English
When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.
Sunday, August 21, 11
But sometimes Italian is still necessary
maestro
virtuoso
al fresco
presto
spaghettibravo
tempo
dilettantefiasco
studio
stucco
barista
cappuccino
espresso
lasagnapizza
pasta
stanza
finale
lottostiletto
vendetta soprano piano
Sunday, August 21, 11
And sometimes Italian is needed
I tuoi perversidefunti!(*)
(*) Your wicked ancestors!Sunday, August 21, 11
Although Sardinian is still effective here
Ancu ti currullidisa giustizia (*)
(*) May the rule of law chase youSunday, August 21, 11
I speak Perl fluently
#!/usr/bin/perluse strict;use warnings;use File::Find;
while (my $line = <stdin>){ # …
Sunday, August 21, 11
Sometimes, I need help from C
use Inline C;print "9 + 16 = ", add(9, 16), "\n";
__END____C__int add(int x, int y) { return x + y;}
Sunday, August 21, 11
Most of my work at the command line is Bash
$ for D in $HOME/*/; do du -sh $D;> done > /tmp/occupancy
#!/bin/bash
Sunday, August 21, 11
And sometimes, a pinch of Perl can help
$ perl -pe 's{(\d\d)-(\d\d)-(\d\d\d\d)}{$3-$1-$2}g' \ < rawfile.txt > betterfile.txt
#!/bin/bash
Sunday, August 21, 11
And then on top of all this, I need to speak SQL
$ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" '
Sunday, August 21, 11
I wish I could say that my SQL has a Perlish accent
$ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" '
use strict;use DBI;?
Sunday, August 21, 11
I wish I could say that my SQL has a Perlish accent
$ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" '
but it wouldn't be true
Sunday, August 21, 11
I could curse in Perl
$ perl -pe 's{(\d\d)-(\d\d)-(\d\d\d\d)}{$3-$1-$2}g' \ < rawfile.txt > betterfile.txt (*)
(*) already translated
But it would not get me far.
Sunday, August 21, 11
Then
Sunday, August 21, 11
Then
I will need to mix languages.
Sunday, August 21, 11
Then
I will need to mix languages.
This talk will show exactly that.
Sunday, August 21, 11
Then
I will need to mix languages.
This talk will show exactly that.
Mixing languages.
Sunday, August 21, 11
Then
I will need to mix languages.
This talk will show exactly that.
Mixing languages.Yes.
Sunday, August 21, 11
Then
I will need to mix languages.
This talk will show exactly that.
Mixing languages.
$ at the command line
Yes.
Sunday, August 21, 11
the Bash shell
bash81
#!
Sunday, August 21, 11
components• commands• aliases• variables• functions
82
Sunday, August 21, 11
commands$ echo "something"something
83
Sunday, August 21, 11
aliases$ alias mydate='mysql -e "select NOW()"'$ mydate+---------------------+| now() |+---------------------+| 2010-04-12 07:14:19 |+---------------------+
84
Sunday, August 21, 11
aliases$ alias myself='mysql' $ myself -e "select NOW()"'+---------------------+| now() |+---------------------+| 2010-04-12 07:14:19 |+---------------------+
85
Sunday, August 21, 11
variables$ MYBATCH='mysql -N -B'$ MYQUERY='select curdate()' $ $MYBATCH -e "$MYQUERY"2010-04-12
86
Sunday, August 21, 11
nesting variables# t.sh CMD="SELECT TABLE_NAME" TBL='information_schema.tables' WHC1="TABLE_SCHEMA='mysql'" WHC2="TABLE_NAME like 'u%' " Q1="$CMD FROM $TBL" Q2="WHERE $WHC1 AND $WHC2" Q="$Q1 $Q2" echo $Q echo $Q | mysql
87
Sunday, August 21, 11
nesting variables$ sh t.shSELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='mysql' AND TABLE_NAME like 'u%'TABLE_NAMEuser
88
Sunday, August 21, 11
the mysql command line client
89
mysql>
Sunday, August 21, 11
SQL commands from a file$ mysql < filename.sqlmysql> source filename.sql
• executes the commands contained in filename.sql
• commands can be queries understood by the server
• and commands for the client
90
Sunday, August 21, 11
SQL commands at the command line$ application | mysql$ echo "SELECT 1" | mysql$ mysql -e "SELECT 1"
• executes the commands from standard input
• commands can be queries understood by the server
• and commands for the client
91
Sunday, August 21, 11
batch mode$ echo "select curdate()" | mysqlcurdate()2010-04-12
92
Sunday, August 21, 11
table mode$ echo "select curdate()" | mysql -t+------------+| curdate() |+------------+| 2010-04-12 |+------------+
93
Sunday, August 21, 11
table mode$ mysql -e "select curdate()"+------------+| curdate() |+------------+| 2010-04-12 |+------------+
94
Sunday, August 21, 11
Forcing batch modemysql -B -e 'select curdate()'curdate()2010-04-12
95
Sunday, August 21, 11
Forcing batch mode - no headermysql -N -B -e 'select curdate()'2010-04-12
96
IMPORTANT!
Sunday, August 21, 11
Shell + mysql$ echo "The current server's date is `mysql -N -B -e 'select curdate()'`."
$ echo "The current server's date is $(mysql -N -B -e 'select curdate()')."
97
Sunday, August 21, 11
advanced command line techniques
loops98
$_
(-:((((((
@@
Sunday, August 21, 11
loops basics$ for N in 1 2 3> do> echo $N> done123
99
Sunday, August 21, 11
loops basics$ for N in 1 2 3 ; do echo $N ; done123
100
Sunday, August 21, 11
loops basics$ for N in $(seq 1 3) ;\ do echo "this is number $N" ;\ done
this is number 1this is number 2this is number 3
101
Sunday, August 21, 11
loops from file$ head /usr/share/dict/words AaaaaalaaliiaamAaniaardvarkaardwolf
102
Sunday, August 21, 11
loops from file$ for U in \$(head /usr/share/dict/words ) ; do perl -le 'print uc shift' $U ; doneAAAAAALAALIIAAMAANI
103
Sunday, August 21, 11
loops from mysql results$ mysql -B -N -e 'show tables from world'CityCountryCountryLanguage
104
Sunday, August 21, 11
loops from mysql results$ for T in $(mysql -B -N -e 'show tables from world') ;\> do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done
CREATE TABLE world2.City LIKE world.City;CREATE TABLE world2.Country LIKE world.Country;CREATE TABLE world2.CountryLanguage LIKE world.CountryLanguage;
105
Sunday, August 21, 11
piping loops back into mysql$ for T in $(mysql -B -N -e 'show tables from world') ;\> do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done \> | mysql -vv
106
Sunday, August 21, 11
advanced command line techniques
nested
107
$_
@ @
#!
:-)
#!#!#!#!#!
loopsSunday, August 21, 11
doing something for every tablein every database$ mysql -e "show databases"+--------------------+| Database |+--------------------+| information_schema || mysql || sakila || world |+--------------------+
108
Sunday, August 21, 11
doing something for every tablein every database$ mysql -e "show tables from world"+-----------------+| Tables_in_world |+-----------------+| City || Country || CountryLanguage |+-----------------+
109
Sunday, August 21, 11
pseudocodefor each database do for each table do print the table name
110
Sunday, August 21, 11
actual codealias mybatch='mysql -B -N'for D in $(mybatch -e "SHOW SCHEMAS")do for T in \ $(mybatch -e "SHOW TABLES FROM $D") do echo "$D.$T" donedone
111
Sunday, August 21, 11
advanced command line techniques
options files
112
$HOME
Sunday, August 21, 11
Default option files/etc/my.cnf/etc/mysql/my.cnf (Debian, Ubuntu)$BASEDIR/my.cnf$HOME/.my.cnf
113
Sunday, August 21, 11
Custom option files$ mysql --no-defaults$ mysql --defaults-file=filename$ mysql --defaults-extra-file=filename
114
Sunday, August 21, 11
Option files sections[mysqld]# option for server
[client]# option for any mysql client
[mysql]# options specific to 'mysql' CLI app
115
Sunday, August 21, 11
Option files example
[client]user=mynamepassword=mypassword
[mysql]prompt="mysql (\h) {\u} [\d] > "
116
Sunday, August 21, 11
Option files: caution# protect your option files!
$ chmod 600 $HOME/.my.cnf
$ ls -l .my.cnf-rw------- 1 [...] 10:04 .my.cnf
117
Sunday, August 21, 11
promptmysql> prompt gm (\h) {\u} [\d] > PROMPT set to 'gm (\h) {\u} [\d] > 'gm (localhost) {root} [(none)] > use testDatabase changedgm (localhost) {root} [test] >
# \h -> host# \u -> user# \d -> database
118
Sunday, August 21, 11
advanced command line techniques
aliases119
#!
(-:
((((((@@
Sunday, August 21, 11
An useful alias$ alias mysql_as_root=\'mysql --defaults_file=~/.my_as_root.cnf'
mysql_as_root -e "select user()"+----------------+| user() |+----------------+| root@localhost |+----------------+
120
Sunday, August 21, 11
An useful alias#.my_as_root.cnf'[client]user=rootpassword=your_mysql_root_password
[mysql]prompt="### [\h] {\u} (\d) > "
121
Sunday, August 21, 11
Another useful aliasalias mybatch='mysql -B -N '
mybatch -e "select version()"5.1.45-log
122
Sunday, August 21, 11
advanced command line techniques
functions123
#!
(-:
((((((@@( )
Sunday, August 21, 11
functions• like aliases, but with parameters
124
Sunday, August 21, 11
sample functionfunction grant_new { mysql_as_root -e \ "GRANT $1 on $2 to $3 identified by \"$4\" "}
$ grant_new SELECT "world.*" john \ john_secret
125
Sunday, August 21, 11
sample function$ mysql_as_root \ -e 'show grants for john\G'*** 1. row ***Grants for john@%: GRANT USAGE ON *.* TO 'john'@'%' IDENTIFIED BY PASSWORD '*73A88CE13374128844E3ABB500F439A9C02C53A3'*** 2. row ***Grants for john@%: GRANT SELECT ON `world`.* TO 'john'@'%'
126
Sunday, August 21, 11
advanced command line techniques
getting query
details127
$_
Sunday, August 21, 11
which query is being executed?# wrong.sql# this script has one erroruse test ;drop table if exists t1 ;create table t1 (id int);insert into t1 values (a);
128
Sunday, August 21, 11
which query is being executed?$ mysql < wrong.sqlERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list'
# WHERE IS THE ERROR?
129
Sunday, August 21, 11
which query is being executed?$ mysql -v < wrong.sql--------------drop table if exists t1----------------------------create table t1 (id int)----------------------------insert into t1 values (a)--------------ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list'
130
Sunday, August 21, 11
getting execution times$ mysql -vv -e \'select rating,count(*) from sakila.film group by rating'+--------+----------+| rating | count(*) |+--------+----------+| G | 178 || PG | 194 || PG-13 | 223 || R | 195 || NC-17 | 210 |+--------+----------+5 rows in set (0.00 sec) 131
Sunday, August 21, 11
$echo 'select rating,count(*) from sakila.film group by rating' | mysql -vv--------------select rating,count(*) from sakila.film group by rating--------------
rating count(*)G 178PG 194PG-13 223R 195NC-17 2105 rows in set
132
getting execution times (batch mode)
Sunday, August 21, 11
getting execution times (batch mode)$ echo 'select rating,count(*) from sakila.film group by rating' | mysql -vvv --------------select rating,count(*) from sakila.film group by rating--------------+--------+----------+| rating | count(*) |+--------+----------+| G | 178 || PG | 194 || PG-13 | 223 || R | 195 || NC-17 | 210 |+--------+----------+5 rows in set (0.00 sec) 133
Sunday, August 21, 11
advanced command line techniques
getting distinct results
134
$_
Sunday, August 21, 11
getting a single value$ mysql -e "show variables like 'port'"+---------------+-------+| Variable_name | Value |+---------------+-------+| port | 3306 |+---------------+-------+
# Here, we only want the port number
135
Sunday, August 21, 11
getting a single value$ mysql -NB -e "show variables like 'port'" port 3306
# no header, batch mode# still two values
136
Sunday, August 21, 11
getting a single value$ mysql -NB \ -e "show variables like 'port'" \ | awk '{print $2}'
3306
$ mysql -NB \ -e "show variables like 'port'" \ | perl -lane 'print $F[1]'
3306 # filtering the wanted value
137
Sunday, August 21, 11
getting a single value: is replication working?
$ mysql \ -e "show slave status \G" \ | grep -i slave_io_running
Slave_IO_Running: Yes
# too many things. # We only want a 'yes' or 'no'
138
Sunday, August 21, 11
getting a single value: is replication working?
$ mysql \ -e "show slave status \G" \ | grep -i "slave_io_running" \ | perl -lane 'print $F[1]'
Yes
# yes!
139
Sunday, August 21, 11
What's perl -lane?
140
Sunday, August 21, 11
What's perl -lane?
140
perl -lane must turn perl
Sunday, August 21, 11
Perl command line must-know facts (1)perl -e 'print "hello\n"'# executes perl code
141
Sunday, August 21, 11
Perl command line must-know facts (2)perl -ne 'print if /^a/'# starts a loop from standard input# and executes the perl code for each# iteration
perl -MO=Deparse -ne 'print if /^a/'LINE: while (defined($_ = <ARGV>)) { print $_ if /^a/;}
142
Sunday, August 21, 11
Perl command line must-know facts (3)perl -pe 's/(^a\w+)/$1/'# like -ne, but prints each line
$ perl -MO=Deparse -pe 's/(^a\w+)/$1/'LINE: while (defined($_ = <ARGV>)) { s/(^a\w+)/$1/;}continue { print $_;}
143
Sunday, August 21, 11
Perl command line must-know facts (4)perl -le 'print "hello"'# strips newlines from input and adds # them to output
$ perl -e 'print "hello"'hello$ $ perl -le 'print "hello"'hello$ 144
Sunday, August 21, 11
Perl command line must-know facts (5)perl -lane 'print $F[0]'# splits input into @F array
$ echo "a b c" | \ perl -lane 'print $F[1]'b $ echo "a b c" | \ perl -lane 'print $F[2]'c 145
Sunday, August 21, 11
Perl and shell contention for $
146
Take the $$$ and run
Perl command line must-know facts (6)
#!/bin/bash
Sunday, August 21, 11
Beware of $VARIABLES on the command line$ perl -le "$x=1; print $x" syntax error at -e line 1, near "="Execution of -e aborted due to compilation errors.
147
Sunday, August 21, 11
Beware of $VARIABLES on the command line$ perl -le "$x=1; print $x" syntax error at -e line 1, near "="Execution of -e aborted due to compilation errors.
147
consumed by the shell, before Perl can see it
double quotes allow the shell to interpolate variables
Sunday, August 21, 11
Beware of $VARIABLES on the command line$ perl -le "$x=1; print $x" syntax error at -e line 1, near "="Execution of -e aborted due to compilation errors.
147
consumed by the shell, before Perl can see it
consumed by the shell, before Perl can see it
consumed by the shell, before Perl can see it
double quotes allow the shell to interpolate variables
Sunday, August 21, 11
Beware of $VARIABLES on the command line$ perl -le "$x=1; print $x" syntax error at -e line 1, near "="Execution of -e aborted due to compilation errors.
147
consumed by the shell, before Perl can see it
consumed by the shell, before Perl can see it$ perl -le '$x=1; print $x'
1
consumed by the shell, before Perl can see it
double quotes allow the shell to interpolate variables
Sunday, August 21, 11
Beware of $VARIABLES on the command line$ perl -le "$x=1; print $x" syntax error at -e line 1, near "="Execution of -e aborted due to compilation errors.
147
consumed by the shell, before Perl can see it
consumed by the shell, before Perl can see it$ perl -le '$x=1; print $x'
1
consumed by the shell, before Perl can see it
single quotes prevent variable consumption by the shell
consumed by the shell, before Perl can see it
double quotes allow the shell to interpolate variables
Sunday, August 21, 11
advanced command line techniques
combining column names
148
Sunday, August 21, 11
combining column names$ mysql -N -e 'desc world.City'+-------------+----------+----+-----+| ID | int(11) | NO | PRI || Name | char(35) | NO | | | CountryCode | char(3) | NO | | | District | char(20) | NO | | | Population | int(11) | NO | | +-------------+----------+----+-----+
149
Sunday, August 21, 11
combining column names# Expected result:# SELECT CONCAT_WS('|', # ID,Name,CountryCode,District, # Population) from City
150
Sunday, August 21, 11
combining column names (I)echo "select concat_ws('|', $(mysql -N -e 'desc world.City' | \ perl -lane 'push @f, $F[0];' \ -e 'END{print join ",", @f}')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Population) from world.City
151
Sunday, August 21, 11
combining column names (II)echo "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="City" and table_schema="world"')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Population) from world.City
152
Sunday, August 21, 11
combining column names (IIa)echo "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) from world.Country"select concat_ws('|', Code,Name,Continent,Region,SurfaceArea,IndepYear,Population,LifeExpectancy,GNP,GNPOld,LocalName,GovernmentForm,HeadOfState,Capital,Code2) from world.Country
153
Sunday, August 21, 11
piping the result back to mysqlecho "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) as line from world.Country" | mysql | head -n 4lineAFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919|22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate|Mohammad Omar|1|AFNLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000|78.3|371362.00|360478.00|Nederland|Constitutional Monarchy|Beatrix|5|NLANT|Netherlands Antilles|North America|Caribbean|800.00|217000|74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of The Netherlands|Beatrix|33|AN
154
Sunday, August 21, 11
advanced command line techniques
dumping data
155
$_
Sunday, August 21, 11
advanced command line techniques
dumping data
155
$_
Sunday, August 21, 11
dumping a tab delimited data set # the naive way$ mysql \ -e "SELECT * into outfile '/tmp/city_data' from world.City"
# it works only if:# - /tmp/city_data does not exist# - /tmp is world writable
156
Sunday, August 21, 11
dumping a tab delimited data set $ mysql -B -N \ -e "SELECT * from world.City" \ > /tmp/city_data
# it works always
# if you omit the "-N" it also # includes a header
157
Sunday, August 21, 11
dumping a CSV $ mysql -B \ -e "SELECT * from world.City" \ | perl -F"\t" -lane \ 'print join ",", map {s/"/""/g; /^[\d.]+$/ ? $_ : qq("$_")} @F ' \ > /tmp/city_data
158
Sunday, August 21, 11
dumping a CSV $ mysql -B \ -e "SELECT * from world.City" \ | perl -F"\t" -lane \ 'print join ",", map {s/"/""/g; /^[\d.]+$/ ? $_ : qq("$_")} @F ' \ > /tmp/city_data
158
splits the line by tabs, instead of the default whitespace
Sunday, August 21, 11
advanced command line techniques
scriptit
159
Sunday, August 21, 11
use the mysql.sh script§ The companion code for this tutorial§ http://tinyurl.com/cli-tut-code-2010
§ It will save you much typing§ But use it at your own risk!
160
Sunday, August 21, 11
Usage• download the script• load it
161
$ source mysql.sh
Sunday, August 21, 11
mysql.sh functions• my_batch "query"• returns the result of the query. • Without header • Without box
162
$ my_batch "SELECT * FROM test.t1"1 aaa2 bbb3 ccc
Sunday, August 21, 11
mysql.sh functions• my_value "what"• returns a single value
• e.g.
163
$ my_value "version()"5.1.49
$ my_value "@@server_id"1
Sunday, August 21, 11
mysql.sh functions• my_var "what"• returns the value of a given variable
• e.g.
164
$ my_var port3306
$ my_var datadir/usr/local/mysql/data
Sunday, August 21, 11
mysql.sh functions• my_status "what"• returns the value of a given status item
• e.g.
165
$ my_status com_select2653
$ my_status connections452
Sunday, August 21, 11
mysql.sh functions• my_dump_dataset "query" [header]• returns the values of a given query separated by tabs• if a header is requested, it will be part of the result• e.g.
166
$ my_dump_dataset "select * from world.City" | head -n 51 Kabul AFG Kabol 17800002 Qandahar AFG Qandahar 2375003 Herat AFG Herat 1868004 Mazar-e-Sharif AFG Balkh 1278005 Amsterdam NLD Noord-Holland 731200
Sunday, August 21, 11
mysql.sh functions• my_csv "query" [separator]• returns the values of a given query separated by
commas, or by a given separator• e.g.
167
$ my_csv 'select * from world.City' |head -n 3"ID","Name","CountryCode","District","Population"1,"Kabul","AFG","Kabol",17800002,"Qandahar","AFG","Qandahar",237500
Sunday, August 21, 11
inside the command line client
handling
output168
$_
@ @
:-)
Sunday, August 21, 11
pagermysql> SELECT * FROM world.City;
rec …rec …rec …
4000 + recordsunreadable
169
Sunday, August 21, 11
pagermysql> pager lessPAGER set to 'less'mysql> SELECT * FROM world.City;
# [ records in a manageable window ]
mysql> nopagerPAGER set to stdout
# back to normal170
Sunday, August 21, 11
More pager tricksmysql> pager vim -# like 'less', but you can also edit# the results and eventually save # them
171
Sunday, August 21, 11
More pager tricksmysql> pager md5sumPAGER set to 'md5sum'mysql> select * from City;5d17ffa50d6da200dee621823ade2543 -4079 rows in set (0.01 sec)
# produce a CRC for the result set# instead of displaying it.# Useful for data comparison.# (Thanks to Baron Schwartz
172
Sunday, August 21, 11
Comparing data sets with md5summysql> select * from City;5d17ffa50d6da200dee621823ade2543 -
mysql> select * from City order by ID;5d17ffa50d6da200dee621823ade2543 -
mysql> select * from City order by name;5ab980f5b69a6e65d3b8cd5a95ec94e0 -
mysql> select * from City order by District;2bf37cbe91b3ffeaebc8c4b6084b8784 -
173
Sunday, August 21, 11
Comparing data sets with md5summysql> select * from City;5d17ffa50d6da200dee621823ade2543 -
mysql> select * from City order by ID;5d17ffa50d6da200dee621823ade2543 -
mysql> select * from City order by name;5ab980f5b69a6e65d3b8cd5a95ec94e0 -
mysql> select * from City order by District;2bf37cbe91b3ffeaebc8c4b6084b8784 -
173
Sunday, August 21, 11
More pager tricksmysql> pager cat > /tmp/t1.txtmysql> select "one" union select "two";mysql> pager cat > /tmp/t2.txtmysql> select "one" union select "TWO";mysql> nopagermysql> \! vimdiff -o /tmp/t[12].txt
# see the difference between two # datasets
174
Sunday, August 21, 11
175
Sunday, August 21, 11
logging sessionsmysql> tee mylog.txt
# logs everything that you see on# the screen.# your commands, the results,# warnings, error messages, and so on.
mysql> notee
# back to normal176
Sunday, August 21, 11
read more§ http://datacharmer.blogspot.com
Sunday, August 21, 11
THANKS!Question
timeComments on Twitter: @datacharmer
?
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Sunday, August 21, 11
to do§ add dead languages
§ comparing Latin and ancient Greek to COBOL and Pascal
§ add new tricks§ {x,y} expansion in bash§ \G in MySQL§
Sunday, August 21, 11
Top Related