®

The 11 commandments..Paul Mooney| Senior Architect | Bluewave Technology

Bill Buchan | Director | HADSL

BP306

3

I

(State your name)

4

Pledge solemnly to a flag that I have affection for....

5

Or to a flag that I like in some particular way....

6

Or possibly to a flag that I never even heard of.....

7

To fill out the evaluation for this session

8

And to fill out the evaluation in full

9

And to include fantastic comments on the quality of the speakers

10

And also, to complete the following

11

11

•Add LocalDomainServers to my terminated group

11

•Add LocalDomainServers to my terminated group

•Remove Option Declare

11

•Add LocalDomainServers to my terminated group

•Remove Option Declare

•Delete 10 user accounts at random

11

•Add LocalDomainServers to my terminated group

•Remove Option Declare

•Delete 10 user accounts at random

•Move all code to initialised

12

Because we are all individuals...

13

So say we all :)

®

The 11 commandments..Paul Mooney| Senior Architect | Bluewave Technology

Bill Buchan | Director | HADSL

BP306

Agenda� We shall cover:

What do you need to know

What MUST you do

� This is a technology free areaThis is a discussion on the methodology of being

– A good Administrator

– A good Developer

Administrators and Developers� Are different types of people

� Some would say evil versus good Or good versus evil

� Consider it heaven versus hell But, how is that best described?

Administrators and Developers� Are different types of people

� Some would say evil versus good Or good versus evil

� Consider it heaven versus hell But, how is that best described?

Time to get into character� Developers

Time to get into character� Administrators

The format� We have 11 commandments each

11 for Administrators 11 for Developers

� All have a common goal Defend your services

� Your service is key servers and applications everything else is secondary

Admins 1: What makes a good admin?� What makes a good Admin:

Stubborn refusal to change crap Absolute hatred of everyone Anti-social behaviour and general air of Menace Feet up on desk - means everything is running Favourite word: No

� Because Change is bad Service is good

– Are you meeting your SLA?– If not - what’s the cause?– Can you address this with low risk?

Developers 1: What makes a good Developer?� What makes a good developer

Good Attitude, empathy Ability to change complex systems without failure Generally good code ability Multiple code platform strengths Very good customer facing skills:

– Don’t call the customers ‘idiots’. Ever Communication Favourite word: Yes

� Because Change is Good!

Admins 2: Thou shall have change control!� Why?

Prevention of infrastructure changes on the fly keeps your service stable– Protect your service

Change control is risk management.....

� Problems caused? “one change” has caused massive service failures over the years

– Point release upgrade without research...– Slight change to replication/routing

� How? Implement strict change control in

your domino environment. What’s your process? Where is your documentation?

Developers 2: Thou shall have change control!� Why?

Change control is risk management.....– All changes are well understood, documented

Change Management protects the environment and protects YOU

� Problems caused? “one change” has caused massive service failures over the years

– Want to generate 4m documents per HOUR in 56 replicas worldwide, creating 56gb of replication conflicts ?

� How? Three stage environment

– Development Sandbox– User Test Environment– Production

Admins 3: Thou shall understand security!� Why ?

Failure to secure means you’re a target

� Problems Caused? Anonymous Editor access to directory from Internet Everyone’s ID still in the NAB with password LotusNotes Anonymous Editor access to donation site - all credit card

details visible Open ECLs?

� How? Be aware of the Lotus Domino Security Model Conduct informal audits

– Catalog.nsf, ‘ACL view’.

Admins 3: Thou shall understand security!� 7 layer security is based on:

Cross Certification Deny/Allow Server Access Lists Database ACL Reader Fields / Form / View security Encrypted Fields Client execution

� Know your server security page Anonymous access switched on ?

� Know your group infrastructure and how this relates to security� Who is allowed Full Access Administration ? Only you ?

� Check your certificates. Should they be rolled over now ? 64-bit RSA is now factorable. Upgraded your certificates since v6 ?

Developers 3: Thou shall understand security!� Why?

Failure to understand security leads to:– Private information becoming public– Unhappy Customers– Reputation as an ass

� Problems Caused? Your users should only be allowed to see what you allow

them to see.

Developers 3: Thou shall understand security!� How:

Developers have a multi-tier security model - Within applications– Database ACL, Roles– Reader/Access Fields– Public. Private encryption keys

Understand Reader Fields– Multi-value array of CANONICALISED Names– Use the notesItem.isReader or notesItem.isAuthor field– Always have a fallback reader field– What happens when AdminP removes your last reader?

Understand Encryption: Use the ones most appropriate for the application

Admins 4: Thou shall deploy the ECL� Why?

Execution Control Lists is a 10 year old Lotus Notes (client) feature Using it, you can control the execution of code on clients

– Prevent unsigned code execution– Provide extra security

� Problems caused Domino is virus free?

– Too many times we have witnessed any code available to execute in Domino– A good/bad developer could have a fun time there

Admins 4: Thou shall deploy the ECL (cont)� How?

Simple Steps Register two signer id files

– One for restricted execution– One of unrestricted execution

NEVER give them to developers Grant these ids rights to run code in the server documents

Admins 4: Thou shall deploy the ECL (cont)� How?

Create an administration ECL from the admin client Add in the signer id files with rights Use policies in directory to push out the ECL to the clients

– Replace/Refresh?

� Be aware This is NOT as simple as it is documented In sites with little control All code must be reviewed This is a slow process!

Developers 4: Thou shall write maintainable code� Why?

In Lotus Notes, we tend to change applications far more than in other environments

We also tend to release code far more frequently - our RAD roots mean more versions

This means that the majority of our time is not spent writing new shiny code - its updating the existing stuff

If updating the existing stuff is hard, then it costs more time and money

� Problems Caused: 1200 lines of code in a single function, no errorhandling Printing out a subroutine (onto many pages of paper) and

drawing lines between if/then/else code sequences Buggy, ‘scary’ code

Developers 4: Thou shall write maintainable code� How:

Split up routines into less than a screenful Use Errorhandling

– On every function thats more than 2 lines long. – Call a central errorhandler that gives you ALL your detail

Use defensive coding:– ‘Trust No-One’– Check everything at the start of the function – Bale out if anything goes wrong

Re-use code– So that the majority of your application has already been

tested! Write Business Logic in Script libraries, OO code

– So that they can be applied to multiple UI’s. ● Notes, Web, Web Service

� Test early, test often

Admins 5: Thou shall understand your environment� Why?

Domino administrators are typically not SAN administrators Domino administrators sometimes are not involved in hardware acquisition

� Problems caused Slow/poorly managed RAID arrays SERIOUSLY hamper server performance

– Split arrays for services– Transaction logs– View index buildings– Data– DAOS– OS

Sharing arrays on SANS with other products can make performance totally inconsistent and unreliable

Developers 5: Thou shall measure twice, cut once!� Why?

More time thinking, less time coding, leads to better architecture

You can always write bad code on a good architecture, and it’ll work. Even writing good code on a bad architecture is always a disaster

� Problems Caused: Ever had a project that just never finished. Always more and

more bugs? Ever had a piece of code that EVERYONE has had to fix? Fragile, Difficult to extend applications Always fighting fires, never writing code?

Developers 5: Thou shall measure twice, cut once!� How?

Spend more time listening and less time talking Get hard and non-ambiguous specification, and get them in writing Write specification for your application - even in the ‘About’ document:

– It forces you to think how the application will work– Anyone maintaining the application has a fighting chance

Show the customer your changes, and make sure they understand the impact Get the customer involved in testing so they understand what they get When making changes, always figure out what NEEDS to change first Dont have a ‘rush of code to the fingers’

Admins 6: Thou shall review the catalog� Why?

The catalog.nsf database records a LOT of information about all databases in your environment– Even ones that are not listed as “include in database catalog”– Open the ($replica) view for proof

● Or build a custom view

� Problems caused A neglected catalog.nsf leads to poor security Applications end up with default access to high

– Or worse, anonymous access Go to the ACL view and look at -Default-

Admins 6: Thou shall review the catalog� How

Periodically review the catalog.nsf database Make it part of your monthly review

Use the DDM to monitor ACL changes– Security review probes

Developers 6: Thou shall understand @Formula!� Why?

So you don’t end up writing something horrible - unmaintainable, buggy, slow - in the wrong language

You should use the best features of each language

� Problems caused: Ever seen XML parsing in raw LotusScript? Ever seen a Lotusscript language parser written in

LotusScript ? Ever seen Bubblesort implemented as @Formula Language -

and then copied/pasted between 50+ fields ? It causes unmaintainable, horrible, code.

Developers 6: Thou shall understand @Formula!� How

Developers - Good developers - should understand:– LotusScript - the key to large, maintainable, predictable

code in Notes Client and Scheduled Agents– @Formula - Form and View specification, error handling,

lookups– Java - Good for network access, Web Services, etc.

You can call an agent from another agent - and pass a string as a parameter– Allows you to call Java from LotusScript or vice versa– Pass a NoteID of a document as a parameter– use the Notes document to pass values between agents

Admins 7: Thou shall monitor mail usage� Why?

Domino is amazingly resilient– Router task is outstanding for managing volume/size of messages

HUGE PROBLEM - ITS TOO GOOD– Often ignored– Quotas exceeded - throw more disk space at them

� Problems caused Unrestricted mail usage internally can hamper server performance Typically external mail size is managed, but often we are too forgiving on internal

messages Attachment control Quota control

– Wait for Paul’s rant on Quota control

Admins 7: Thou shall monitor mail usage

� How Message size restrictions Rules - More efficient in a way, but difficult to manage Quotas

– Implement quotas!– Push back at the business, protect the service

Ask Lotus to change the way attachment quotas work? Implement DAOS!

42

User Stupidity : Space saving� Detailed, scientific study taken....

Attachment Duplicates Understand “anykey”

42

User Stupidity : Space saving� Detailed, scientific study taken....

Attachment Duplicates Understand “anykey”

42

User Stupidity : Space saving� Detailed, scientific study taken....

Attachment Duplicates Understand “anykey”

Developers 7: Thou Shall Understand Versions!� Why?

If you use a version of notes to develop code that is ahead of the version that your clients and your servers use, you may write some code that will be impossible for the client or the server to execute

� Problems Caused? You use features that will not run on your users Notes Client Web services - you save an R8 Web Service that does not run

on your R7 Servers Forms, hide-whens, pages, views, all stop working... Servers crash

Developers 7: Thou Shall Understand Versions!� How ?

You should always be developing in a version that is not greater than your oldest client version or your oldest server version

Good practice to have multiple versions of designer, to make sure that you can test and develop in the right place– Difficult to do with nd8. VMWare is your friend.

Admins 8: Thou shall maintain your servers� Why?

Wrong maintenance tasks running on wrong servers Wrong parameters Too frequent/not frequent enough!

� Problems caused Tasks running causing DBID’s to change

– Invalidating transaction log backups Tasks running into daytime Tasks clashing

Admins 8: Thou shall maintain your servers� How?

Understand the tasks! What does compact do? Update versus Updall Fixup.. when.. consequences

� http://www-01.ibm.com/support/docview.wss?uid=swg27006573

Developers 8: Thou shall keep up with Evolution!� Why?

Things change - new releases, new techniques, changes to business

New versions mean new capabilities

� Problems caused: Old, ugly inefficient apps Old, ugly inefficient code Old, ugly inefficient developers

Developers 8: Thou shall keep up with Evolution!� How?

Attitude: – You never will know it all– Everyone knows something you don’t– Get used to it, and learn from others

Learning:– Courses, Books, Blogs,

● http://www.planetlotus.org– Skim the blogs - always something useful

Resources– http://openntf.org - a repository of free, open templates

and code Conferences and User Groups

– Lotusphere, iLug, UKLug, SNUG, etc.– Notes.net

� Remember: You are not alone!

Admins 9: Thou shall NOT set 300 year lifespans� Why?

Its done because its easy No need to re-certify like you had to before No chance of hundreds of people expiring at the same time

� Problems caused Security issues Are your deny lists correct? Domino is as secure as you make it

Admins 9: Thou shall NOT set 300 year lifespans� How?

Use your registration settings in policy documents Use the expiration view in the nab/catalog License tracking? Delegate user management to help desk

– SECURELY

Developers 9: Thou Shall Log!� Why?

Domino has a multi-platform, server and client based, event triggered and scheduled agent execution manager in THREE different languages

Code is executed on servers (on multiple platforms) in multiple locations and on multiple versions

Lets not pretend ALL code is perfect ALL the time on ALL platforms in ALL circumstances. – That’s what Project Mangers think

We need to know before the users tell us:– What went wrong, Why, Where and When– How we can fix it

We need to know when agents have NOT ran– Yearly scheduled agent anyone ?

Developers 9: Thou Shall Log!� Problems Caused:

Complex Business Applications stop, or are unreliable

� How: You need to add Error Handling to all your scheduled routines

– Java, LotusScript, @Formula You need to collect these logs in a logical manner You need to ensure that the process of logging doesn’t slow the application

down significantly, nor generate terrabytes of useless information – Just like an unmonitored DomLog.nsf – We have CEO’s for that

Check out OpenLog on OpenNTF – By Julian Robichaux - openNsf.com

Admins 10: Thou shall manage the minions� Why?

You owe a level of service to your end users... yes... I said that They need to leave you alone You need to make that happen

� Problems caused Look at your support calls

– They are resolved sometimes in firefight mode Look at them again..

– How can you prevent the problem from happening Top 10 support calls are the most frequent

– Wrong location document– Wrong replica– Archive issues etc etc

Admins 10: Thou shall manage the minions� How?

POLICIES– Lock them down– Manage archives.. use server archiving if you can

Location documents– Delete all the location documents that are not used

● Home dialup?– Especially if you are on citrix

Bookmarks– Use policies

● Dont give them an excuse to say they can’t find a database

Developer 10: Thou Shall Be Organised!� Why?

We are asked to do lots of things each day– Develop complex systems– Fix this bug– Add this feature– Send me the specifications for...

� Problems Caused: You forget stuff You’re always apologising You never have enough time to do it all

Developer 10: Thou Shall Be Organised!� How?

Track all bugs for all systems in a database– Check out OpenNtf for bug-tracking templates

Track all your personal actions– Get Things Done - session with Eric Mac and Dave Allen

Stay on top of things Allocate time in your day for actual management of

information as opposed to work Always add ‘contingency’ to your estimates Less Haste More Speed Always ask yourself: Is there a smarter way to do this?

Admins 11: Thou shall learn NSDs� Why?

The NSD is a great tool with excellent data Unfortunately it is written in Kilngon It IS trying to tell you what the problem is...

– You just have to give them a chance

� Problems caused Repetitive crashes

– Could be easily preventable Poor service Disrupted users Homicidal bosses

Admins 11: Thou shall learn NSDs� How?

Take a look at last years “Die Hard” presentation– Introduction to NSDs

Download and use the NSD debugger tool from IBM sandbox– http://www.pmooney.net/pmooney/pmooneynet.nsf/d6plinks/PMOY-77DR4R

The NSD is trying to tell you what’s wrong– Typically a little practice and you end up liking them!

Developers 11: Thou Shall Learn Web Services� Why

– Web Services are an application to application protocol to exchange information

– Web Services give a huge amount of integration potential to your applications

� Problems Solved:– No more using LEI/NotesPump/Notrix to mirror other data

sources in Notes– Real time, online lookup of other data– Far more integration on the .glass

Developers 11: Thou Shall Learn Web Services� How?

Notes 7.x allowed Domino to host Web services Notes 8.x allows notes clients to consume web services

– But alternatives available for older releases Notes 7 allows LotusScript based web services

– So there’s no excuse! nd8 allows Java and LotusScript based web services Use SoapUI for testing

– http://www.soapui.org Other platforms can easy consume Domino Data

– BlackBerry Handsets using MDS Studio– .Net applications using Visual Studio 2005 or later– Adobe Flex– And many many more!

� Admins and Developers share more goals than we all suspect: Service Stability Environment Stability Predictable change management Easy life

� Professionalism Choose where/when to have excitement thrust upon you Dull, in this context, is professional!

– Chinese Curse: May you live in interesting times

Wrapup

� Paul Mooney pmooney@pmooney.net www.pmooney.net

� Bill Buchan bill@hadsl.com www.billbuhan.com

Thank You

Legal disclaimer

© IBM Corporation 2008. All Rights Reserved.

The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both. the Simpsons is a trademark of Fox TV, in the United States, other countries, or both.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Batteries not included. Objects in the mirror may appear closer. Shinks when wet. Buyer beware. Investments may go up as well as down. Portions (c) Ned Flanders, Smithfield. Magnetic North may vary. Without Prejudice.