Lingering Objects

104
Active Directory Replication Troubleshooting Troubleshooting Lingering Objects DRAFT V9.3 Released: October 17, 2011

description

Lingering Objects

Transcript of Lingering Objects

Page 1: Lingering Objects

Active Directory Replication Troubleshooting

Troubleshooting Lingering Objects

DRAFT V9.3 Released: October 17, 2011

Page 2: Lingering Objects

About the Authors

Author: Justin Turner

Bio:

Justin is a Sr. Support Escalation Engineer with the Directory Services group based in Irving Texas with over 10 years of support and Active Directory experience. Justin has created or contributed too many training courses and KB articles for the Microsoft Knowledgebase.

Project Lead: Justin Turner

Bio:

Page 3: Lingering Objects

Table of Contents

1.0 TAP .......................................................................................................................................................... 1

1.1 Topic ................................................................................................................................................................ 1

1.2 Audience .......................................................................................................................................................... 1

1.3 Purpose ........................................................................................................................................................... 1

1.4 Format ............................................................................................................................................................. 1

2.0 Problem ................................................................................................................................................... 2

2.1 The Problem .................................................................................................................................................... 2

2.2 Potential Challenges ........................................................................................................................................ 2

2.3 Learner’s Needs ............................................................................................................................................... 2

2.4 Instructor’s Needs ........................................................................................................................................... 3

3.0 Learning Expectations ............................................................................................................................. 4

3.1 Learning Goals and Objectives ........................................................................................................................ 4

3.2 Lesson Components ........................................................................................................................................ 4

3.3 Resources ........................................................................................................................................................ 5

4.0 Learning Activities ................................................................................................................................... 6

Focus on goals ....................................................................................................................................................... 6

Connect to prior knowledge .................................................................................................................................. 6

Gain and integrate content knowledge ................................................................................................................. 6

Take action and monitor learning progress .......................................................................................................... 6

Synthesize and evaluation ..................................................................................................................................... 6

Extend and transfer ............................................................................................................................................... 7

5.0 Assessment ............................................................................................................................................. 8

5.1 Assessment Objectives .................................................................................................................................... 8

5.2 Post-course exam ............................................................................................................................................ 9

5.3 Post-course exam Answer Key ...................................................................................................................... 13

5.4 Performance Assessment .............................................................................................................................. 14

5.5 Performance Assessment Rubric ................................................................................................................... 15

6.0 Evaluation ............................................................................................................................................. 16

6.1 Survey Questions ........................................................................................................................................... 16

7.0 Timeline ................................................................................................................................................. 18

8.0 Job Aid ................................................................................................................................................... 20

8.1 Instructor Job Aid .................................................................................................................................. 21

Page 4: Lingering Objects

Course Parameters .............................................................................................................................................. 21

Note to Trainers .................................................................................................................................................. 22

Obtaining Access to Virtual Machines ................................................................................................................. 23

Activities .............................................................................................................................................................. 24

8.2 Learner Job Aid .................................................................................................................................................. 25

Lingering Object Terminology ............................................................................................................................. 25

Tombstone Lifetime Default Values .................................................................................................................... 26

Replication Consistency Settings ......................................................................................................................... 26

Troubleshooting Overview .................................................................................................................................. 29

Repadmin /removelingeringobjects Quick Reference ........................................................................................ 29

Un-hosting a partition ......................................................................................................................................... 30

Manually adding a replication connection using repadmin.exe ......................................................................... 31

Repldiag quick reference ..................................................................................................................................... 32

9.0 Course Workbook ................................................................................................................................. 36

Document Conventions ........................................................................................................................................... 36

Program Code and Commands............................................................................................................................ 36

Notes ................................................................................................................................................................... 37

Tables and Figures ............................................................................................................................................... 37

Course Document and Slide Numbering ............................................................................................................. 37

Lesson 1: Lingering Objects Fundamentals ................................................................................................. 39

What You Will Learn ............................................................................................................................................ 39

Terminology associated with Lingering Object issues ............................................................................................. 39

Lingering Objects ................................................................................................................................................. 39

Tombstone .......................................................................................................................................................... 39

Tombstone Lifetime (TSL).................................................................................................................................... 39

Strict and Loose Replication Constancy .............................................................................................................. 42

Loose Replication Consistency ............................................................................................................................ 42

Strict Replication Consistency ............................................................................................................................. 43

Abandoned object ............................................................................................................................................... 46

Abandoned delete ............................................................................................................................................... 46

Lesson 2: Symptoms and Cause .................................................................................................................. 48

What You Will Learn ............................................................................................................................................ 48

Symptoms of Lingering Objects ............................................................................................................................... 48

Detection of Domain Controllers That Have Not Replicated in the Tombstone Lifetime ................................... 48

Page 5: Lingering Objects

Replication Errors Caused by Lingering Objects .................................................................................................. 50

Cause of Lingering Objects ...................................................................................................................................... 51

How lingering objects occur ................................................................................................................................ 51

Five Causes of Lingering Objects ......................................................................................................................... 51

Lingering Object Prevention ................................................................................................................................ 53

Lesson 3: Identification and Classification .................................................................................................. 54

What You Will Learn ............................................................................................................................................ 54

Create a replication health report ........................................................................................................................... 54

Try This: Generate an AD Replication report using repadmin ............................................................................. 55

Use AD Replication report and repadmin to determine the scope of the problem ................................................ 55

Lesson 4: Lingering Object Removal ........................................................................................................... 57

What You Will Learn ............................................................................................................................................ 57

Methods to Remove Lingering Objects ................................................................................................................... 57

Removing Lingering Objects with Repadmin....................................................................................................... 57

Events Associated with Lingering Object Removal.............................................................................................. 58

Details of Repadmin’s Lingering Object Removal Mechanism ............................................................................ 59

Remove Lingering Objects Using Repldiag .......................................................................................................... 59

Remove Lingering Objects Using Replfix ............................................................................................................. 63

Remove Lingering Object using LDP or Script ..................................................................................................... 63

Remove Lingering Objects by partition re-host operation .................................................................................. 63

Lesson 5: Real World Application ............................................................................................................... 67

What You Will Learn ............................................................................................................................................ 67

Determining What to Do with a Lingering Object ............................................................................................... 67

10.0 Lab Guide ............................................................................................................................................ 69

Lab Sessions ................................................................................................................................................ 70

Setting Up Your Lab Environment ........................................................................................................................... 70

Hardware ............................................................................................................................................................. 71

Software .............................................................................................................................................................. 71

Network Layout ................................................................................................................................................... 72

Computer Names and IP Addresses .................................................................................................................... 72

Configuring Your Computer(s)............................................................................................................................. 73

Accounts and Group Membership ...................................................................................................................... 74

Domain Membership........................................................................................................................................... 74

Shares on Instructor Computer(s) ....................................................................................................................... 75

Page 6: Lingering Objects

Using the Keyboard and Mouse in a Virtual Machine ............................................................................................. 75

Using the Keyboard ............................................................................................................................................. 75

Using the Mouse ................................................................................................................................................. 76

Lab 1: Exploring Lingering Object Fundamentals ........................................................................................ 78

Configuring Your Computer(s) ................................................................................................................................. 78

Configuring Your Virtual Machine Environment ................................................................................................. 78

Accounts and Group Membership ...................................................................................................................... 78

Domain Membership........................................................................................................................................... 79

Exercise 1: Determine Tombstone Lifetime Setting ................................................................................................ 79

Exercise 2: Determine forest and DC replication consistency settings ................................................................... 81

Lab 2: Lingering Object Diagnosis and Documentation .............................................................................. 83

Exercise 1: Lingering Object Diagnosis .................................................................................................................... 83

Exercise 2: Lingering Object Documentation .......................................................................................................... 84

Lab 3: Lingering Object removal using repadmin ....................................................................................... 86

Exercise 1: <Problem Solving Exercise Title> ........................................................................................................... 86

Exercise 2: <Simulation Exercise Title> .................................................................................................................... 87

Lab 4: Lingering Object removal using ldp and repldiag ............................................................................. 89

Lab 5: Abandoned Object and Abandoned Deleted object remediation ................................................... 90

Lab 6: Lingering Link identification and cleanup ......................................................................................... 91

10.0 Presentation Slides.............................................................................................................................. 92

Page 7: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 1

1.0 TAP This will be a half-day course covering Troubleshooting Lingering Objects. The proposed

solution will consist of lecture, classroom discussion, case study and a hands-on laboratory

environment using virtualized domain controllers on a Hyper-V server.

Client: Stacy Raynor | Support Escalation Manager | Microsoft Corporation

Problem: High case TMPI and escalation rate for AD Replication (lingering object) issues

Solution: 6 hour training module

1.1 Topic

Troubleshooting Lingering Objects: Symptom, Cause and Resolution

1.2 Audience

Support Engineers at Microsoft Corporation

1.3 Purpose

The purpose of this workshop is to equip Microsoft Support Engineers with the necessary

background knowledge and skills required to troubleshoot and resolve Active Directory

Replication failures involving Lingering Objects.

1.4 Format

Instructor Led in classroom and remotely through Live Meeting consisting of:

Lecture

Classroom discussion

Case study

Lab

Assessment

Page 8: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 2.0 Problem

2 © 2011 Microsoft Corporation. All rights reserved.

2.0 Problem Analysis of over 3,000 cases revealed that the Total Minutes per Incident (TMPI) for Active

Directory replication issues involving “lingering objects” is more than twice the TMPI average

of standard Active Directory replication cases. Interviews of SMEs and other engineers who

work these issues revealed the following as likely contributors to the higher TMPI metric:

Lack of consolidated documentation

Complicated terminology, troubleshooting and remediation methods

2.1 The Problem

There is one technology area within Active Directory replication that has a higher than

normal TMPI statistic: Lingering Objects. Cases that fall into this area are escalated to the

next level of engineers frequently and take longer to resolve. Engineers will escalate cases for

a number of reasons, one of them being that they do not feel they have the skills to resolve

the problem. While there are a number of factors that can increase a case’s TMPI and

escalation rate, case analysis and engineer interviews reveal that targeted training is the right

approach for this particular area. A targeted 3-5 course module should be sufficient.

2.2 Potential Challenges

Active Directory (AD) Replication is a somewhat broad support topic and the particular

issues that occur within that support topic can vary greatly. Training on such a broad topic in

the past is usually conducted over the course of several days. Targeted, in-depth training on

the more complicated scenarios is preferred over the standard, which is typically broad in

scope with little technical depth. Additionally, support for Microsoft is handled world-wide so

this solution would need to consider options available for remote delivery and/or some type

of self-study component.

Challenges that we may have to deal with:

Consolidation of existing resources

Creation of a comprehensive lab environment in Hyper-V

Course length and modality

2.3 Learner’s Needs

Interviews with SMEs and many engineers that routinely work these issues revealed the

following needs:

Consolidated documentation

o Too many sources of information exist

o “I have over 30 articles to look through when working these issues”

Updated documentation (there are several scenarios un-accounted for in existing

documentation)

Page 9: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 3

o Repldiag was created several years ago to make lingering object cleanup faster and

easier. Case data and SME interviews suggest that this tool is rarely used.

o “The SMEs ask if I’ve already tried X. How would I know to try something when it’s

not documented?”

Terminology is well defined and easy to understand

o “There are a lot of different terms used when SMEs discuss lingering objects. The

terminology is difficult to grasp. How can I understand your action plan if I don’t

know what you’re saying?”

Practice performing the different clean-up procedures.

o Lab materials that support the course (Hands-on experience with analysis and

resolution steps)

To be able to understand the full scope of a lingering object problem in a large environment

o “I understand how to fix one or two DCs, but it’s a little scary when the customer has

hundreds of servers and most of them have problems.”

To be able to understand which method to use

o “There are five or more methods that do the same thing. Which one should I use?”

2.4 Instructor’s Needs

Supporting materials:

o Documentation

o Visual Aids

o Well defined lab materials

o Available Resources

Page 10: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 3.0 Learning Expectations

4 © 2011 Microsoft Corporation. All rights reserved.

3.0 Learning Expectations

3.1 Learning Goals and Objectives

1.0 To understand the cause, identify the symptoms, and identify ways to resolve lingering

object issues

1.1 The learner will be able summarize seven terms commonly used in lingering object

scenarios.

1.2 The learner will be able to explain three ways in which lingering objects are created.

1.3 The learner will be able to list four symptoms of lingering objects.

1.4 The learner will be able to identify the currently configured tombstone lifetime and

replication consistency settings in a lab environment.

2.0 To be able to explain how an Active Directory Administrator can avoid lingering objects in

the future

2.1 The learner will be able to list at least three methods to prevent lingering objects

3.0 To be able to accurately determine the scope of a lingering object problem

3.1 The learner will be able to use repadmin.exe to generate diagnostic data for analysis

3.2 Given diagnostic data, the learner will be able to identify the scope of the problem by

listing all partitions and all servers containing lingering objects.

4.0 To be able to document which method to use to resolve the issue and why

4.1 Given diagnostic data, the learner will be able to create a detailed action plan that will

remove the lingering objects in all partitions on all servers.

4.2 Given five different scenarios, the leaner will be able to recommend the correct

method to remove lingering objects.

4.3 Given a subpar action plan, the learner will be able to recommend changes that will

result in a better solution

5.0 To be able to apply that knowledge in a lab environment and resolve a lingering object

scenario

5.1 The learner will be able to execute the steps in an action plan in order to remove

lingering objects

5.2 The learner will be able to remove lingering objects using five different methods.

3.2 Lesson Components

The course will consist of PowerPoint slides, supporting documentation in Microsoft Word,

and a laboratory environment where the methods and procedures can be practiced on

virtualized domain controllers running on a Windows Server 2008 R2 Hyper-V server. The

Page 11: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 5

course workbook will contain all necessary supporting documentation and will include real-

world examples of actual cases in a "Did you know?" format.

3.3 Resources

The instructor and students will have pre-requisite knowledge of Active Directory

replication troubleshooting

The instructor and students will have a computer running Windows 7 with Microsoft Office

2010 and remote desktop access to a Server running Windows Server 2008 R2 with Hyper-

V.

Hyper-V will contain the required virtualized domain controllers.

The classroom will have a project, screen, and whiteboard

Page 12: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 4.0 Learning Activities

6 © 2011 Microsoft Corporation. All rights reserved.

4.0 Learning Activities

Focus on goals

Each lesson:

Begins with an overview and explanation of the goals of the lesson

Instructor will ask questions to generate curiosity and judge prior knowledge

Connect to prior knowledge

Classroom discussion

Instructor will facilitate discussion of student's prior knowledge

Gain and integrate content knowledge

Case study

Present problems and demonstrate how to solve, explicitly stating the strategies that

were used.

Real-world examples

Present new information in context in which it will be used

Lecture with slides, workbook and hands-on lab

Present information through multiple modes of representation

Allow learners to revisit information as needed

Provide adequate resources

Take action and monitor learning progress

Hands-on lab

Provide support and coaching as needed when learners are performing tasks

Ask learners to demonstrate skill; provide corrective feedback

Synthesize and evaluation

Short-answer, matching, multiple choice, and free recall format exam

Posttest on knowledge

Performance based assessment

Have leaners demonstrate procedure or skill

Page 13: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 7

Have learners demonstrate their own summaries

Case study

Present case studies, role lays, or simulations in which learners demonstrate skills,

knowledge, attitudes

Extend and transfer

Hands-on labs

Provide practice in a variety of situations

Gradually remove prompts and cues

Provide opportunity to apply skills in realistic contexts

Workbook and Quick-reference handouts "Cube-note"

Provide job aids

Provide access to additional information on the topic

Page 14: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 5.0 Assessment

8 © 2011 Microsoft Corporation. All rights reserved.

5.0 Assessment There are two different assessments: One is accessible via an Intranet web page and consists

of a short-answer, matching, multiple choice, and free recall format exam. The other

assessment is a performance-based lab assessment where the student is presented with a

common lingering object scenario and has to document the issue, action plan and perform the

procedure to correctly remove the lingering objects.

5.1 Assessment Objectives

1.1 The learner will be able identify seven terms commonly used in lingering object

scenarios and match them to the corresponding definition. (exam)

1.2 The learner will be able to explain three ways in which lingering objects are created.

(Performance assessment)

1.3 The learner will be able to identify four symptoms of lingering objects. (exam)

1.4 The learner will be able to identify the currently configured tombstone lifetime and

replication consistency settings in a lab environment. (Performance assessment)

2.0 To be able to explain how an Active Directory Administrator can avoid lingering objects in

the future

2.1 The learner will be able to list at least three methods to prevent lingering objects

(exam)

3.0 To be able to accurately determine the scope of a lingering object problem

3.1 The learner will be able to use repadmin.exe to generate diagnostic data for analysis

(exam and Performance assessment)

3.2 Given diagnostic data, the learner will be able to identify the scope of the problem by

listing all partitions and all servers containing lingering objects. (exam and Performance

assessment)

4.0 To be able to document which method to use to resolve the issue and why

4.1 Given diagnostic data, the learner will be able to create a detailed action plan that will

remove the lingering objects in all partitions on all servers. (exam and Performance

assessment)

4.2 Given five different scenarios, the leaner will be able to recommend the correct

method to remove lingering objects. (exam and performance assessment)

4.3 Given a subpar action plan, the learner will be able to recommend changes that will

result in a better solution (performance assessment)

5.0 To be able to apply that knowledge in a lab environment and resolve a lingering object

scenario

Page 15: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 9

5.1 The learner will be able to execute the steps in an action plan in order to remove

lingering objects (performance assessment)

5.2 The learner will be able to remove lingering objects using five different methods.

(Performance assessment)

5.2 Post-course exam

Multiple Choice: For each of the following questions, circle the letter of the answer that best

answers the question. (5 points each)

1. Which of the following commands would generate a forest-wide replication status report to be used to aid in lingering object analysis? [Objective 3.1]

A. repadmin /replsum /xls >repl.xls B. repadmin /replsum /verbose >repl.xml C. repadmin /showrepl * /csv >repl.csv D. repadmin /showrepl /verbose >repl.txt E. ldp | removelingeringobjects F. A and D G. All of the above

2. Which of the following lingering object removal methods automates the removal of lingering objects?

[Objective 4.2]

A. repadmin /unhost B. repadmin /removelingeringobjects C. repadmin /rehost D. repldiag /removelingeringobjects E. ldp removelingeringobjects primitive F. replfix G. None of the above

3. Which of the following lingering object removal methods will remove objects on Windows 2000 -

Windows 2008 R2 and will remove abandoned objects? [Objective 4.2]

A. repadmin /unhost B. repadmin /removelingeringobjects C. repadmin /rehost D. repldiag /removelingeringobjects E. ldp removelingeringobjects primitive F. replfix G. None of the above

4. Which of the following lingering object removal methods allow you to review which objects will be removed prior to actually removing the objects? [Objective 4.2]

A. repadmin /unhost B. repadmin /removelingeringobjects C. repadmin /rehost D. repldiag /removelingeringobjects E. ldp removelingeringobjects primitive F. replfix

Page 16: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 5.0 Assessment

10 © 2011 Microsoft Corporation. All rights reserved.

G. B and F H. D and E

True or False: For each statement, circle True or False. (2 points each) [Objective 1.3]

True False 5. Replication status 8606 indicates that lingering objects are present on the source DC in a replication report.

True False 6. Event ID 1988 indicates that the source DC contains one or more lingering objects.

True False 7. Replication status 8453 indicates that lingering objects are present on the destination DC.

True False 8. Event ID 1388 indicates a lingering object was purged from the database.

True False 9. Event ID 1945 indicates that a lingering object was detected after running repadmin /removelingeringobjects.

True False 10. Abandoned objects can be removed using repadmin /removelingeringobjects.

Fill in the Blank and Matching: Into each sentence below, copy a term from the word bank that

correctly completes the sentence. (5 points each) [objective 1.1]

Lingering Links Lingering Object Tombstone

Abandoned Object Loose Replication

Consistency

Tombstone

Lifetime

Abandoned Delete Strict Replication

Consistency

11. The length of time that a deleted object will remain in the database is referred to as _______.

12. A _________ is an object that is present on one replica, but has been deleted and garbage collected

on another replica.

13. A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale

references are referred to as ___________.

14. An object that has been deleted but not yet garbage collected. _________

15. An object created on one DC that never got replicated to other DCs hosting a writable copy of the NC

but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes

offline prior to replicating the originating write to other DCs that contain a writable copy of the

partition. _________

16. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it

does not have, the entire object is replicated to the target for the sake of replication consistency. This

undesirable behavior causes a lingering object to be “reanimated.” _________

17. An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC

for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that

originated the object deletion goes offline prior to replicating the change to other DCs hosting a

writable copy of the partition. ____________

Page 17: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 11

18. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it

does not have, replication is blocked with the source DC for the partition where the lingering object

was detected. __________

19. Essay Question: List three or more methods to prevent lingering objects (8 points) (objective 2.1)

Use Figure 1 Replication Status to answer the remaining questions.

Figure 1 Replication Status

20. Essay Question: Use Figure 1 Replication Status, document every DC containing lingering objects

and for which partition. (10 points) (objective 3.2)

Page 18: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 5.0 Assessment

12 © 2011 Microsoft Corporation. All rights reserved.

21. Essay Question: Using Figure 1 Replication Status and the following information, provide the exact

command line syntax to log all lingering objects on DC 5thWardCorpDC to the event log, and the

syntax to remove those lingering objects. (10 points) (objective 4.1)

Repadmin /removelingeringobjects <Dest_DSA_LIST> <Source DSA GUID> <NC> [/ADVISORY_MODE] The following DCs host writable copies of the partition in question: Dallas\DALCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 87ccb4f8-1057-4cfa-aed6-79b5626db9fd DC invocationID: 56f7cb84-0a67-43c1-93de-9d01f53e02c5 Dallas\NYCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 4009aef6-b279-43d2-82f6-4298f02505e8 DC invocationID: a29c83ab-5dea-4829-bbbf-1343f037098d Liverpool\LONCONTOSODC DC Options: IS_GC Site Options: (none) DC object GUID: a29bbfda-8425-4cb9-9c66-8e07d505a5c6 DC invocationID: d58a6322-6a28-4708-82d3-53b7dcc13c1a Liverpool\LONEMEADC DC Options: IS_GC Site Options: (none) DC object GUID: ba9bcfb2-7445-2cd9-8c66-9b27d534a4b3 DC invocationID: e38b6355-fb31-3785-71b1-42c6ddc23f8e Houston\5THWARDCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 DC invocationID: e0cb69c0-5d24-4254-b830-99b0c9b4da1f

Page 19: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 13

5.3 Post-course exam Answer Key

1. C

2. D

3. E

4. G

5. True

6. True

7. False

8. False

9. True

10. False

11. Tombstone Lifetime

12. Lingering Object

13. Lingering Link

14. Tombstone

15. Abandoned Object

16. Loose Replication Consistency

17. Abandoned Delete

18. Strict Replication Consistency

19. At least 3 of the following:

o Resolve replication failures within TSL

o Ensure Strict Replication Consistency is enabled

o Ensure large jumps in system time are blocked via registry key or policy

o Don't remove replication quarantine with "allowDivergent" setting without

removing LOs first

o Don't restore system backups that are near TSL number of days old

o Don't bring DCs back online that haven't replicated within TSL

20. LONCONTOSODC: DomainDNSZones, Configuration

5THWARDCORPDC: Configuration

DALCORPDC: Configuration

FOURTHDC1: Configuration, ForestDNSZones

NYCORPDC: Configuration

CONTOSOROOTDC1: Configuration

FOURTHDC2: Configuration

21.

Repadmin /removelingeringobjects 5thwardCorpDC ba9bcfb2-7445-2cd9-8c66-9b27d534a4b3 cn=configuration,dc=contoso,dc=com /advisory_mode

Page 20: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 5.0 Assessment

14 © 2011 Microsoft Corporation. All rights reserved.

Repadmin /removelingeringobjects 5thwardCorpDC ba9bcfb2-7445-2cd9-8c66-

9b27d534a4b3 cn=configuration,dc=contoso,dc=com

5.4 Performance Assessment

Students take their performance assessment in the hands-on lab environment. The

performance assessment is a culmination of all prior lab tasks without the benefit of step-by-

step guidance. The lab environment is broken via several scripts. After the scripts run, both

lingering objects and abandoned objects are present. The students receive a handout with

intentionally vague problem descriptions. They are instructed to document the issue

thoroughly and then resolve the problems. Good documentation consists of symptoms,

cause, and resolution. The symptoms section should contain a list of all "problematic

objects." The resolution section should have a thoroughly documented action plan. Here is

the text they are prompted with:

You are the consultant for Adatum Corporation. Please help resolve the following

problems in our environment.

Changes are not propagated amongst DCs for the Adatum domain.

Unable to create the following user account in the West domain: Mike Miller

Ann Wallace's account in the East domain does not show up on any other domain's

GC

Users that send email to the CorpVP mail-enabled universal group receive NDRs on

occasion. Additionally, our Exchange 2010 mailbox server cannot generate an

Offline Address Book. This worked on our Exchange 2007 mailbox server.

Please ensure that you document each problem thoroughly. This documentation should

include forest and DC environment settings (tombstone lifetime and replication

consistency), symptom, cause and resolution sections. The symptoms section should

contain a list of all "problematic objects." The resolution section should have a thoroughly

documented action plan. Implement your action plan after documenting the issue.

Page 21: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 15

5.5 Performance Assessment Rubric

Hands-on Assessment Rubric: Troubleshooting Lingering Objects

Student Name: _____________________________________

Assessment Criteria Max.

Points Exceptional (all points) Average (65-85%) Poor (0 - 65%) Comments &

Points Earned

Documentation (objectives 1.4, 3.1, 3.2, 4.1, 4.2)

10 Symptom, cause and resolution sections

The symptoms section contains a list of all objects

The resolution section has a thoroughly documented action plan

Symptom, cause and resolution sections are mostly documented

The symptoms section contains a partial list of all objects

The action plan is missing one to two steps

Symptom, cause and resolution sections is inadequate

Less than 25% of all objects are listed

The action plan will not resolve the issue or will make things worse

AD Replication and Lingering object cleanup (5.1, 5.2)

25 All lingering objects are removed from the environment

AD Replication is successful

Most (greater than 75%) of lingering objects are removed.

Less than 25% of lingering objects are removed

AD Replication is not successful

Abandoned object cleanup (5.1, 5.2)

25 Abandoned object is no longer present on any DC

new object is created in its place

Abandoned object is no longer present on most DCs (greater than 75%)

Abandoned object is still present on most DCs

Abandoned delete resolution (5.1, 5.2)

25 Object completely removed from the environment

Object mostly removed from the environment

Object is still present on most DCs in the environment

Lingering Link cleanup (5.1, 5.2)

15 CorpVP group contains correct group membership on all DCs

Group still has the same objectSID

CorpVP group contains correct group membership on all DCs

Group does not have the same ObjectSID

CorpVP group has inconsistent group membership on most DCs

TOTAL: 100

Page 22: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 6.0 Evaluation

16 © 2011 Microsoft Corporation. All rights reserved.

6.0 Evaluation Following the conclusion of the course, the students are emailed a link to a survey to take

online.

6.1 Survey Questions

# Question

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

1 I was provided with the information I needed

(logistics, pre-work) for the training in a

timely manner.

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

2 The classroom setup and hardware (if

supplied) functioned appropriately to

support face-to-face learning.

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

3 The instructor was knowledgeable about the

subject matter.

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

4 The instructor's presentation skills helped

me better understand the content.

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

5 The instructor consistently linked the course

content to Microsoft’s business and/or my

role.

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

6 The length of the course was appropriate Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

7 Overall, I was satisfied with this course. Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

8 This course builds skills improving how I sell,

market, and/or provide services to our

customers and partners.

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

9 This course was a valuable use of my time. Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

Page 23: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 17

10 I would recommend this course. Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

11 The messaging in this course is relevant to

Microsoft's customers and/or partners.

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

12 If not, please provide additional feedback.

13 How soon will you be able to apply this

learning?

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

14 My manager and I have discussed how I will

apply this training to my job.

Strongly

Agree

Agree Neither

Agree

nor

Disagree

Disagree Strongly

Disagree

Don't

Know

15 What are you going to do differently as a

result of this course?

16 What was the most useful portion of this

course? (Please provide specifics, e.g.

instructor effectiveness, content quality,

materials usefulness).

17 What was the least useful portion of this

course? (Please provide specifics, e.g.

instructor effectiveness, content quality,

materials usefulness).

18 Please provide any additional comments (e.g. learning environment, instructor effectiveness, content/materials quality, content level, relevance, application).

Page 24: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 7.0 Timeline

18 © 2011 Microsoft Corporation. All rights reserved.

7.0 Timeline The following proposed timeline should allow for sufficient coverage of the course material.

Time Objectives Activities / Training Methods

Materials

9:00 AM

15 minutes

Welcome and Instructor Introduction

1.2

Intro and Classroom discussion

Slide 1: Course Title and

Instructor Name

9:15 AM

20 minutes

Lingering Object Fundamentals

1.1

Lecture and discussion

Lesson 1 Slides

9:35 AM

15 minutes

Exploring Lingering Object Fundamentals

1.4

Lab 1exercise Lab 1 guide and lab environment

9:50 AM

20 minutes

Symptoms and Cause

1.2, 1.3, 2.1

Lecture and discussion

Lesson 2 Slides

Provide real-word scenarios

10:10 AM

20 minutes

Identification and Classification

3.1, 3.2

Lecture and discussion

Lesson 3 Slides

Show prior case action plans

10:30 AM

10 minutes

Break

10:40 AM

45 minutes

Lingering Object Diagnosis and Documentation

3.1, 3.2, 4.1

Lab 2 exercise Lab 2 guide and lab environment

11:25 AM

20 minutes

Lingering Object Removal

5.1, 5.2

Lecture and discussion

Lesson 4 Slides

11:45 AM

60 minutes

Lunch

1:00 PM

90 minutes

Lingering Object removal labs

5.1, 5.2

Lab exercise 4 - 6

LabLab documentation, Hyper-V images

Page 25: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 19

2:30 PM

10 minutes

Break

2:40 PM

10 minutes

Real World Application

4.2, 4.3

Lecture and discussion

Lesson 5 Slides

2:50 PM

30 Minutes

Real-world case study

4.2, 4.3

Case Study Case data in instructor share

Case Details, Diagnostic Data

Present the high-level symptoms.

What data do you want to see?

Show the data

What is the action plan?

3:20 PM

10 minutes

Question Time Ask if there are any questions

3:30 PM

30 minutes

Assessment Post-course test Share assessment URL on-screen

4:00 PM

10 minutes

Break

4:10 PM

60 minutes

Performance assessment

Lab-based assessment

VMAS connection instructions for post-course performance assesment

10 minutes Summary and questions

1.1 - 5.2

Course Summary and wrap-up

Slide

Page 26: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 8.0 Job Aid

20 © 2011 Microsoft Corporation. All rights reserved.

8.0 Job Aid

Page 27: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 21

8.1 Instructor Job Aid

Course Parameters

Course title Troubleshooting Lingering Objects

Course Length 6 hours (1 day)

Course Objectives At the completion of this workshop, the engineer shall be able to:

1. Explain how a user's group membership is stored in Active Directory

2. Explain what happens during a user deletion

3. Understand why a special procedure is needed to restore users along with

their group membership.

4. Explain the three methods of recovery after deletion

5. Identify recommendations and considerations for a better recovery

experience

6. Perform the most common (and preferred) method of recovery for our

customers

Target Audience Microsoft Support Engineers (Platforms, Directory Services)

Prerequisites Trainee:

1. Knowledge of Active Directory replication

2. Familiarity with Active Directory concepts and terminology

3. Experience with Hyper-v for the lab session

Instructor:

1. Real world experience with Active Directory replication and Lingering

Object troubleshooting procedures

2. Hyper-V user experience for demonstration session

3. PowerPoint user experience

Room arrangement Classroom setting

Materials/equipment PowerPoint setup, whiteboard and markers, Computer for

demonstration, and one computer for each workshop participant.

On the computers: Microsoft Windows 7, Microsoft Office 2010,

Intranet access, PowerPoint presentation, and supporting reference

documentation

Evaluation/

Assignments

Learning exercises for participants and online Instructor/classroom

evaluation form

Instructor Justin Turner is a Sr. Support Escalation Engineer on the Microsoft

Platforms Directory Services Support team where he obtained his first-

hand knowledge of the material. He has been with Microsoft for over

ten years, and is currently pursuing his MS in Computed Education and

Cognitive Systems degree from the University of North Texas.

Page 28: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid

22 © 2011 Microsoft Corporation. All rights reserved.

Note to Trainers

Checklist of Supplies

Print out slides with “notes pages.” The notes pages provide the necessary material to help explain

the contents of each slide.

Alternatively, you can have the students copy the course materials to their computer and print out

the slides to a new Microsoft OneNote notebook.

The student lab guide is stored electronically on the hyper-v image: DC1

Room Arrangement

Standard Microsoft classroom configuration: Classroom style with whiteboard and projector screen

at the front of the room

Handouts / Visual Aids

Print out one copy of the slide deck in "Handouts" format for each student (or print to OneNote).

Course workbook and lab guide are available on the Instructor computer.

Lab Computer setup

Microsoft Windows 7

Office 2010

Connection to the corporate Intranet

Preparation

Before Class starts:

1. Have PowerPoint slide deck opened up

2. On instructor machine: Launch Hyper-V, and launch DC1s image

3. Ensure classroom has intranet connectivity

Page 29: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 23

Obtaining Access to Virtual Machines

To access VMs provisioned for your use during this course, perform the following steps:

4. Log onto the physical computer using your Corpnet credentials.

5. Access the VMAS server that hosts your VMs using the link provided by your instructor.

6. Open the VMAS menu and select Manage VMAS VMs.

7. Use Manage My VMs to access virtual machines referenced in lab exercises.

Note: For more information, click links in the Documents section on the right to open course documents included in the VM package.

Page 30: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid

24 © 2011 Microsoft Corporation. All rights reserved.

Activities

Introduction

Welcome the students to the course. Ask them to share the following:

Name

Role

Time at Microsoft

Something that no one (at work) else knows about them or something unique

Classroom Discussion

After the introduction, lead a discussion to gauge student's prior knowledge. Ask probing

questions like:

What is a lingering object?

Why do I care about removing them from my environment?

What does tombstone lifetime have to do with this?

Who can explain the different between strict and loose replication consistency?

What is an abandoned object? How is that different from a lingering object?

What is a lingering linked value?

Who here has worked a lingering object issue? Were you able to resolve it? How long

did it take?

Who here has used repldiag? What did you think about it?

Real-world examples

Where appropriate, provide examples of actual cases worked. Highlight the successes and

failures (what went right and what went wrong).

Present new information in context in which it will be used

Case Study

The case study within the course includes real diagnostics data from an actual customer case.

The data was scrubbed to remove personally identifiable information (PII). Present the facts

of the case and encourage the students to play the role of engineer. There is an action plan

included in the case study. The action plan is intentionally poor in quality and if implemented

would result in disastrous results. Together come up with the appropriate action plan to

resolve the problem.

Page 31: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 25

Present case studies, role plays, or simulations in which learners demonstrate skills,

knowledge, attitudes

Present problems and demonstrate how to solve, explicitly stating the strategies that

were used.

Lab Activities

Students have access to their lab environment through the VMAS site. Each lab activity

corresponds to a lesson in the course. You may be tempted to do the entire lecture at once

and then all lab activities at the end of the course. It is important not to do this. Please have

the students complete the lab activities along with the appropriate lesson in the course.

Hands-on lab

If unfamiliar with the lab environment and lab material, you should work through each

lab activity at least one time prior to the course

Provide support and coaching as needed when learners are performing tasks

Ask learners to demonstrate skill; provide corrective feedback

8.2 Learner Job Aid

Lingering Object Terminology

Table 1: Lingering Object Terminology

Term Definition

Abandoned delete An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.

Abandoned object

An object created on one DC that never got replicated to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition.

Lingering link

A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale references are referred to as lingering links.

Page 32: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid

26 © 2011 Microsoft Corporation. All rights reserved.

Lingering Object

An object that is present on one replica, but has been deleted and garbage collected on another replica.

Loose Replication Consistency With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This undesirable behavior causes a lingering object to be “reanimated.”

Strict Replication Consistency With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, replication is blocked

with the source DC for the partition where the lingering object was detected

Tombstone An object that has been deleted but not yet garbage collected

Tombstone Lifetime (TSL)

The amount of time tombstones are retained in Active Directory before being garbage collected and permanently purged from the database.

Tombstone Lifetime Default Values

Table 2: Default TSL Values

OS Install Path Default TSL

Windows 2000 RTM 60 days

Windows 2003 RTM, 2003 R2 60 days

Windows 2000RTM upgrade to Windows 2003 SP1 60 days

Windows 2003SP1, 2003SP2, 2008, 2008R2 180 days

NT4 upgrade to Windows 2003 SP1 180 days

Replication Consistency Settings

Strict Replication Consistency

Defines how a destination DC behaves if a source DC sends updates to an object that does

not exist in the destination DC’s local copy of Active Directory.

Page 33: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 27

o Destination DCs should see USN for creates before object is modified

o Only modifies for lingering objects arrive for object not on destination DC

o Only destination DC’s enforce strict replication and log events

Destination DCs stop replicating from source DC’s partitions containing LO’s

Lingering objects are quarantined on source DCs where they can be detected

End-to-end replication may be impacted for partitions containing lingering objects

Administrators must remove lingering objects to restore replication

Enabling Strict Replication

Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command

prompt:

For all domain controllers, type:

repadmin /regkey * +strict

For all global catalog servers, type:

repadmin /regkey gc: +strict

You can also enable strict replication by manually setting the Strict Replication Consistency

registry value to 1.

HKLM\System\CurrentControlSet\Services\NTDS\Parameter Strict Replication Consistency

(Reg_DWORD) to 1

1 (enabled): Inbound replication of the specified directory partition from the source is

stopped on the destination.

Warning: Ensure you are prepared to deal with replication failures after enabling strict replication consistency due to the existence of lingering objects.

Loose Replication Consistency

If you enable Loose Replication Consistency, if a destination receives a change to an object

that it does not have, the entire object is replicated to the target for the sake of replication

consistency. This behavior causes a lingering object to be reapplied to all domain controllers

in the replication topology.

Enable Loose Replication

Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command

prompt:

For all domain controllers, type:

repadmin /regkey * -strict

Page 34: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid

28 © 2011 Microsoft Corporation. All rights reserved.

For all global catalog servers, type:

repadmin /regkey gc: -strict

You can also enable strict replication by manually setting the Strict Replication Consistency

registry value to 0.

HKLM\System\CurrentControlSet\Services\NTDS\Parameters Value: Strict Replication Consistency Type: (Reg_DWORD) Value Data: 0 0 (disabled): The destination requests the full object from the source domain controller, and the lingering object is revived in the directory as a new object.

Critical: The Loose Replication Consistency setting will cause the undesirable behavior of reanimation of lingering objects.

Default Settings for Strict Replication Consistency

Upgrade Path Default Notes

Windows NT 4.0 Loose

Windows 2000 RTM Root Loose A post-SP2 NTDSA.DLL defaulted to strict replication consistency but was quickly recalled. Windows 2000 Services 1 through 4 all default to loose replication consistency.

Windows NT 4.0 to Windows 2000 Root

Loose

Windows 2000 to Windows Server 2003 SP1

Loose Upgrading a Windows 2000 forest to Windows Server 2003 slipstreamed with SP1 does not enabled strict replication consistency.

Windows Server 2003 RTM Root

Strict DCPROMO creates an operational GUID that causes Windows Server 2003 domain controllers to inherit strict replication mode but is ignored by Windows 2000 domain controllers.

Windows Server 2003 SP1 root

Strict Same as above.

Windows NT 4.0 to Windows Server 2003 root

Strict DCPROMO creates an operational GUID that causes

Page 35: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 29

Windows Server 2003 domain controllers to inherit strict replication mode but is ignored by Windows 2000 domain controllers.

The default value for the strict replication consistency registry entry is determined by the

conditions under which the domain controller was installed into the forest.

Note: Raising the domain or forest functional level does not change the replication

consistency setting on any domain controller.

More Information: For more information about this topic, see:

http://blogs.technet.com/b/askds/archive/2010/02/15/strict-replication-consistency-myth-versus-reality.aspx

Troubleshooting Overview

Common methods to remove lingering objects include: Repadmin /Removelingeringobjects Replfix Repldiag Manually through LDP or using script Rehost the partition:

Repadmin /rehost (or /unhost and /add) (only if the partition is not-writable on the DC containing lingering objects)

Un-GC (but you don’t really have control over who the DCs sources the partition from) Demote and Promote (DCPromo)

Repadmin /removelingeringobjects Quick Reference

Have the customer run the following command: repadmin /showrepl * /csv >showrepl.csv Once you have this, filter column K for 8606, so that you know exactly which DCs have lingering objects and in which partitions. The DCs in the SourceDC column contain lingering objects. You can use the repadmin /removelingeringobjects command to remove lingering objects. In some cases it may make sense to just rehost the partition with the repadmin /rehost command. In order to use the /removelingeringobjects command you need to know three things:

1. You need to know which dc's contain lingering objects 2. Which partition the lingering object resides in 3. A good reference DC that hosts that partition that does not contain lingering objects

Page 36: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid

30 © 2011 Microsoft Corporation. All rights reserved.

Repadmin RLO example usage:

The command is: repadmin /removelingeringobjects LingeringDC ReferenceDC_DSA_GUID Partition Where: LingeringDC: FQDN of DC that has the lingering objects ReferenceDC_DSA_GUID: The DSA GUID of a domain controller that hosts a writeable copy of the partition Partition: The distinguished name of the directory partition where the lingering objects exist So for example: We have a server named DC1.contoso.com that contains lingering objects. We know that the lingering object is in the childdomain.contoso.com partition. We know that DC3.childdomain.contoso.com hosts a writeable copy of the partition and doesn't contain any lingering objects. We need to find the DSA GUID of DC3 is, so we run: repadmin /showrepl DC3.childdomain.contoso.com At the top of the output, locate the DC Object GUID entry. This is the GUID you need to enter in the command for the reference DC. The command would be repadmin /removelingeringobjects DC1.contoso.com 5ed02b33-a6ab-4576-b109-bb688221e6e3 dc=childdomain,dc=contoso,dc=com

-------------------------------------------------------------------------------------------------

Detailed troubleshooting guidance is located here:

2028495 Troubleshooting Active Directory operations that fail with error 8606: Insufficient attributes were given to create an object. http://support.microsoft.com/default.aspx?scid=kb;en-US;2028495

Un-hosting a partition

It is sometimes necessary to remove a partition from the database of a DC temporarily.

Repadmin includes a /rehost option that allows you to do this, but the /unhost option allows

you to exercise more control over the procedure. Take note that /unhost only allows you to

remove a read-only copy of the partition. With the exception of application partitions, you

cannot remove a writable copy of a partition from a DC without using DCPROMO.

Repadmin /?:unhost Remove a specific read-only partition from a GC. [SYNTAX] /unhost DSA <Naming Context> Repadmin /unhost ContosoDC1 dc=corp,dc=contoso,dc=com

Event ID 1659 indicates the status of the un-host operation. Do not re-add the partition until

event ID 1660 is logged in the Directory Services event log. The re-host operation may fail

with error 8339 if you attempt to re-add the partition too soon after the un-host.

Page 37: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 31

Manually adding a replication connection using

repadmin.exe

The add command will create a RepsFrom attribute on the destination domain

controller for the specified naming context and initiate a replication request. During a

normal replication cycle, the destination domain controller will request updates from

the source domain controller. When creating temporary replication links between

replication partners, the process could fail if the KCC starts while you are performing

the procedure. The KCC will delete any replication links for which no corresponding

connection object exists. Since these commands can take a very long time to

complete as they trigger the replication of the corresponding naming context, it is

important to ensure that KCC do not disturb the process. This is where you would use

+DISABLE_NTDSCONN_XLATE which effectively disables KCC's capability to translate

connection objects to replication links.

Disable KCC connection translation so that KCC doesn’t remove our temporary

replication connection:

Repadmin /options ContosoDC1 +disable_ntdsconn_xlate

Then add a replication connection for the configuration partition of the server we

want to source the partition from:

Repadmin /add <Naming Context> <Dest DSA> <Source DSA> [/readonly] [/selsecrets] <Source DSA> The source DSA must be specified by fully qualified computername. repadmin /add cn=configuration,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.

Add a replication connection to the server for the domain partition that we

need to source from (/readonly is specified if the partition is a GC non-writable

partition /selsecrets needs to be specified if the destination DC is an RODC):

repadmin /add dc=emea,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com /readonly One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.

Page 38: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid

32 © 2011 Microsoft Corporation. All rights reserved.

If you need to replicate the other way, then just reverse the order of the server names

in the commands.

To begin a normal sync of the partition using the new replication connection:

Repadmin /replicate <Dest_DSA_LIST> <Source DSA_NAME> <Naming Context> [/force] [/async] [/full] [/addref] [/readonly] repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly

To begin a full sync of that partition using the new replication connection:

repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly /full Sync from LONEMEADC.Emea.contoso.com to ContosoDC1.contoso.com completed successfully.

Turn KCC connection translation back on when you no longer need the connection:

Repadmin /options ContosoDC1 -disable_ntdsconn_xlate

Repldiag quick reference

Removing lingering objects from a forest with repldiag is as simple as running repldiag

/removelingeringobjects. However, it is usually best to exercise some control over the

process in larger environments. The option /OverRideReferenceDC allows you to select

which DC is used for cleanup. The option /outputrepadmincommandlinesyntax allows you to

see what a forest-wide cleanup looks like using repadmin.

Repldiag /removelingeringobjects /outputrepadmincommandlinesyntax

This will give you output of corresponding repadmin /removelingeringobjects syntax. It will first select one DC per partition to be used as a reference DC. It will then clean the reference DCs up against all other DCs for the partition(s) it was selected to be used as a reference for. Finally it cleans up all other DCs in the forest with the new “cleaned up” reference DCs as sources. The /outputrepadmincommandlinesyntax option does not actually attempt object cleanup. You would need to leave this option off if you want to execute lingering object cleanup.

Number Complete,Status,Server Name,Naming Context,Reference DC,Duration,Error Code,Error Message repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=forestdnszones,dc=contoso,dc=com

Page 39: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 33

repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=corp,dc=contoso,dc=com Reference NCs cleaned in 0h:0m:0s. Cleaning everything else against reference NCs. repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com

Page 40: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid

34 © 2011 Microsoft Corporation. All rights reserved.

repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com All NCs cleaned in 0h:0m:0s.

This output can also be viewed in Excel: Copy commands to a text file. Modify the text file to include only the command portion of the output. Then open up the text file in Exel. (space delimited)

More control: /OverRideReferenceDC

This option allows you to specify a DC that you want to be used as a reference DC for the partition specified. In a large distributed environment, take careful consideration when choosing the reference DC. Things to consider when choosing a suitable reference DC: Well connected: Fast WAN link. Performance: Excellent server class hardware: Disk, RAM, CPU and NIC Critical Network Applications / Services do not depend on this DC: Such as an Exchange

facing DC Other DCs don’t report replication failures with reference DC as the source: filter repadmin

/showrepl * /csv ouput, or use the topology report created by repldiag /save.

repldiag /removelingeringobjects /overridedefaultreferencedc:"cn=configuration,dc=contoso,dc=com":nycorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=corp,dc=contoso,dc=com":seacorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=forestdnszones,dc=contoso,dc=com":5thwardcorpdc.corp.contoso.com /outputrepadmincommandlinesyntax Replication topology analyzer. Written by [email protected] Version: 2.0.3397.24022 Command Line Switch: /removelingeringobjects Command Line Switch: /overridedefaultreferencedc:cn=configuration,dc=contoso,dc=com:nycorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=corp,dc=contoso,dc=com:seacorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=forestdnszones,dc=contoso,dc=com:5thwardcorpdc.corp.contoso.com Command Line Switch: /outputrepadmincommandlinesyntax Attempting to override NC cn=configuration,dc=contoso,dc=com with DC nycorpdc.corp.contoso.com... Overriden Attempting to override NC dc=corp,dc=contoso,dc=com with DC seacorpdc.corp.contoso.com... Overriden Attempting to override NC dc=forestdnszones,dc=contoso,dc=com with DC 5thwardcorpdc.corp.contoso.com... Overriden

/UseRobustDCLocation

Page 41: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 35

Query each and every DC for a list of DCs in forest. Ensures replication instability does not cause any to be missed. We’ve had cases where we clean up lingering objects in the forest but do to an AD topology problem some DCs were not cleaned up. This option is almost always recommended if you want it to do a thorough job.

Page 42: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 9.0 Course Workbook

36 © 2011 Microsoft Corporation. All rights reserved.

9.0 Course Workbook

Document Conventions The following conventions are used in the course materials:

Acronyms appear in all uppercase letters.

Path and file names may appear in a combination of uppercase and lowercase letters.

Unless otherwise indicated, paths and file names entered in dialog boxes or at a

command prompt are not case-senstitive

File extensions without a file name appear in all lower-case letters.

Book titles and URLs appear in Italic.

Window, dialog box, menu titles, menu items, and section titles appear in Bold.

Other document conventions are described below.

Program Code and Commands

Program code listings, diagnostic output, entries typed at a command prompt or in scripts or

initialization files, and other text mode content appear in a console font with a grey

background formatted as shown in the following example. Descriptive comments may be

inserted in line with the listing.

d:\%systemroot%>dir /ad

where:

d: is the drive letter where the operating system is installed.

%systemroot% is the folder where the operating system is installed.

Volume in drive C is Main Volume Serial Number is 000A-BCDE Directory of C:\Windows 12/19/2004 11:56 AM <DIR> . 12/19/2004 11:56 AM <DIR> .. 07/07/2003 06:57 AM <DIR> addins 11/17/2004 02:45 PM <DIR> Application Compatibility Scripts 11/17/2004 02:47 PM <DIR> AppPatch 11/17/2004 02:42 PM <DIR> Cache ...

The ellipsis (...) on the last line indicates a partial listing.

The following conventions apply to all commands and program code listings:

Type command statement elements that appear in Bold exactly as they appear in the

example, including quotation marks.

Italic elements in command statements indicate placeholders for variable information.

Page 43: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 37

Braces ({ }) enclose required items as shown by {parameter1, parameter2, “title”} in the

example. Commas separate multiple items. Type quotation marks as shown; do not

type the braces.

Square brackets ([ ]) enclose optional items as shown by [option1 | option2] in the

example. Pipe symbols ( | ) indicate alternate choices. If multiple options are listed,

only type one option. Do not type the brackets or pipe symbols.

Notes

Icons and labels call attention to informational notes and reader alerts as shown in the

following table.

Table 3. Note Icons and Labels

Icon Label Description

Note/Important Emphasizes content and provides additional information.

Important Strongly emphasizes key content.

Tip Highlights a best practice.

Critical Indicates strongly recommended actions.

Warning Indicates strongly recommended actions required to prevent

data loss or other undesirable results.

Do Not Warns against actions that may cause system failure or data loss.

More Information Link to reference material.

More Help Link to guides, white papers, or KB articles.

Trends Indicates industry trends, top support issue trends, etc.

Tables and Figures

Each table and figure is preceded by Caption. Captions are numbered sequentially

throughout each module.

Course Document and Slide Numbering

Modules may be numbered sequentially within a course. Lessons, demonstrations, and videos

may be numbered sequentially within a module. Topic and subtopic headings are not

numbered. Lab sessions may be numbered sequentially throughout the course. Individual

exercises are numbered sequentially within each lab session.

Page 44: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 9.0 Course Workbook

38 © 2011 Microsoft Corporation. All rights reserved.

In each module, slide number paragraphs shown in the following figure identify the

presentation slide that accompanies the topic.

Figure 2. Slide Number Paragraph

Slide ##

The first slide in each presentation is unnumbered. Subsequent slides and slide indicator

paragraphs in each module are numbered sequentially starting with 1.

Note: Each presentation slide corresponds to a topic section in the module. Topic sections that include supplemental information may not be referenced on corresponding presentation slides.

Page 45: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 39

Lesson 1: Lingering Objects Fundamentals There is a lot of technical jargon associated with Lingering Object issues that you will need to

understand. The following section provides a definition for each term with context to enable

you to speak confidently when dealing with lingering object issues.

What You Will Learn

After completing this lesson, you will be able to:

Summarize seven terms commonly used in lingering object scenarios.

Terminology associated with Lingering Object issues

Lingering Objects

A lingering object is an object that is present on one replica, but has been deleted and garbage

collected on another replica.

Tombstone

When Active Directory deletes an object from the directory, it does not physically remove the

object from the database. Instead, Active Directory marks the object as deleted by setting the

object's isDeleted attribute to TRUE, stripping most of the attributes from the object,

renaming the object, and then moving * the object to a special container in the object's

naming context (NC) named CN=Deleted Objects. The object, now called a tombstone, is

invisible to normal directory operations.

Note: Some objects don’t get moved upon deletion and will therefore not be moved into the Deleted Objects container.

Tombstone Lifetime (TSL)

When an object is deleted, Active Directory replicates the deletion as a tombstone object. By

inbound-replicating this object, other domain controllers in the domain and forest become

aware of the deletion. The tombstone is retained in Active Directory for a specified period

called the tombstone lifetime. At the end of the tombstone lifetime, the tombstone is deleted

from the directory permanently.

More Help: For more help on this topic, see:

Determine the tombstone lifetime for the forest http://technet.microsoft.com/en-us/library/cc784932(WS.10).aspx

Page 46: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 1: Lingering Objects Fundamentals

40 © 2011 Microsoft Corporation. All rights reserved.

In most cases, the default value is 60 days. If the forest was built on 2008 or later, it should

be 180. The minimum setting is 2 days.

Do Not: Do not reduce TSL to 2 days. (Unless directed to do so by a senior AD Replication SME)

Refer to the following table to determine TSL default values

Table 4: Default TSL Values

OS Install Path Default TSL

Windows 2000 RTM 60 days

Windows 2003 RTM, 2003 R2 60 days

Windows 2000RTM upgrade to Windows 2003 SP1 60 days

Windows 2003SP1, 2003SP2, 2008, 2008R2 180 days

NT4 upgrade to Windows 2003 SP1 180 days

Removing Outdated Objects Following Expiration of Tombstone Lifetime

If a domain controller fails to replicate for a number of days exceeding the tombstone

lifetime, replicas of objects that have been deleted from a writable partition might remain in

that domain controller's directory. Because the tombstones of the deleted objects are

permanently removed from the directory at the end of the tombstone lifetime, a domain

controller that fails to replicate changes for tombstoned objects never deletes or garbage

collects deleted objects.

This condition can occur for a variety of reasons, including the following:

Prolonged misconfigurations (such as those that cause 1311 events);

Prolonged errors in name resolution, authentication, or the replication engine, each of

which blocks inbound replication;

Turning on a domain controller that has been offline for more than 60 days;

and,

Advancing system time or reducing TSL values in an attempt to accelerate garbage

collection before end-to-end replication has occurred for all naming contexts in the forest.

To avoid such conditions, incorporate monitoring regimens that detect domain controller

replication problems.

Page 47: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 41

Outdated objects can also occur due to hardware and software problems that render the

domain controller unreachable. Regardless of the reason, a deleted object can remain on a

domain controller in either of the following circumstances.

A domain controller goes offline immediately before the deletion of an object on another

domain controller, and remains offline for a period that exceeds the tombstone lifetime.

A domain controller goes offline immediately after the deletion of an object on another

domain controller, but before receiving replication of the tombstone, and remains offline for

a period that exceeds the tombstone lifetime.

The following provides information for a legacy operating system but is included here as it is

still relevant. Additionally, some pre-Windows 2000 SP3 domain controllers experience a

replication error condition after a non-authoritative restore. A large number of objects

created after the restore may never be considered for replication.

More Information: For more information about this topic, see:

Microsoft Knowledge Base Article 316829, “Possible Active Directory Inconsistency after You Restore a Domain Controller.”

On domain controllers that are running Windows Server 2003 or later, you can use the

Repadmin support tool to analyze and remove lingering objects from a domain controller that

you suspect or know has not replicated for a tombstone lifetime. This tool includes the

RemoveLingeringObjects command. This command removes objects that are outdated (do

not exist in a replica of the same directory partition on the source domain controller).

Problems with Lingering Objects

In Windows 2000, if an attribute for a lingering object had been replicated, the inbound

domain controller that had previously processed the deletion would re-animate the entire

object. However, this is undesirable for a number of reasons.

The lingering object is holding a value on a unique attribute, such as samAccountName, that

another object wants to use. This commonly occurs when the lingering object exists in the

read-only naming context but not the domain naming context.

The lingering object is a security risk. For example, it might represent a user that should be

deleted.

The lingering object only exists in the read-only naming context (global catalog). This

behavior makes the object difficult to delete in Windows 2000.

Important: A deleted user or group account remains in the global address list (GAL) on Exchange

servers. Therefore, although the account name appears in the GAL, attempts to send e-mail messages result in errors.

Multiple copies of an object appear in the object picker or GAL for an object that should be unique in the forest. Duplicate objects sometimes appear with altered

Page 48: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 1: Lingering Objects Fundamentals

42 © 2011 Microsoft Corporation. All rights reserved.

names, causing confusion on directory searches. For example, if the relative distinguished name of two objects cannot be resolved, conflict resolution appends "*CNF:GUID" to the name, where * represents a reserved character, CNF is a constant that indicates a conflict resolution, and GUID represents the objectGUID attribute value.

E-mail messages are not delivered to a user whose Active Directory account appears to be current. After an outdated domain controller or global catalog server becomes reconnected, both instances of the user object appear in the global catalog. Because both objects have the same e-mail address, e-mail messages cannot be delivered.

A universal group that no longer exists continues to appear in a user’s access token. Although the group no longer exists, if a user account still has the group in its security token, the user might have access to a resource that you intended to be unavailable to that user.

A new object or Exchange mailbox cannot be created, but you do not see the object in Active Directory. An error message reports that the object already exists.

Searches that use attributes of an existing object incorrectly find multiple copies of an object of the same name. One object has been deleted from the domain, but it remains in an isolated global catalog server.

Strict and Loose Replication Constancy

If the attributes on a lingering object never change, the object is never considered for

replication. However, if an attribute changes, the attribute is considered for outbound

replication. The problem is that the receiving domain controller does not hold the object for

the attribute being replicated. An update cannot be performed because the entire object does

not exist on the partner domain controller. What happens next depends on the replication

consistency set on the domain controller.

Loose Replication Consistency

When replication consistency is set to loose, the receiving domain controller detects that it

does not have the object for the attribute that is being replicated. The inbound partner

requests the entire object from the outbound partner, and reanimates the object on its copy

of the directory. The same process repeats on all domain controllers that do not have a copy

of the object.

This mechanism can be used to “reanimate” lingering objects across the entire forest. If a

lingering object is discovered and its presence is appropriate, then you may perform any

update to that object. As long as replication consistency is set to loose on all domain

controllers, the object will be reanimated as it replicates around the forest.

“Loose replication consistency” is the default for Windows 2000 domain controllers (except

on domain controllers that have the Security Rollup Package installed from November 2001).

Page 49: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 43

Strict Replication Consistency

Because of the issues outlined above in the Problems section, the default behavior for

Windows Server 2003 (and upgraded Windows NT 4.0 domain controllers) is to block

inbound replication per naming context when a domain controller receives an update to an

object that it does not have. Replication is halted in the naming context for the object until the

lingering object is removed or the replication mode is set to loose.

Storage for Consistency Setting

The setting for replication consistency is in the registry on each domain controller.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

Strict Replication Consistency

Value: 1 (Set to 0 to disable)

Data type: REG_DWORD

Note

A post-SP2 hot fix (also included in the SRP) from November of 2001 used a different registry

value. A setting of 0 will not recreate the missing object (strict), and a setting of 1 will create

the missing object. This value is only needed with the November version of the hot fix.

Value Name: Correct Missing Objects

Data type: REG_DWORD

Value: 1

Defines how a destination DC behaves if a source DC sends updates to an object that does

not exist in the destination DC’s local copy of Active Directory.

o Destination DCs should see USN for creates before object is modified

o Only modifies for lingering objects arrive for object not on destination DC

o Only destination DC’s enforce strict replication and log events

Destination DCs stop replicating from source DC’s partitions containing LO’s

Lingering objects are quarantined on source DCs where they can be detected

End-to-end replication may be impacted for partitions containing lingering objects

Administrators must remove lingering objects to restore replication

Enabling Strict Replication

Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command

prompt:

For all domain controllers, type:

repadmin /regkey * +strict

Page 50: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 1: Lingering Objects Fundamentals

44 © 2011 Microsoft Corporation. All rights reserved.

For all global catalog servers, type:

repadmin /regkey gc: +strict

You can also enable strict replication by manually setting the Strict Replication Consistency

registry value to 1.

HKLM\System\CurrentControlSet\Services\NTDS\Parameter Strict Replication Consistency

(Reg_DWORD) to 1

1 (enabled): Inbound replication of the specified directory partition from the source is

stopped on the destination.

Warning: Ensure you are prepared to deal with replication failures after enabling strict replication consistency.

Loose Replication Consistency

If you enable Loose Replication Consistency, if a destination receives a change to an object

that it does not have, the entire object is replicated to the target for the sake of replication

consistency. This behavior causes a lingering object to be reapplied to all domain controllers

in the replication topology.

Enable Loose Replication

Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command

prompt:

For all domain controllers, type:

repadmin /regkey * -strict

For all global catalog servers, type:

repadmin /regkey gc: -strict

You can also enable strict replication by manually setting the Strict Replication Consistency

registry value to 0.

HKLM\System\CurrentControlSet\Services\NTDS\Parameters Value: Strict Replication Consistency Type: (Reg_DWORD) Value Data: 0 0 (disabled): The destination requests the full object from the source domain controller, and the lingering object is revived in the directory as a new object.

Critical: The Loose Replication Consistency setting will cause the undesirable behavior of reanimation of lingering objects.

Page 51: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 45

Ensure Strict Replication Consistency Is Enabled On Newly Promoted

Domain Controllers

If you are upgrading a forest that was originally created using a computer running Windows

2000 Server, you should ensure that the forest is configured to enable strict replication

consistency on newly promoted domain controllers to help avoid lingering objects. After you

update the forest, all new domain controllers that you subsequently add to the forest are

created with strict replication consistency disabled. However, you can implement a forest

configuration change that causes new domain controllers to have strict replication

consistency enabled. To ensure that new domain controllers that you add to the forest have

strict replication consistency enabled, you can use Ldifde.exe to create an object in the

configuration directory partition of the forest. This object is responsible for enabling strict

replication consistency on any Windows Server 2003 domain controller that is promoted into

the forest.

The object that you create is an operational GUID with the following name:

CN=94fdebc6-8eeb-4640-80de-ec52b9ca17fa,CN=Operations,CN=ForestUpdates,CN=Configuration,DC=<ForestRootDomain>

Perform the following procedure on any domain controller in the forest to add this object to

the configuration directory partition.

Requirements:

Administrative credentials: To complete this procedure, you must be a member of the

Domain Admins group.

Tools: Ldifde.exe, Notepad

To create the object that ensures strict replication consistency on new domain

controllers

1. In a text editor such as Notepad, create the following text file:

dn: CN=94fdebc6-8eeb-4640-80de-ec52b9ca17fa,CN=Operations,CN=ForestUpdates,CN=Configuration,DC=<ForestRootDomain> changetype: add objectClass: container showInAdvancedViewOnly: TRUE name: 94fdebc6-8eeb-4640-80de-ec52b9ca17fa objectCategory: CN=Container,CN=Schema,CN=Configuration,DC=<ForestRootDomain>

Where <ForestRootDomain> contains all domain components (DC=) of the forest root

domain. For example, for the contoso.com forest, DC=contoso,DC=com; for the

fineartschool.net forest, DC=fineartschool,DC=net.

Page 52: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 1: Lingering Objects Fundamentals

46 © 2011 Microsoft Corporation. All rights reserved.

2. Open a Command Prompt as an administrator: On the Start menu, right-click Command

Prompt, and then click Run as administrator. If the User Account Control dialog box

appears, provide Enterprise Admins credentials, if required, and then click Continue.

3. At the command prompt, type the following command and then press ENTER:

ldife -i –f <Path\FileName>

Value Description

-i Specifies import mode. If not specified, the default mode is

export.

-f Identifies the import or export file name.

<Path\FileName> The path and name of the import file that you created in step 1.

For example, C:\ldifde.txt.

More Information: For more information about this topic, see:

http://technet.microsoft.com/en-us/library/cc780362(WS.10).aspx

Abandoned object

An object created on one DC that is not replicated to other DCs hosting a writable copy of the NC but is replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition. The net effect is the object exists only in read-only copies of the partition. The object is present on RODCs or GCs hosting a read-only copy of the partition.

Abandoned delete

An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.

Table 5: Lingering Object Terminology

Term Definition

Abandoned delete An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.

Abandoned object An object created on one DC that never got replicated

Page 53: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 47

to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition.

Lingering link

A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale references are referred to as lingering links.

Lingering Object

An object that is present on one replica, but has been deleted and garbage collected on another replica.

Loose Replication Consistency With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This undesirable behavior causes a lingering object to be “reanimated.”

Strict Replication Consistency With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, replication is blocked

with the source DC for the partition where the lingering object was detected

Tombstone An object that has been deleted but not yet garbage collected

Tombstone Lifetime (TSL)

The amount of time tombstones are retained in Active Directory before being garbage collected and permanently purged from the database.

Page 54: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 2: Symptoms and Cause

48 © 2011 Microsoft Corporation. All rights reserved.

Lesson 2: Symptoms and Cause It is uncommon for an Administrator to be aware of and want to resolve a lingering object

problem without first experiencing some other problem in their environment that leads them

to discover the lingering object issue. This lesson will present common symptoms and causes

of lingering objects.

What You Will Learn

After completing this lesson, you will be able to:

Identify four symptoms of lingering object issues

Explain three ways in which lingering objects are created

List at least three methods to prevent lingering objects.

Symptoms of Lingering Objects

Detection of Domain Controllers That Have Not Replicated in

the Tombstone Lifetime

Windows Server 2003 records the last time a domain controller has replicated (directly or

transitively). Each domain controller will periodically compare the last time a domain

controller replicated with the forest’s tombstone lifetime. If a domain controller does not

replicate within the tombstone lifetime, event 1864 is posted to the directory service (DS)

log.

Event ID: 1864 NTDS Replication This is the replication status for the following directory

partition on the local domain controller. The local domain controller has not recently

received replication information from a number of domain controllers. The count of domain

controllers is shown, divided into the following intervals.

More than 24 hours: 1

More than a week: 1

More than one month: 1

More than two months: 1

More than a tombstone lifetime: 1

Tombstone lifetime (days): 60

If a domain controller in this state attempts to replicate, the inbound domain controller will

block replication and alert the administrator with the message below (event 2042). In this

case, the administrator has the following options.

1. Forcefully demote or reinstall the domain controllers that have not replicated, and then

perform a metadata cleanup.

Page 55: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 49

2. Remove any lingering objects on the non-replicating domain controller, and then enable

replication with divergent or corrupt partners (as follows).

a. Run repadmin /removelingeringobjects (see “Removing Lingering Objects with

Repadmin” for instructions).

b. Enable replication with divergent or corrupt partners by adding the following

registry key.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

Allow Replication With Divergent and Corrupt Partner

Value: 1 (Set to 0 to disable)

Data type: REG_DWORD

Important

Before using the above-mentioned key to override this replication safeguard, be sure to use

repadmin /removelingeringobjects command to prevent the spread of unwanted lingering

objects. Once replication has succeeded, be sure to remove the “Replication With Divergent

and Corrupt Partner” value, or set it to zero.

Event Source: NTDS Replication

Event Type: Error

Event Category: Replication

Event ID: 2042

Description:

It has been too long since this machine last replicated with the named source machine. The time

between replications with this source has exceeded the tombstone lifetime. Replication has been

stopped with this source.

The reason that replication is not allowed to continue is that the two machine's views of deleted

objects may now be different. The source machine may still have copies of objects that have been

deleted (and garbage collected) on this machine. If they were allowed to replicate, the source machine

might return objects which have already been deleted.

Time of last successful replication:

<date and time of last replication>

Invocation ID of source:

<invocation ID of the source DC>

Name of source:

<replication guid._msdcs.forest.root of source DC>

Tombstone lifetime (days):

60

The replication operation has failed.

User Action:

Determine which of the two machines was disconnected from the forest and is now out of date. You

have three options:

1. Demote or reinstall the machine(s) that were disconnected.

Page 56: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 2: Symptoms and Cause

50 © 2011 Microsoft Corporation. All rights reserved.

2. Use the "repadmin /removelingeringobjects" tool to remove inconsistent deleted objects and then

resume replication.

3. Resume replication. Inconsistent deleted objects may be introduced. You can continue replication by

using the following registry key. Once the systems replicate once, it is recommended that you remove

the key to reinstate the protection.

Registry Key:

HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Allow Replication With Divergent and

Corrupt Partner

Replication Errors Caused by Lingering Objects

If a domain controller has replicated within the tombstone lifetime and replication

consistency is set to loose, administrators cannot be alerted to the presence or replication of

lingering objects. If an attribute is changed, the object is re-animated by all participating

domain controllers without notification. If strict replication is enabled in the domain,

replication of the partition hosting the object is halted on all inbound domain controllers.

Replication for the partition stops until the object is removed or replication consistency is set

to loose. When the replication is halted, the following error message is reported in the DS log

on the inbound domain controller.

Event ID: 1988

Event Type: Error

Event Source: NTDS Replication

Event Category: Replication

Description:

Another domain controller has attempted to replicate into this domain controller an object which is

not present on this domain controller. The object may have been deleted and already garbage collected

(a tombstone lifetime or more has past since the object was deleted) on this domain controller.

Replication will not continue with the source domain controller until the situation has been resolved.

Source DC:<DC guid>._msdcs.<forestroot>

Object:<dn of object>

Object GUID: <guid of object>

User Action:

Verify that the object was deleted on this domain controller or in the forest. If object restoration is

desired, authoritatively restore the object on the source domain controller. If restoration isn't desired,

install the support tools included on the installation CD and use "repadmin /removelingeringobjects"

on the source domain controller to remove the object from the forest and continue replication. To

allow automatic restoration of this object and future similar objects on this domain controller, the

following registry key can be deleted.

Registry Key:

HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Strict Replication Consistency

Repadmin /showreps

Page 57: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 51

In addition to the above-mentioned event, repadmin reports the following.

Sitename\<DC sending lingering object> via RPC

DC object GUID: <dsa guid of dc>

Last attempt @ 2002-07-19 19:14:43 failed, result 8606

(0x219e):

Insufficient attributes were given to create an object. This object may not exist because it may have

been deleted and already garbage collected.

Cause of Lingering Objects

How lingering objects occur

When a domain controller is disconnected for a period that is longer than the TSL, one or

more objects that are deleted from Active Directory on all other domain controllers may

remain on the disconnected domain controller. Such objects are called lingering objects.

Because the domain controller is offline during the time that the tombstone is alive, the

domain controller never receives replication of the tombstone.

When this domain controller is reconnected to the replication topology, it acts as a source

replication partner that has an object that its destination partner does not have.

Replication problems occur when the object on the source domain controller is updated. In

this case, when the destination partner tries to inbound-replicate the update, the destination

domain controller responds in one of two ways:

If the destination domain controller has Strict Replication Consistency enabled, the

controller recognizes that it cannot update the object. The controller locally stops

inbound replication of the directory partition from the source domain controller.

If the destination domain controller has Strict Replication Consistency disabled, the

controller requests the full replica of the updated object. In this case, the object is

reintroduced into the directory.

Five Causes of Lingering Objects

Cause 1: The source DC sends updates to object that have already been garbage collected on the destination Dc either because the source DC has been offline or has failed replicati An object deleted on

one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.

Page 58: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 2: Symptoms and Cause

52 © 2011 Microsoft Corporation. All rights reserved.

An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.

on for TSL elapsed # of days

The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC2 experiences a motherboard failure. Meanwhile, DC1 makes originating deletes for stale security groups over each of the next 90 days. After being offline for 90 days , DC2 gets its a replacement motherboard, powers up then originates an ACL change on all user accounts before it inbound replicates knowledge of originating deletes from DC1. DC1 logs 8606 errors for updates security groups purged on DC1 for the 1st 30 days that DC2 was offline.

Cause 2: The Source DC sends updates to objects @ the cusp of TSL expiration

that have already been garbage collected by a strict mode destination DC

The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. DC1 is in-place upgraded to W2K8 R2 which stamps new attributes on all objects in the configuration and writable domain partitions, including objects currently in the deleted objects container, some of which were deleted 60 days ago and now at the cusp of tombstone expiration. DC2 garbage collects some of the objects deleted TSL days ago before the replication schedule opens with DC2. Error 8606 is logged until DC1 garbage collects the blocking objects. Any updates to the partial attribute set can cause temporary lingering objects that, like the addition of the 1st W2K8 R2 DC to an existing forest, will clear themselves up once source DCs garbage collect deleted objects @ the cusp of TSL expiration.

Cause 3: A time jump on a destination DC prematurely accelerates the garbage

collection of deleted objects on a destination DC

The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. The reference time source used by DC1 (but not DC2) rolls forward to calendar year 2039, causing DC2 to also adopt a system time in CY2039 which causes DC1 to prematurely purge objects deleted today from its deleted objects container. DC2 meanwhile originates changes to attributes on users, computers and groups that are live on DC2 but deleted and now prematurely garbage collected on DC1. DC1 will log error 8606 when it next inbound-replicates changes for the premature deleted objects.

Cause 4: An object is reanimated at the cusp of TSL expiration

The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. An OU containing users, computers and groups is

Page 59: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 53

accidentally deleted. A system state backup made at the cusp of TSL in the past is auth restored on DC2. The backup contains objects that are live on DC2 but already deleted and garbage collected DC1.

Cause 5: A USN bubble is triggered the logging of the 8606

Say you create an object in a USN bubble, such that it doesn’t outbound replicate because the destination DC "thinks" it has the object due to the bubble. Now, after the bubble closes and new changes start replicating again, a change is created for that object on the source DC and appears as a lingering object to the destination DC which logs the 8606 event.

Lingering Object Prevention

It's easy to come up with methods to prevent lingering objects, now that you know how they

are caused. Keep the following in mind the next time someone asks you what they need to do

to ensure they hit this issue again.

Important: o Resolve replication failures within TSL

o Ensure Strict Replication Consistency is enabled

o Ensure large jumps in system time are blocked via registry key or policy

o Don't remove replication quarantine with "allowDivergent" setting without removing LOs first

o Don't restore system backups that are near TSL number of days old

o Don't bring DCs back online that haven't replicated within TSL

Page 60: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 3: Identification and Classification

54 © 2011 Microsoft Corporation. All rights reserved.

Lesson 3: Identification and Classification

What You Will Learn

After completing this lesson, you will be able to:

Use repadmin.exe to generate diagnostic data for analysis

Use diagnostic data to determine the scope of the problem by listing all partitions

and all servers containing lingering objects.

Create a replication health report A good first step in tracking down the cause of Active Directory replication failures is to get a

list of the replication errors encountered. This is a very simple procedure using repadmin

/showrepl with the /csv option. For every domain controller in the forest, the spreadsheet

shows the source replication partner, the time that replication last occurred, and the time

that the last replication failure occurred for each naming context (directory partition). By

using Autofilter in Excel, you can view the replication health for working domain controllers

only, failing domain controllers only, or domain controllers that are the least or most current,

and you can see the replication partners that are replicating successfully.

To generate a forest-wide replication status spreadsheet for domain controllers:

1. Open a Command Prompt as an administrator: On the Start menu, right-click

Command Prompt, and then click Run as administrator. If the User Account Control

dialog box appears, provide Enterprise Admins credentials, if required, and then click

Continue.

2. At the command prompt, type the following command, and then press ENTER

repadmin /showrepl * /csv >showrepl.csv

3. Open Microsoft Excel.

4. Click the Office button (File menu for versions prior to Excel 2010), click Open,

navigate to showrepl.csv, and then click Open.

5. Hide or delete column A and column G, as follows:

To hide a column, right click the column header then click Hide

To delete a column, right click the column header then click Delete

6. Select a column that you want to hide or delete.

7. Select row 1 beneath the column heading row. On the View tab, click Freeze Panes, and

then click Freeze Top Row.

8. Select any cell. On the Data tab, click Filter.

Page 61: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 55

9. In the Last Failure Status column, click the filter down arrow, deselect the value 0.

You now have a filtered report showing only the replication failures. Deselect all values

except value 8606 to display just the replication failures caused by lingering objects..

Try This: Generate an AD Replication report using repadmin

Take what you have learned and try to use repadmin.exe to generate a forest-wide AD

Replication report

1. Connect to DC1 in your lab environment.

2. Use the steps documented above to generate a filtered report.

3. Save the report to the desktop as showrepltimestamp.xls

Use AD Replication report and repadmin to determine the scope of the problem

The list of DCs in the Source DC column contain lingering objects when the replication report

is filtered on value 8606 in column K. This display gives you the following information:

DC containing lingering objects

Partition where lingering objects exist

This is two of the three data points needed for repadmin /removelingeringobjects.

Important: Repadmin /RemoveLingeringObjects DestinationDC SourceDC_Guid DirectoryPartition (Optional switch /advisory_mode)

DC containing lingering objects = DestinationDC

Partition where lingering objects exist = DirectoryPartition

A common misconception is that the list you have just generated is comprehensive and once

you remove lingering objects from the DCs in the Source DC column your job is done.

However, that may not be the case as this is only a list of DCs where replication is currently

blocked. It is entirely possible that once you remove lingering objects from these DCs,

replication will begin failing with these now-clean DCs as the destination and a new list of

DCs as the source. Once you have a list of DCs containing lingering objects

Tip: To save time, act as if all DC / GCs contain lingering objects for the partition in question.

Page 62: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 3: Identification and Classification

56 © 2011 Microsoft Corporation. All rights reserved.

Run repadmin /removelingeringobjects in /advisory_mode first to see what objects are

considered lingering on the DC. Event ID 1946 is logged once per lingering objects on the

destination DCs Directory Services event log.

Tip: Increase the size of the Directory Services event log prior to running repadmin /removelingeringobjects with the /advisory_mode option. It is common to see the event log wrap when this command is run and the event log is the default size.

You can also use ldifde and replfix.exe to generate a list of lingering objects. This process is describe in Lesson 4.

Page 63: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 57

Lesson 4: Lingering Object Removal

What You Will Learn

After completing this lesson, you will be able to:

Execute the steps in an action plan in order to remove lingering objects

Remove lingering objects using five different methods

Methods to Remove Lingering Objects Common methods to remove lingering objects include:

Repadmin /Removelingeringobjects

Repldiag

Replfix

Manually through LDP or using script

Rehost the partition: o Repadmin /rehost (or /unhost and /add) (only if the partition is not-writable on the

DC containing lingering objects) o Un-GC (but you don’t really have control over who the DCs sources the partition

from) o Demote and Promote (DCPromo)

Removing Lingering Objects with Repadmin

Repadmin includes an advanced switch (view using /experthelp) to remove lingering objects

from a specific server.

To remove outdated (lingering) objects from a directory partition on a domain

controller that has not replicated for a tombstone lifetime, perform the following.

1. Using Repadmin, type the following at the command line:

Repadmin /RemoveLingeringObjects DestinationDC SourceDC_Guid

DirectoryPartition (Optional switch /advisory_mode)

where

DestinationDC is the DNS name or IP address of the domain controller that has

outdated objects; and,

SourceDC_Guid is the domain controller’s object GUID.

To obtain the object’s GUID, do one of the following.

Page 64: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal

58 © 2011 Microsoft Corporation. All rights reserved.

o Use Repadmin /showrepl SourceDCName. The domain controller’s object GUID is

listed as “domain controller object GUID.”

-or-

o In Active Directory Sites and Services, find the Source domain controller under

Sites\<the domain controller’s Site>\ Servers\ DCname\ NTDS Settings\ Properties.

Look in the DNS Alias box. The GUID prior to _msdcs.forestrootname.com is the

domain controllers Object GUID. Repadmin only needs the GUID. Omit

_msdcs.forestrootname.com from the Repadmin syntax.

DirectoryPartition is the distinguished name of the directory partition from which to

remove outdated objects.

2. Repeat the procedure for the following partitions, as needed.

Domain directory partition dc=DomainName…,dc=ForestRootDomainName

Configuration directory partition cn=configuration,dc=DomainName…,dc=ForestRootDomainName

Application directory partition or partitions cn=ApplicationDirectoryPartitionName,dc=DomainName…,dc=ForestRootDomainName

Schema directory partition cn=schema,cn=configuration,dc=ForestRootDomainName

The following is an example of the command syntax.

C:\>repadmin /removelingeringobjects lonemeadc.emea.contoso.com B0AE6093-15F5-

4DB8-836B-4495F3B19493 dc=contoso,dc=com /advisory_mode

RemoveLingeringObjects successful on lonemeadc.emea.contoso.com

Events Associated with Lingering Object Removal

When removing lingering objects, the target domain controller (the domain controller with

the lingering objects) will record all removal information, including source domain controller,

objects removed, and a total count of all objects removed.

Event ID 1937: NTDS Replication. Lingering Object Removal has been initiated on this

domain controller. All objects on this DC will have their existence verified on the following

source domain controller. Objects that have been deleted and garbage collected from the

source domain controller will be DELETED from this domain controller if they still exist.

Subsequent event logs will list all deleted objects.

Source DC: <source DC guid ._msdcs.<forest root>

Event ID 1945: NTDS Replication. Lingering Object Removal will DELETE the following

object. Its deletion and garbage collection was detected on the source domain controller

without replicating the deletion to this domain controller.

Page 65: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 59

Object:DC= <dn of lingering object>

Object GUID:<objectGUID>

Source DC: <dc guid> ._msdcs.<forest root>

Event ID 1939: NTDS Replication. Lingering Object Removal has executed successfully on

this domain controller. All objects on this domain controller have had their existence

verified on the source domain controller. Objects that had been deleted and garbage

collected from the source domain controller were DELETED from this domain controller.

Previous event logs list all such objects.

Source DC: <source DC guid> ._msdcs.<forest root>

Lingering Objects Deleted 23

Details of Repadmin’s Lingering Object Removal Mechanism

To be added after external reviews are complete.

Remove Lingering Objects Using Repldiag

Removing lingering objects from a forest with repldiag is as simple as running repldiag

/removelingeringobjects. However, it is usually best to exercise some control over the

process in larger environments. The option /OverRideReferenceDC allows you to select

which DC is used for cleanup. The option /outputrepadmincommandlinesyntax allows you to

see what a forest-wide cleanup looks like using repadmin.

Tip: Repldiag is by far the easiest and fastest way to remove lingering objects. The other methods are important to know when repldiag is not an option.

Help

Replication topology analyzer. Written by [email protected] Version: 2.0.3397.24022 Command Line Options: ReplDiag [/Save] [/CheckForStableReplTopology] [/RemoveLingeringObjects] [/ImportData:<FileName.XML>] [/ShowTestCases] [/OverrideDefaultReferenceDC:"dc=namingcontext,dc=com":domainController.namingcontext.com] /UseRobustDCLocation -Query each and every DC for a list of DCs in

forest. Ensures replication instability does not cause any to be missed.

/Save -Save out the data from the current environment to XML. File is named "ReplicationData.xml" and is located in the current directory.

/ImportData -Import the XML that was saved during a prior execution of this utility. Run one of the other options to do something with the data.

/ShowTestCases -Show detail about test cases. Lingering Object Cleanup: /RemoveLingeringObjects -Use the current forest topology to clean all the

Page 66: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal

60 © 2011 Microsoft Corporation. All rights reserved.

NCs in the forest. WILL NOT CLEAN WINDOWS 2000 SYSTEMS!!!

/AdvisoryMode -Check for lingering objects only, do not clean. Must be used with /RemoveLingeringObjects.

/OverrideDefaultReferenceDC -Specify reference DC for a naming context when when removing lingering objects, can be used multiple times for different NCs. Only functional if using /RemoveLingeringObjects.

/OutputRepadminCommandLineSyntax -Output the command line syntax for repadmin. Only active in conjunction with /RemoveLingeringObjects.

Example syntax: ReplDiag /Save - Collect the AD replication topology from the environment and save it. ReplDiag /ImportData:"ReplicationData.xml" - Load in previously collected data and check replication status. ReplDiag /RemoveLingeringObjects /OverrideDefaultReferenceDC:"cn=Configuration,dc=forestroot,dc=com":dc1.forestroot.com /OverrideDefaultReferenceDC:"dc=forestroot,dc=com":dc2.forestroot.com

Sample output

Repldiag.exe /save

Open ReplicationData.xml in Excel

Repldiag /removelingeringobjects /outputrepadmincommandlinesyntax This will give you output of corresponding repadmin /removelingeringobjects syntax. It will first select one DC per partition to be used as a reference DC. It will then clean the reference DCs up against all other DCs for the partition(s) it was selected to be used as a reference for. Finally it cleans up all other DCs in the forest with the new “cleaned up” reference DCs as sources. The /outputrepadmincommandlinesyntax option does not actually attempt object cleanup. You would need to leave this option off if you want to execute lingering object cleanup.

Number Complete,Status,Server Name,Naming Context,Reference DC,Duration,Error Code,Error Message repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=forestdnszones,dc=contoso,dc=com

Page 67: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 61

repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=corp,dc=contoso,dc=com Reference NCs cleaned in 0h:0m:0s. Cleaning everything else against reference NCs. repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com All NCs cleaned in 0h:0m:0s.

Page 68: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal

62 © 2011 Microsoft Corporation. All rights reserved.

This output can also be viewed in Excel: Copy commands to a text file. Modify the text file to include only the command portion of the output. Then open up the text file in Exel. (space delimited)

More control: /OverRideReferenceDC This option allows you to specify a DC that you want to be used as a reference DC for the partition specified. In a large distributed environment, take careful consideration when choosing the reference DC. Things to consider when choosing a suitable reference DC: Well connected: Fast WAN link. Performance: Excellent server class hardware: Disk, RAM, CPU and NIC Critical Network Applications / Services do not depend on this DC: Such as an Exchange

facing DC Other DCs don’t report replication failures with reference DC as the source: filter repadmin

/showrepl * /csv ouput, or use the topology report created by repldiag /save. repldiag /removelingeringobjects /overridedefaultreferencedc:"cn=configuration,dc=contoso,dc=com":nycorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=corp,dc=contoso,dc=com":seacorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=forestdnszones,dc=contoso,dc=com":5thwardcorpdc.corp.contoso.com /outputrepadmincommandlinesyntax Replication topology analyzer. Written by [email protected] Version: 2.0.3397.24022 Command Line Switch: /removelingeringobjects Command Line Switch: /overridedefaultreferencedc:cn=configuration,dc=contoso,dc=com:nycorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=corp,dc=contoso,dc=com:seacorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=forestdnszones,dc=contoso,dc=com:5thwardcorpdc.corp.contoso.com Command Line Switch: /outputrepadmincommandlinesyntax Attempting to override NC cn=configuration,dc=contoso,dc=com with DC nycorpdc.corp.contoso.com... Overriden Attempting to override NC dc=corp,dc=contoso,dc=com with DC seacorpdc.corp.contoso.com... Overriden Attempting to override NC dc=forestdnszones,dc=contoso,dc=com with DC 5thwardcorpdc.corp.contoso.com... Overriden

/UseRobustDCLocation Query each and every DC for a list of DCs in forest. Ensures replication instability does not cause any to be missed. We’ve had cases where we clean up lingering objects in the forest but do to an AD topology problem some DCs were not cleaned up. This option is almost always recommended if you want it to do a thorough job.

Page 69: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 63

Remove Lingering Objects Using Replfix

Remove Lingering Object using LDP or Script

Removing Lingering Objects in Windows 2000

Unfortunately, Windows 2000 provides no easy way to detect and remove lingering objects. A

supported method to delete these objects is documented in MSKB 314282: “Lingering Objects

May Remain After You Bring an Out-of-Date Global Catalog Server Back Online”

In Windows 2000 SP3 (and in the post-SP2 hot fix), enhancements were made that allow an

administrator to enable strict replication. This will help identify lingering objects and prevent

them from replicating. However, lingering objects will not be detected unless an attribute on

the object is changed.

Note: Even though this method was first used for Windows 2000, it is still sometimes needed in certain scenarios.

Remove Lingering Objects by partition re-host operation

When one of the other methods is not an option, it is sometimes necessary to re-host the

partition from a DC containing a good clean writable copy of the partition. This may be a

temporary solution if the problem is widespread since the DC may later replicate with a DC

that is not clean.

Tip: If re-host is necessary, it is usually best to identify all GCs needing the procedure and clean them up at the same time to prevent recurrence.

Un-hosting a partition

It is sometimes necessary to remove a partition from the database of a DC temporarily.

Repadmin includes a /rehost option that allows you to do this, but the /unhost option allows

you to exercise more control over the procedure. Take note that /unhost only allows you to

remove a read-only copy of the partition. With the exception of application partitions, you

cannot remove a writable copy of a partition from a DC without using DCPROMO.

Repadmin /?:unhost Remove a specific read-only partition from a GC. [SYNTAX] /unhost DSA <Naming Context> Repadmin /unhost ContosoDC1 dc=corp,dc=contoso,dc=com

Page 70: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal

64 © 2011 Microsoft Corporation. All rights reserved.

Event ID 1659 indicates the status of the un-host operation. Do not re-add the partition until

event ID 1660 is logged in the Directory Services event log.

Warning: The re-host operation may fail with error 8339 if you attempt to re-add the partition too soon after the un-host.

Manually adding a replication connection using repadmin.exe

The add command will create a RepsFrom attribute on the destination domain

controller for the specified naming context and initiate a replication request. During a

normal replication cycle, the destination domain controller will request updates from

the source domain controller. When creating temporary replication links between

replication partners, the process could fail if the KCC starts while you are performing

the procedure. The KCC will delete any replication links for which no corresponding

connection object exists. Since these commands can take a very long time to

complete as they trigger the replication of the corresponding naming context, it is

important to ensure that KCC do not disturb the process. This is where you would use

+DISABLE_NTDSCONN_XLATE which effectively disables KCC's capability to translate

connection objects to replication links.

Disable KCC connection translation so that KCC doesn’t remove our temporary

replication connection:

Repadmin /options ContosoDC1 +disable_ntdsconn_xlate

Then add a replication connection for the configuration partition of the server we

want to source the partition from:

Repadmin /add <Naming Context> <Dest DSA> <Source DSA> [/readonly] [/selsecrets] <Source DSA> The source DSA must be specified by fully qualified computername. repadmin /add cn=configuration,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.

Add a replication connection to the server for the domain partition that we

need to source from (/readonly is specified if the partition is a GC non-writable

partition /selsecrets needs to be specified if the destination DC is an RODC):

Page 71: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 65

repadmin /add dc=emea,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com /readonly One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.

If you need to replicate the other way, then just reverse the order of the server names

in the commands.

To begin a normal sync of the partition using the new replication connection:

Repadmin /replicate <Dest_DSA_LIST> <Source DSA_NAME> <Naming Context> [/force] [/async] [/full] [/addref] [/readonly] repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly

To begin a full sync of that partition using the new replication connection:

repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly /full Sync from LONEMEADC.Emea.contoso.com to ContosoDC1.contoso.com completed successfully.

Turn KCC connection translation back on when you no longer need the connection:

Repadmin /options ContosoDC1 -disable_ntdsconn_xlate

DSASTAT Dsastat can be used to compare the number of objects that exist on two domain controllers.

However, it cannot report on which objects exist on one and not the other. Likewise, it cannot

make an intelligent determination about the differences. Replication latency or other factors

might result in valid cases where an object exists but has not replicated out. Some objects are

set to not replicate (like the Universal group membership cache). For this reason, DSASTAT

can only be used as a guideline for comparisons between naming contexts hosted on different

domain controllers.

Read-only Naming Context (Global Catalogs) The global catalog is particularly susceptible to problems caused by lingering objects. This is

because an object can exist on a read-only naming context, but not in the domain naming

context from which it originally replicated. If it still existed in the domain naming context, it

could be deleted there, and the tombstone could remove it from the global catalog. The other

problem is that global catalogs can replicate from each other. The global catalog function

might be removed from a computer, and then reinstated in an attempt to re-replicate the

partial attribute set from a domain controller hosting the writable copy of the naming

Page 72: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal

66 © 2011 Microsoft Corporation. All rights reserved.

context. In this case, the global catalog might replicate from another global catalog. This

would return the object you were trying to delete. A better solution is to determine whether

the object exists on all global catalogs. If it does not, remove the global catalog function from

all servers that contain the object. Then reinstate the global catalog function on all of them,

and let a clean copy of the directory replicate in. In larger environments, removing and

reinstating the global catalog function might be undesirable and prohibited. Applications

such as Microsoft Exchange Server depend on the global catalog to operate. Moreover, the

additional traffic incurred as the domains re-replicate into the global catalog might be

undesirable. In this case, use the post-SP2 hot fix and process described in the following

article.

MSKB 314282: “Lingering Objects May Remain After You Bring an Out-of-Date Global Catalog

Server Back Online”

Page 73: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 67

Lesson 5: Real World Application

What You Will Learn

After completing this lesson, you will be able to:

Create a detailed action plan that will remove the lingering objects in all partitions on

all servers

Recommend the correct method to remove lingering objects given five different

scenarios

Recommend changes that will result in a better solution given a subpar action plan

Determining What to Do with a Lingering Object

In most cases, a lingering object results from a missed tombstone. In other words, the object

was intentionally deleted because one or more domain controllers received the instruction to

delete the object. In rare cases, the object was not actually deleted. In such cases, the

existence of the object may be intended. Determining what to do with a deleted object

depends on whether or not it was intentionally deleted. First, let’s look at some common

lingering object scenarios, and then discuss the recommended corrective action.

Known Cause: Domain controller has not replicated beyond the tombstone lifetime. If a

domain controller has not replicated within the tombstone lifetime, it will likely have

missed the deletions of some attributes (or inbound replicated tombstones). Left alone, the

objects will persist without any replication. If any of the attributes are changed, or if

another domain controller is doing a full sync (as when a global catalog is populating its

copy of the domain partition), the objects will attempt to replicate out and cause problems.

These lingering objects are unintended, and should be removed using repadmin (see

below).

Unknown Cause: Security principal is attempting to replicate in. In some cases, a user or

computer object has become a lingering object without any known cause. These are almost

always undesirable. However, before removing them, check the event log and see what

object is being replicated in. If the object is desired, enable loose consistency. (Refer to the

section Intended Objects below.)

Unknown Cause: Deletion is replicating in (tombstone replicating in). If the inbound object is

a deletion (the object will include DEL in the name), it is probably harmless and not needed.

However, if the deleted object still exists on another domain controller somewhere else in

the forest, removing this lingering object will actually turn the other .live. copy into a

lingering object.

The next section examines what to do with intended and unintended objects.

Unintended Objects

Use Repadmin to delete these lingering objects (see below).

Page 74: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lesson 5: Real World Application

68 © 2011 Microsoft Corporation. All rights reserved.

Intended Objects

Change the replication consistency on the inbound domain controller. The object will be re-

animated on this domain controller. When using this method, the following things should be

considered.

After the object has been reanimated and replicated into the domain controller, it will

replicate out to the domain controller’s other partners. It is not likely that the other

partners will have the object, and inbound replication will be blocked until the consistency

setting is changed. This might result in the lingering object or re-animation moving

throughout the domain. To animate the object fully, you might have to .chase. the replication

failures throughout the forest. Use Eventcomb to monitor for the lingering object detection

event.

While the idea of chasing a lingering object around a forest might not seem like much fun,

there is a good reason to do it. It is possible to turn off replication consistency in a domain

or forest (using scripts or custom ADM files with Group Policy). However, this could have

some unwanted side effects: for example, replication would be blocked for the first lingering

object.

Page 75: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 69

10.0 Lab Guide This lab manual describes the environment required to perform practice exercises in this

course and lab sessions included in this manual.

Before You Begin

Before starting this course you should:

Complete... course prerequisite.

Review... course prerequisite.

Practice exercises are performed on physical and virtual machines on one computer per

participant. To complete the exercises, your computer hardware and software must be

configured as described in this section.

For additional details, refer to the Classroom Setup Guide that accompanies this course.

Critical: Lab sessions that accompany this course use a preconfigured virtual machine environment. If you start or modify VMs in any way prior to use in lab exercises, exercise tasks and steps will not work as intended. DO NOT start or modify any VM until instructed to do so in the lab exercises.

Preconfigured VMs use lab environment scripts to complete certain steps at first launch based on the computer name entered in mini-setup. Failure to enter computer names specified in the lab exercises exactly as shown will incorrectly configure VMs, which will cause lab exercise tasks and steps to fail.

What You Will Learn

After completing the labs in this course, you will be able to:

Describe | Explain… course objective.

Install | Configure… course objective.

Analyze | Troubleshoot... course objective.

Page 76: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab Sessions

70 © 2011 Microsoft Corporation. All rights reserved.

Lab Sessions This manual includes the following lab sessions. Each lab includes step-by-step instructions

to complete the exercises. You can use the problem solving lab exercises in your workbook to

challenge your understanding of course material and refer to the Lab Manual for detailed

steps if needed.

Lab 1: Exploring Lingering Object Fundamentals

During this lab, you will identify the forest's configured tombstone lifetime and replication

consistency settings

Estimated time to complete this lab: 15 minutes

Lab 2: Lingering Object Diagnosis and Documentation

During this lab, you will generate diagnostic data via repadmin, ldifde and replfix. You will

then analyze that data and document all lingering objects in the environment.

Estimated time to complete this lab: 30 minutes

Lab 3: Lingering Object removal using repadmin

During this lab, you will remove lingering objects from the environment using repadmin

/removelingeringobjects.

Estimated time to complete this lab: 30 minutes

Lab 4: Lingering Object removal using ldp and repldiag

During this lab, you will remove a single lingering object using ldp. You will then remove the

remaining lingering objects using repldiag.

Estimated time to complete this lab: 30 minutes

Lab 5: Abandoned Object and Abandoned Deleted object remediation

During this lab, you will identify and remove an abandoned object. You will then remediate

and abandoned deleted object scenario.

Estimated time to complete this lab: 30 minutes

Lab 6: Lingering Link identification and cleanup

During this lab, you will identify all lingering-linked values in the environment. You will them

remove them in order to ensure group membership consistency.

Estimated time to complete this lab: 45 minutes

Setting Up Your Lab Environment To complete this lab, you will need the hardware and software configuration described in this

section.

Page 77: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 71

Hardware

Practice exercises assume that all lab hardware is listed on the Hardware Compatibility List

(HCL) as compatible with operating systems and applications described later in this section.

The following table describes minimum hardware requirements for practice exercises.

Table 6: Minimum Hardware Requirements

Minimum System Requirements

Computer/Processor Computer with a 2.4 GHz processor or higher

(If available, disable hyperthreading and enable hardware virtualization)

Operating System <Host OS>; see Classroom Setup Guide for details

Memory 4 GB RAM

Storage 160 GB hard drive

CD or DVD drive (DVD drive recommended)

Display Super VGA (800 x 600) or higher-resolution monitor with 256 color

(Recommended: 1024 x 768 with 16-bit or higher color)

Peripherals Microsoft Mouse or compatible pointing device

Microsoft or compatible keyboard

Software

Operating systems and applications listed in the following table must be installed on all

computers.

Table 7: Lab Computer requirements

Software Version tested and notes

Microsoft® Windows® 7, Enterprise Edition Service Pack 1

Current Microsoft® Windows® 7, Enterprise Edition Service Pack 1 and Critical Updates

Office 2010 Professional Service Pack 1

Microsoft Office 2010 OneNote Service Pack 1

Current Office 2010 Service Pack 1 and Critical Updates

Microsoft .NET Framework Version 2.0 Retail

Current .NET Framework 2.0 Service Pack and Critical Updates

Microsoft .NET Framework Version 3.0 Retail

Current .NET Framework 3.0 Service Pack and Critical Updates

Page 78: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab Sessions

72 © 2011 Microsoft Corporation. All rights reserved.

Software Version tested and notes

Current Adobe Reader Version Retail

Current Adobe Reader Critical Updates

Network Layout

The following figure illustrates the lab network. The lab network must be isolated from

production networks.

Figure 3: Network Layout

Individual computer configurations are described in detail in the next section.

Computer Names and IP Addresses

Table 8: Lab Computer Names and IP Addresses on page 73 lists computer configurations

for the classroom lab network.

Replace <Host> in Computer Name with <site>-<room> as follows:

<site> Site name abbreviation (Example: For Las Colinas, use LC1 or LC2)

<room> Room number (Example: For Rio Grande classroom, use 1693)

Page 79: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 73

For example, participant computer 1 in the Rio Grande classroom in Las Colinas

Building 2 would be named LC2-1693-1 or LC2-1693-1A (see table).

Replace x in IP address with the classroom number or any representative number that

is unique on the overall classroom subnet and reference this number in all lab

exercises.

Important: This computer naming convention eliminates potential issues when multiple classrooms are connected to the same subnet during classroom configuration or course delivery.

Table 8: Lab Computer Names and IP Addresses

Computer Name IP Address Preferred DNS Server Role

<Host>- Instr-1 172.168.1.200 192.168.x.200 Stand-alone Server

<Host>-Instr-2 172.168.x.201 192.168.x.200 Stand-alone Server

<Host>-1 172.168.x.101 192.168.x.200 Stand-alone Server

<Host>-2 172.168.x.102 192.168.x.200 Stand-alone Server

<Host>-3 172.168.x.103 192.168.x.200 Stand-alone Server

<Host>-4 172.168.x.104 192.168.x.200 Stand-alone Server

<Host>-5 172.168.x.105 192.168.x.200 Stand-alone Server

<Host>-6 172.168.x.106 192.168.x.200 Stand-alone Server

<Host>-7 172.168.x.107 192.168.x.200 Stand-alone Server

<Host>-8 172.168.x.108 192.168.x.200 Stand-alone Server

<Host>-9 172.168.x.109 192.168.x.200 Stand-alone Server

<Host>-10 172.168.x.110 192.168.x.200 Stand-alone Server

<Host>-11 172.168.x.111 192.168.x.200 Stand-alone Server

<Host>-12 172.168.x.112 192.168.x.200 Stand-alone Server

<Host>-13 172.168.x.113 192.168.x.200 Stand-alone Server

<Host>-14 172.168.x.114 192.168.x.200 Stand-alone Server

<Host>-15 172.168.x.115 192.168.x.200 Stand-alone Server

<Host>-16 172.168.x.116 192.168.x.200 Stand-alone Server

Configuring Your Computer(s)

Each student requires one physical machine with a fully configured virtual machine

environment. Before starting this lab, make sure your computer is configured as follows:

<Operating System and Version> installed and started

Page 80: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab Sessions

74 © 2011 Microsoft Corporation. All rights reserved.

Virtual Server 2005 R2 SP1 (may be preinstalled on classroom computers)

Note: Virtual Server 2005 R2 SP1 may already be installed on your computer. If these applications are not installed, you may obtain a free download of the installation files from:

http://www.microsoft.com/downloads/details.aspx?FamilyID=bc49c7c8-4840-4e67-8dc4-1e6e218acce4&DisplayLang=en

Windows Server 2008 DVD media or installation ISO file in <path>.

Virtual machines installed or created on the computer:

o <VMName>: <OS | Role | description>

o <VMName>: <OS | Role | description>

o <VMName>: <OS | Role | description>

Course files located in the C:\Labfiles and C:\VS folders on your computer or

accessible from a network share on the instructor computer.

Accounts and Group Membership

Important: You must log on as an administrative user in order to perform some of the tasks in this lab.

The following user accounts and passwords must be configured on the physical computer and

in all virtual machines:

Administrative username and password

Username: Administrator Password: LS1setup!

Member of: Local Administrators

Normal username and password

Username: Studentn Password: LS1setup!

Member of: Local Users

Replace n in Studentn with the number assigned to your classroom computer by the

instructor.

Domain Membership

Your physical computer is not joined to a domain.

Lab exercises may require you to join the following virtual domain(s):

Contoso.com

Page 81: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 75

Virtual machines joined to a virtual domain require group and account configurations as

shown in the following table.

Table 9: Groups and Accounts

Group Members

Domain Groups

Domain Administrators Administrator

Domain Users Studentn

Local Groups

Administrators Administrator, Domain Administrators

Users Studentn, Domain Users

Shares on Instructor Computer(s)

During lab exercises, you may be required to access the following shares on the instructor

computer(s):

\\<Host>-Instr-1\Labfiles

Includes lab files installed on participant host computers

\\<Host>-Instr-1\VS

Includes files required for virtual machine environment

Using the Keyboard and Mouse in a Virtual Machine This course includes virtual machines for lab exercises. You use the keyboard and a mouse to

control a virtual machine much as you would a physical computer. This section explains how

to use the keyboard and mouse in virtual machines and describes special keys and menu

items.

Using the Keyboard

In general, the keyboard works the same for a virtual machine as it does for a physical

computer. However, some keyboard shortcuts such as Ctrl+Alt+Delete do not work within a

virtual machine because of the interaction between the host operating system and the guest

operating system. Virtual Server 2005 provides much of the required keyboard functionality

with a Host key and keyboard shortcuts. By default, the Host key is the Right-Alt key. You

can use the Host key in two ways:

If a virtual machine has captured the pointer, press the Host key to return control of the

mouse to the host operating system.

Page 82: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab Sessions

76 © 2011 Microsoft Corporation. All rights reserved.

Use the Host key in combination with other keys for specific functions as described in

the following table.

Table 10. Keyboard Shortcuts for Virtual Machines

Key Combination Description

Host Key+Delete Sends Ctrl+Alt+Delete functionality to the virtual machine operating system.

Host Key+C Connects the Remote Control or VMRC to the VMRC server.

Host Key+A Switches the Remote Control or VMRC to the Administrator Display.

Host Key+I Displays connection information.

Host Key+V Sets the virtual machine so that the guest operating system cannot be manipulated. You can only view the virtual machine window.

Host Key+H Displays the control to set the Host key.

Host Key+Enter Switches the virtual machine window to full-screen display. This option is available only when you connect to a virtual machine using the VMRC client.

Host Key+Left Arrow Switches to the previous virtual machine. This option is available only when you connect to a virtual machine using the VMRC client.

Host Key+Right Arrow Switches to the next virtual machine. This option is available only when you connect to a virtual machine using the VMRC client.

Tip: As shown in the preceding table, you can use Host Key+Delete to send the functionality of the Ctrl+Alt+Delete keyboard shortcut to a guest operating system running in a virtual machine. You can also use Send Ctrl+Alt+Del from the Remote Control menu of either the VMRC or Remote View page.

Using the Mouse

The way you use the mouse depends on whether Virtual Machine Additions is installed.

If Virtual Machine Additions is installed on the virtual machine, you can move the

pointer freely between the virtual machine window and the host operating system.

This simplifies switching among virtual machines and the host operating system.

If Virtual Machine Additions is not installed on the virtual machine, the virtual machine

must capture the pointer before the mouse can be used within the virtual machine

window. The virtual machine captures the pointer when you click the pointer inside

the virtual machine window.

If a pointer is captured by a virtual machine on which Virtual Machine Additions is not

installed, the virtual machine must release it before you can use the mouse on the host

Page 83: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 77

operating system or in another virtual machine window. You can use the Host key to return

the use of the mouse to the host operating system.

Page 84: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab 1: Exploring Lingering Object Fundamentals

78 © 2011 Microsoft Corporation. All rights reserved.

Lab 1: Exploring Lingering Object Fundamentals During this lab, you will identify the forest's configured tombstone lifetime and replication

consistency settings

Estimated time to complete this lab: 15 minutes

Before You Begin

To complete this lab:

Complete Lesson 1

What You Will Learn

After completing this lab, you will be able to determine the Active Directory settings that

govern how it handles tombstones and lingering objects.

Exercise 1 Determine tombstone lifetime.

Exercise 2 Determine DC replication consistency setting.

Scenario

You are assisting a customer who is having issues with…

Configuring Your Computer(s) Each student requires at least one physical computer and a fully configured local or remote

hosted virtual machine environment. Before starting this lab, make sure your computer is

configured as described in About This Lab.

Configuring Your Virtual Machine Environment

Exercises in this Lab require the following virtual machines:

VMname: DC1

Exercises may also require files located in the C:\Labfiles folder on your computer or

accessible from a network share on the instructor computer.

Accounts and Group Membership

Important: You must log on as an administrative user in order to perform some of the tasks in this lab.

The following user accounts and passwords must be configured on the physical computer and

in all virtual machines:

Page 85: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 79

Administrative username and password

Username: Administrator Password: LS1Setup!

Member of: Local Administrators

Normal username and password

Username: Usern Password: LS1Setup!

Member of: Local Users

Replace n in Usern with the number assigned to your classroom computer by the instructor.

Domain Membership

Lab exercises may require virtual machines to be joined to the following virtual domain(s):

Contoso.com

Virtual machines joined to a virtual domain require group and account configurations as

shown in the following table.

Table 11. Groups and Accounts

Group Members

Domain Groups

Domain Administrators Administrator

Domain Users Usern

Local Groups

Administrators Administrator, Domain Administrators

Users Usern, Domain Users

Exercise 1: Determine Tombstone Lifetime Setting In this exercise, you will attempt to determine the tombstone lifetime setting of the forest.

Scenario

You are assisting a customer that is having issues

Task Detailed Steps

Complete these steps by connecting to DC1

Page 86: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab 1: Exploring Lingering Object Fundamentals

80 © 2011 Microsoft Corporation. All rights reserved.

Task Detailed Steps

Task Description 1. Step.

a. Sub-step.

Setting | Parameter Value

Item 1

Item 2

b. Sub-step.

c. Sub-step.

2. Step.

Task Description 1. Step.

a. Sub-step.

b. Edit the registry as shown below:

Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1

c. Sub-step

2. Step.

Task Description 1. Step.

a. Sub-step.

b. Sub-step.

2. Step.

Task Description 1. Step.

a. Sub-step.

b. Sub-step.

2. Step.

Review

1. <Question>

Answer

Page 87: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 81

2. <Question>

Answer

Exercise 2: Determine forest and DC replication consistency settings

In this exercise, you will identify the replication consistency settings for each DC in the environment and will determine if the f

Scenario You are assisting a customer that is having issues <add scenario here>.

Task Detailed Steps

Complete these steps by connecting to <VM name>

Task Description 3. Step.

a. Sub-step.

Setting | Parameter Value

Item 1

Item 2

b. Sub-step.

c. Sub-step.

4. Step.

Task Description 3. Step.

a. Sub-step.

b. Edit the registry as shown below:

Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1

c. Sub-step

4. Step.

Task Description 3. Step.

a. Sub-step.

b. Sub-step.

4. Step.

Page 88: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab 1: Exploring Lingering Object Fundamentals

82 © 2011 Microsoft Corporation. All rights reserved.

Task Description 3. Step.

a. Sub-step.

b. Sub-step.

4. Step.

Review 3. <Question>

Answer

4. <Question>

Answer

Page 89: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 83

Lab 2: Lingering Object Diagnosis and Documentation

During this lab, you will generate diagnostic data via repadmin, ldifde and replfix. You will

then analyze that data and document all lingering objects in the environment.

Estimated time to complete this lab: 30 minutes

Before You Begin

To complete this lab:

Complete Lesson 1 and Lesson 2

What You Will Learn

After completing this lab you will be able to <Lab terminal objective>.

After completing the exercises you will be able to:

Exercise 1 enabling objective.

Exercise 2 enabling objective.

Scenario

You are assisting a customer who is having issues with…

Exercise 1: Lingering Object Diagnosis <Briefly describe the goal of the exercise>

Scenario

You are assisting a customer that is having issues <add scenario here>.

Tasks

<Define starting conditions, including virtual machines and lab files required>.

1. <Task>.

a. <Step>.

i. <Sub-step>.

ii. <Sub-step>.

b. <Step>.

2. <Task>.

Page 90: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab 2: Lingering Object Diagnosis and Documentation

84 © 2011 Microsoft Corporation. All rights reserved.

Setting | Parameter Value

Item 1

Item 2

3. <Task>.

a. <Step>.

Edit the registry as shown below:

Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1

b. <Step>.

Review

1. <Question>

Answer

2. <Question>

Answer

Exercise 2: Lingering Object Documentation <Briefly describe the goal of the exercise>

Scenario

You are assisting a customer that is having issues <add scenario here>.

Tasks

<Define starting conditions, including virtual machines and lab files required>.

Page 91: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 85

4. <Task>.

a. <Step>.

i. <Sub-step>.

ii. <Sub-step>.

b. <Step>.

5. <Task>.

Setting | Parameter Value

Item 1

Item 2

6. <Task>.

a. <Step>.

Edit the registry as shown below:

Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1

b. <Step>.

Review

3. <Question>

Answer

4. <Question>

Answer

Page 92: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab 3: Lingering Object removal using repadmin

86 © 2011 Microsoft Corporation. All rights reserved.

Lab 3: Lingering Object removal using repadmin During this lab, you will remove lingering objects from the environment using repadmin

/removelingeringobjects.

Estimated time to complete this lab: 30 minutes

Before You Begin

To complete this lab:

Complete lessons 1-4

What You Will Learn

After completing this lab you will be able to remove lingering objects using repadmin.

After completing the exercises you will be able to:

Exercise 1 enabling objective.

Exercise 2 enabling objective.

Scenario

You are assisting a customer who is having issues with…

Exercise 1: <Problem Solving Exercise Title> <Briefly describe the goal of the exercise>

Scenario

You have completed recovering files from a back up and now need to restore the files.

Tasks

<Define starting conditions, including virtual machines and lab files required>.

1. <Task>.

a. <Step>.

b. <Step>.

Setting | Parameter Value

Item 1

Item 2

2. <Task>.

Page 93: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 87

Sample solution

Your result should look something like the Sample in <Lab Title>, <Exercise Title> in the Lab

Manual that accompanies this course.

For step by step instructions, see <Lab Title>, <Exercise Title> in the Lab Manual that

accompanies this course.

Review

1. <Question>

Answer

2. <Question>

Answer

Exercise 2: <Simulation Exercise Title> <Briefly describe the goal of the exercise>

Scenario

You have received email from your manager requesting a maintenance action.

Tasks

1. Read Email from your manager explaining the situation.

<Add email text here>

2. Review supporting documents in <local path>:

a. Company organization chart.

b. Company ____ data.

c. Report on problems with the ____ system.

Page 94: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab 3: Lingering Object removal using repadmin

88 © 2011 Microsoft Corporation. All rights reserved.

3. Open the VM containing the company system and resolve the issues.

Sample solution

Your result should look something like the Sample in <Lab Title>, <Exercise Title> in the Lab

Manual that accompanies this course.

For step by step instructions, see <Lab Title>, <Exercise Title> in the Lab Manual that

accompanies this course.

Review

1. <Question>

Answer

2. <Question>

Answer

Page 95: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 89

Lab 4: Lingering Object removal using ldp and repldiag

During this lab, you will remove a single lingering object using ldp. You will then remove the

remaining lingering objects using repldiag.

Estimated time to complete this lab: 30 minutes

Before You Begin

To complete this lab:

Complete <list lesson(s) etc.>.

What You Will Learn

After completing this lab, you will be able to <Lab terminal objective>.

Exercise 1 enabling objective.

Exercise 2 enabling objective.

Scenario

You are assisting a customer who is having issues with…

Page 96: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 Lab 5: Abandoned Object and Abandoned Deleted object remediation

90 © 2011 Microsoft Corporation. All rights reserved.

Lab 5: Abandoned Object and Abandoned Deleted object remediation

During this lab, you will identify and remove an abandoned object. You will then remediate

and abandoned deleted object scenario.

Estimated time to complete this lab: 30 minutes

Before You Begin

To complete this lab:

Complete 1-4

Configure | verify your lab environment:

o Virtual machines <VM name(s)> installed and configured.

o <Application name> installed and configured.

o <List and link to specific lab files if needed>.

What You Will Learn

After completing this lab, you will be able to <Lab terminal objective>.

Exercise 1 enabling objective.

Exercise 2 enabling objective.

Scenario

You are assisting a customer who is having issues with…

Page 97: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 91

Lab 6: Lingering Link identification and cleanup During this lab, you will identify all lingering-linked values in the environment. You will them

remove them in order to ensure group membership consistency.

Estimated time to complete this lab: 45 minutes

Before You Begin

To complete this lab:

Complete lessons 1-4

Configure | verify your lab environment:

o Virtual machines <VM name(s)> installed and configured.

o <Application name> installed and configured.

o <List and link to specific lab files if needed>.

What You Will Learn

After completing this lab, you will be able to <Lab terminal objective>.

Exercise 1 enabling objective.

Exercise 2 enabling objective.

Scenario

You are assisting a customer who is having issues with…

Page 98: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 10.0 Presentation Slides

92 © 2011 Microsoft Corporation. All rights reserved.

10.0 Presentation Slides

Page 99: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 93

Page 100: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 10.0 Presentation Slides

94 © 2011 Microsoft Corporation. All rights reserved.

Page 101: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 95

Page 102: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 10.0 Presentation Slides

96 © 2011 Microsoft Corporation. All rights reserved.

Page 103: Lingering Objects

DRAFT V9.3 Active Directory Replication Troubleshooting

[email protected] Microsoft Corporation 97

Page 104: Lingering Objects

Troubleshooting Lingering Objects DRAFT V9.3 10.0 Presentation Slides

98 © 2011 Microsoft Corporation. All rights reserved.