Lingering Objects
-
Upload
servergeeks -
Category
Documents
-
view
72 -
download
2
description
Transcript of Lingering Objects
Active Directory Replication Troubleshooting
Troubleshooting Lingering Objects
DRAFT V9.3 Released: October 17, 2011
About the Authors
Author: Justin Turner
Bio:
Justin is a Sr. Support Escalation Engineer with the Directory Services group based in Irving Texas with over 10 years of support and Active Directory experience. Justin has created or contributed too many training courses and KB articles for the Microsoft Knowledgebase.
Project Lead: Justin Turner
Bio:
Table of Contents
1.0 TAP .......................................................................................................................................................... 1
1.1 Topic ................................................................................................................................................................ 1
1.2 Audience .......................................................................................................................................................... 1
1.3 Purpose ........................................................................................................................................................... 1
1.4 Format ............................................................................................................................................................. 1
2.0 Problem ................................................................................................................................................... 2
2.1 The Problem .................................................................................................................................................... 2
2.2 Potential Challenges ........................................................................................................................................ 2
2.3 Learner’s Needs ............................................................................................................................................... 2
2.4 Instructor’s Needs ........................................................................................................................................... 3
3.0 Learning Expectations ............................................................................................................................. 4
3.1 Learning Goals and Objectives ........................................................................................................................ 4
3.2 Lesson Components ........................................................................................................................................ 4
3.3 Resources ........................................................................................................................................................ 5
4.0 Learning Activities ................................................................................................................................... 6
Focus on goals ....................................................................................................................................................... 6
Connect to prior knowledge .................................................................................................................................. 6
Gain and integrate content knowledge ................................................................................................................. 6
Take action and monitor learning progress .......................................................................................................... 6
Synthesize and evaluation ..................................................................................................................................... 6
Extend and transfer ............................................................................................................................................... 7
5.0 Assessment ............................................................................................................................................. 8
5.1 Assessment Objectives .................................................................................................................................... 8
5.2 Post-course exam ............................................................................................................................................ 9
5.3 Post-course exam Answer Key ...................................................................................................................... 13
5.4 Performance Assessment .............................................................................................................................. 14
5.5 Performance Assessment Rubric ................................................................................................................... 15
6.0 Evaluation ............................................................................................................................................. 16
6.1 Survey Questions ........................................................................................................................................... 16
7.0 Timeline ................................................................................................................................................. 18
8.0 Job Aid ................................................................................................................................................... 20
8.1 Instructor Job Aid .................................................................................................................................. 21
Course Parameters .............................................................................................................................................. 21
Note to Trainers .................................................................................................................................................. 22
Obtaining Access to Virtual Machines ................................................................................................................. 23
Activities .............................................................................................................................................................. 24
8.2 Learner Job Aid .................................................................................................................................................. 25
Lingering Object Terminology ............................................................................................................................. 25
Tombstone Lifetime Default Values .................................................................................................................... 26
Replication Consistency Settings ......................................................................................................................... 26
Troubleshooting Overview .................................................................................................................................. 29
Repadmin /removelingeringobjects Quick Reference ........................................................................................ 29
Un-hosting a partition ......................................................................................................................................... 30
Manually adding a replication connection using repadmin.exe ......................................................................... 31
Repldiag quick reference ..................................................................................................................................... 32
9.0 Course Workbook ................................................................................................................................. 36
Document Conventions ........................................................................................................................................... 36
Program Code and Commands............................................................................................................................ 36
Notes ................................................................................................................................................................... 37
Tables and Figures ............................................................................................................................................... 37
Course Document and Slide Numbering ............................................................................................................. 37
Lesson 1: Lingering Objects Fundamentals ................................................................................................. 39
What You Will Learn ............................................................................................................................................ 39
Terminology associated with Lingering Object issues ............................................................................................. 39
Lingering Objects ................................................................................................................................................. 39
Tombstone .......................................................................................................................................................... 39
Tombstone Lifetime (TSL).................................................................................................................................... 39
Strict and Loose Replication Constancy .............................................................................................................. 42
Loose Replication Consistency ............................................................................................................................ 42
Strict Replication Consistency ............................................................................................................................. 43
Abandoned object ............................................................................................................................................... 46
Abandoned delete ............................................................................................................................................... 46
Lesson 2: Symptoms and Cause .................................................................................................................. 48
What You Will Learn ............................................................................................................................................ 48
Symptoms of Lingering Objects ............................................................................................................................... 48
Detection of Domain Controllers That Have Not Replicated in the Tombstone Lifetime ................................... 48
Replication Errors Caused by Lingering Objects .................................................................................................. 50
Cause of Lingering Objects ...................................................................................................................................... 51
How lingering objects occur ................................................................................................................................ 51
Five Causes of Lingering Objects ......................................................................................................................... 51
Lingering Object Prevention ................................................................................................................................ 53
Lesson 3: Identification and Classification .................................................................................................. 54
What You Will Learn ............................................................................................................................................ 54
Create a replication health report ........................................................................................................................... 54
Try This: Generate an AD Replication report using repadmin ............................................................................. 55
Use AD Replication report and repadmin to determine the scope of the problem ................................................ 55
Lesson 4: Lingering Object Removal ........................................................................................................... 57
What You Will Learn ............................................................................................................................................ 57
Methods to Remove Lingering Objects ................................................................................................................... 57
Removing Lingering Objects with Repadmin....................................................................................................... 57
Events Associated with Lingering Object Removal.............................................................................................. 58
Details of Repadmin’s Lingering Object Removal Mechanism ............................................................................ 59
Remove Lingering Objects Using Repldiag .......................................................................................................... 59
Remove Lingering Objects Using Replfix ............................................................................................................. 63
Remove Lingering Object using LDP or Script ..................................................................................................... 63
Remove Lingering Objects by partition re-host operation .................................................................................. 63
Lesson 5: Real World Application ............................................................................................................... 67
What You Will Learn ............................................................................................................................................ 67
Determining What to Do with a Lingering Object ............................................................................................... 67
10.0 Lab Guide ............................................................................................................................................ 69
Lab Sessions ................................................................................................................................................ 70
Setting Up Your Lab Environment ........................................................................................................................... 70
Hardware ............................................................................................................................................................. 71
Software .............................................................................................................................................................. 71
Network Layout ................................................................................................................................................... 72
Computer Names and IP Addresses .................................................................................................................... 72
Configuring Your Computer(s)............................................................................................................................. 73
Accounts and Group Membership ...................................................................................................................... 74
Domain Membership........................................................................................................................................... 74
Shares on Instructor Computer(s) ....................................................................................................................... 75
Using the Keyboard and Mouse in a Virtual Machine ............................................................................................. 75
Using the Keyboard ............................................................................................................................................. 75
Using the Mouse ................................................................................................................................................. 76
Lab 1: Exploring Lingering Object Fundamentals ........................................................................................ 78
Configuring Your Computer(s) ................................................................................................................................. 78
Configuring Your Virtual Machine Environment ................................................................................................. 78
Accounts and Group Membership ...................................................................................................................... 78
Domain Membership........................................................................................................................................... 79
Exercise 1: Determine Tombstone Lifetime Setting ................................................................................................ 79
Exercise 2: Determine forest and DC replication consistency settings ................................................................... 81
Lab 2: Lingering Object Diagnosis and Documentation .............................................................................. 83
Exercise 1: Lingering Object Diagnosis .................................................................................................................... 83
Exercise 2: Lingering Object Documentation .......................................................................................................... 84
Lab 3: Lingering Object removal using repadmin ....................................................................................... 86
Exercise 1: <Problem Solving Exercise Title> ........................................................................................................... 86
Exercise 2: <Simulation Exercise Title> .................................................................................................................... 87
Lab 4: Lingering Object removal using ldp and repldiag ............................................................................. 89
Lab 5: Abandoned Object and Abandoned Deleted object remediation ................................................... 90
Lab 6: Lingering Link identification and cleanup ......................................................................................... 91
10.0 Presentation Slides.............................................................................................................................. 92
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 1
1.0 TAP This will be a half-day course covering Troubleshooting Lingering Objects. The proposed
solution will consist of lecture, classroom discussion, case study and a hands-on laboratory
environment using virtualized domain controllers on a Hyper-V server.
Client: Stacy Raynor | Support Escalation Manager | Microsoft Corporation
Problem: High case TMPI and escalation rate for AD Replication (lingering object) issues
Solution: 6 hour training module
1.1 Topic
Troubleshooting Lingering Objects: Symptom, Cause and Resolution
1.2 Audience
Support Engineers at Microsoft Corporation
1.3 Purpose
The purpose of this workshop is to equip Microsoft Support Engineers with the necessary
background knowledge and skills required to troubleshoot and resolve Active Directory
Replication failures involving Lingering Objects.
1.4 Format
Instructor Led in classroom and remotely through Live Meeting consisting of:
Lecture
Classroom discussion
Case study
Lab
Assessment
Troubleshooting Lingering Objects DRAFT V9.3 2.0 Problem
2 © 2011 Microsoft Corporation. All rights reserved.
2.0 Problem Analysis of over 3,000 cases revealed that the Total Minutes per Incident (TMPI) for Active
Directory replication issues involving “lingering objects” is more than twice the TMPI average
of standard Active Directory replication cases. Interviews of SMEs and other engineers who
work these issues revealed the following as likely contributors to the higher TMPI metric:
Lack of consolidated documentation
Complicated terminology, troubleshooting and remediation methods
2.1 The Problem
There is one technology area within Active Directory replication that has a higher than
normal TMPI statistic: Lingering Objects. Cases that fall into this area are escalated to the
next level of engineers frequently and take longer to resolve. Engineers will escalate cases for
a number of reasons, one of them being that they do not feel they have the skills to resolve
the problem. While there are a number of factors that can increase a case’s TMPI and
escalation rate, case analysis and engineer interviews reveal that targeted training is the right
approach for this particular area. A targeted 3-5 course module should be sufficient.
2.2 Potential Challenges
Active Directory (AD) Replication is a somewhat broad support topic and the particular
issues that occur within that support topic can vary greatly. Training on such a broad topic in
the past is usually conducted over the course of several days. Targeted, in-depth training on
the more complicated scenarios is preferred over the standard, which is typically broad in
scope with little technical depth. Additionally, support for Microsoft is handled world-wide so
this solution would need to consider options available for remote delivery and/or some type
of self-study component.
Challenges that we may have to deal with:
Consolidation of existing resources
Creation of a comprehensive lab environment in Hyper-V
Course length and modality
2.3 Learner’s Needs
Interviews with SMEs and many engineers that routinely work these issues revealed the
following needs:
Consolidated documentation
o Too many sources of information exist
o “I have over 30 articles to look through when working these issues”
Updated documentation (there are several scenarios un-accounted for in existing
documentation)
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 3
o Repldiag was created several years ago to make lingering object cleanup faster and
easier. Case data and SME interviews suggest that this tool is rarely used.
o “The SMEs ask if I’ve already tried X. How would I know to try something when it’s
not documented?”
Terminology is well defined and easy to understand
o “There are a lot of different terms used when SMEs discuss lingering objects. The
terminology is difficult to grasp. How can I understand your action plan if I don’t
know what you’re saying?”
Practice performing the different clean-up procedures.
o Lab materials that support the course (Hands-on experience with analysis and
resolution steps)
To be able to understand the full scope of a lingering object problem in a large environment
o “I understand how to fix one or two DCs, but it’s a little scary when the customer has
hundreds of servers and most of them have problems.”
To be able to understand which method to use
o “There are five or more methods that do the same thing. Which one should I use?”
2.4 Instructor’s Needs
Supporting materials:
o Documentation
o Visual Aids
o Well defined lab materials
o Available Resources
Troubleshooting Lingering Objects DRAFT V9.3 3.0 Learning Expectations
4 © 2011 Microsoft Corporation. All rights reserved.
3.0 Learning Expectations
3.1 Learning Goals and Objectives
1.0 To understand the cause, identify the symptoms, and identify ways to resolve lingering
object issues
1.1 The learner will be able summarize seven terms commonly used in lingering object
scenarios.
1.2 The learner will be able to explain three ways in which lingering objects are created.
1.3 The learner will be able to list four symptoms of lingering objects.
1.4 The learner will be able to identify the currently configured tombstone lifetime and
replication consistency settings in a lab environment.
2.0 To be able to explain how an Active Directory Administrator can avoid lingering objects in
the future
2.1 The learner will be able to list at least three methods to prevent lingering objects
3.0 To be able to accurately determine the scope of a lingering object problem
3.1 The learner will be able to use repadmin.exe to generate diagnostic data for analysis
3.2 Given diagnostic data, the learner will be able to identify the scope of the problem by
listing all partitions and all servers containing lingering objects.
4.0 To be able to document which method to use to resolve the issue and why
4.1 Given diagnostic data, the learner will be able to create a detailed action plan that will
remove the lingering objects in all partitions on all servers.
4.2 Given five different scenarios, the leaner will be able to recommend the correct
method to remove lingering objects.
4.3 Given a subpar action plan, the learner will be able to recommend changes that will
result in a better solution
5.0 To be able to apply that knowledge in a lab environment and resolve a lingering object
scenario
5.1 The learner will be able to execute the steps in an action plan in order to remove
lingering objects
5.2 The learner will be able to remove lingering objects using five different methods.
3.2 Lesson Components
The course will consist of PowerPoint slides, supporting documentation in Microsoft Word,
and a laboratory environment where the methods and procedures can be practiced on
virtualized domain controllers running on a Windows Server 2008 R2 Hyper-V server. The
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 5
course workbook will contain all necessary supporting documentation and will include real-
world examples of actual cases in a "Did you know?" format.
3.3 Resources
The instructor and students will have pre-requisite knowledge of Active Directory
replication troubleshooting
The instructor and students will have a computer running Windows 7 with Microsoft Office
2010 and remote desktop access to a Server running Windows Server 2008 R2 with Hyper-
V.
Hyper-V will contain the required virtualized domain controllers.
The classroom will have a project, screen, and whiteboard
Troubleshooting Lingering Objects DRAFT V9.3 4.0 Learning Activities
6 © 2011 Microsoft Corporation. All rights reserved.
4.0 Learning Activities
Focus on goals
Each lesson:
Begins with an overview and explanation of the goals of the lesson
Instructor will ask questions to generate curiosity and judge prior knowledge
Connect to prior knowledge
Classroom discussion
Instructor will facilitate discussion of student's prior knowledge
Gain and integrate content knowledge
Case study
Present problems and demonstrate how to solve, explicitly stating the strategies that
were used.
Real-world examples
Present new information in context in which it will be used
Lecture with slides, workbook and hands-on lab
Present information through multiple modes of representation
Allow learners to revisit information as needed
Provide adequate resources
Take action and monitor learning progress
Hands-on lab
Provide support and coaching as needed when learners are performing tasks
Ask learners to demonstrate skill; provide corrective feedback
Synthesize and evaluation
Short-answer, matching, multiple choice, and free recall format exam
Posttest on knowledge
Performance based assessment
Have leaners demonstrate procedure or skill
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 7
Have learners demonstrate their own summaries
Case study
Present case studies, role lays, or simulations in which learners demonstrate skills,
knowledge, attitudes
Extend and transfer
Hands-on labs
Provide practice in a variety of situations
Gradually remove prompts and cues
Provide opportunity to apply skills in realistic contexts
Workbook and Quick-reference handouts "Cube-note"
Provide job aids
Provide access to additional information on the topic
Troubleshooting Lingering Objects DRAFT V9.3 5.0 Assessment
8 © 2011 Microsoft Corporation. All rights reserved.
5.0 Assessment There are two different assessments: One is accessible via an Intranet web page and consists
of a short-answer, matching, multiple choice, and free recall format exam. The other
assessment is a performance-based lab assessment where the student is presented with a
common lingering object scenario and has to document the issue, action plan and perform the
procedure to correctly remove the lingering objects.
5.1 Assessment Objectives
1.1 The learner will be able identify seven terms commonly used in lingering object
scenarios and match them to the corresponding definition. (exam)
1.2 The learner will be able to explain three ways in which lingering objects are created.
(Performance assessment)
1.3 The learner will be able to identify four symptoms of lingering objects. (exam)
1.4 The learner will be able to identify the currently configured tombstone lifetime and
replication consistency settings in a lab environment. (Performance assessment)
2.0 To be able to explain how an Active Directory Administrator can avoid lingering objects in
the future
2.1 The learner will be able to list at least three methods to prevent lingering objects
(exam)
3.0 To be able to accurately determine the scope of a lingering object problem
3.1 The learner will be able to use repadmin.exe to generate diagnostic data for analysis
(exam and Performance assessment)
3.2 Given diagnostic data, the learner will be able to identify the scope of the problem by
listing all partitions and all servers containing lingering objects. (exam and Performance
assessment)
4.0 To be able to document which method to use to resolve the issue and why
4.1 Given diagnostic data, the learner will be able to create a detailed action plan that will
remove the lingering objects in all partitions on all servers. (exam and Performance
assessment)
4.2 Given five different scenarios, the leaner will be able to recommend the correct
method to remove lingering objects. (exam and performance assessment)
4.3 Given a subpar action plan, the learner will be able to recommend changes that will
result in a better solution (performance assessment)
5.0 To be able to apply that knowledge in a lab environment and resolve a lingering object
scenario
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 9
5.1 The learner will be able to execute the steps in an action plan in order to remove
lingering objects (performance assessment)
5.2 The learner will be able to remove lingering objects using five different methods.
(Performance assessment)
5.2 Post-course exam
Multiple Choice: For each of the following questions, circle the letter of the answer that best
answers the question. (5 points each)
1. Which of the following commands would generate a forest-wide replication status report to be used to aid in lingering object analysis? [Objective 3.1]
A. repadmin /replsum /xls >repl.xls B. repadmin /replsum /verbose >repl.xml C. repadmin /showrepl * /csv >repl.csv D. repadmin /showrepl /verbose >repl.txt E. ldp | removelingeringobjects F. A and D G. All of the above
2. Which of the following lingering object removal methods automates the removal of lingering objects?
[Objective 4.2]
A. repadmin /unhost B. repadmin /removelingeringobjects C. repadmin /rehost D. repldiag /removelingeringobjects E. ldp removelingeringobjects primitive F. replfix G. None of the above
3. Which of the following lingering object removal methods will remove objects on Windows 2000 -
Windows 2008 R2 and will remove abandoned objects? [Objective 4.2]
A. repadmin /unhost B. repadmin /removelingeringobjects C. repadmin /rehost D. repldiag /removelingeringobjects E. ldp removelingeringobjects primitive F. replfix G. None of the above
4. Which of the following lingering object removal methods allow you to review which objects will be removed prior to actually removing the objects? [Objective 4.2]
A. repadmin /unhost B. repadmin /removelingeringobjects C. repadmin /rehost D. repldiag /removelingeringobjects E. ldp removelingeringobjects primitive F. replfix
Troubleshooting Lingering Objects DRAFT V9.3 5.0 Assessment
10 © 2011 Microsoft Corporation. All rights reserved.
G. B and F H. D and E
True or False: For each statement, circle True or False. (2 points each) [Objective 1.3]
True False 5. Replication status 8606 indicates that lingering objects are present on the source DC in a replication report.
True False 6. Event ID 1988 indicates that the source DC contains one or more lingering objects.
True False 7. Replication status 8453 indicates that lingering objects are present on the destination DC.
True False 8. Event ID 1388 indicates a lingering object was purged from the database.
True False 9. Event ID 1945 indicates that a lingering object was detected after running repadmin /removelingeringobjects.
True False 10. Abandoned objects can be removed using repadmin /removelingeringobjects.
Fill in the Blank and Matching: Into each sentence below, copy a term from the word bank that
correctly completes the sentence. (5 points each) [objective 1.1]
Lingering Links Lingering Object Tombstone
Abandoned Object Loose Replication
Consistency
Tombstone
Lifetime
Abandoned Delete Strict Replication
Consistency
11. The length of time that a deleted object will remain in the database is referred to as _______.
12. A _________ is an object that is present on one replica, but has been deleted and garbage collected
on another replica.
13. A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale
references are referred to as ___________.
14. An object that has been deleted but not yet garbage collected. _________
15. An object created on one DC that never got replicated to other DCs hosting a writable copy of the NC
but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes
offline prior to replicating the originating write to other DCs that contain a writable copy of the
partition. _________
16. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it
does not have, the entire object is replicated to the target for the sake of replication consistency. This
undesirable behavior causes a lingering object to be “reanimated.” _________
17. An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC
for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that
originated the object deletion goes offline prior to replicating the change to other DCs hosting a
writable copy of the partition. ____________
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 11
18. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it
does not have, replication is blocked with the source DC for the partition where the lingering object
was detected. __________
19. Essay Question: List three or more methods to prevent lingering objects (8 points) (objective 2.1)
Use Figure 1 Replication Status to answer the remaining questions.
Figure 1 Replication Status
20. Essay Question: Use Figure 1 Replication Status, document every DC containing lingering objects
and for which partition. (10 points) (objective 3.2)
Troubleshooting Lingering Objects DRAFT V9.3 5.0 Assessment
12 © 2011 Microsoft Corporation. All rights reserved.
21. Essay Question: Using Figure 1 Replication Status and the following information, provide the exact
command line syntax to log all lingering objects on DC 5thWardCorpDC to the event log, and the
syntax to remove those lingering objects. (10 points) (objective 4.1)
Repadmin /removelingeringobjects <Dest_DSA_LIST> <Source DSA GUID> <NC> [/ADVISORY_MODE] The following DCs host writable copies of the partition in question: Dallas\DALCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 87ccb4f8-1057-4cfa-aed6-79b5626db9fd DC invocationID: 56f7cb84-0a67-43c1-93de-9d01f53e02c5 Dallas\NYCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 4009aef6-b279-43d2-82f6-4298f02505e8 DC invocationID: a29c83ab-5dea-4829-bbbf-1343f037098d Liverpool\LONCONTOSODC DC Options: IS_GC Site Options: (none) DC object GUID: a29bbfda-8425-4cb9-9c66-8e07d505a5c6 DC invocationID: d58a6322-6a28-4708-82d3-53b7dcc13c1a Liverpool\LONEMEADC DC Options: IS_GC Site Options: (none) DC object GUID: ba9bcfb2-7445-2cd9-8c66-9b27d534a4b3 DC invocationID: e38b6355-fb31-3785-71b1-42c6ddc23f8e Houston\5THWARDCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 DC invocationID: e0cb69c0-5d24-4254-b830-99b0c9b4da1f
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 13
5.3 Post-course exam Answer Key
1. C
2. D
3. E
4. G
5. True
6. True
7. False
8. False
9. True
10. False
11. Tombstone Lifetime
12. Lingering Object
13. Lingering Link
14. Tombstone
15. Abandoned Object
16. Loose Replication Consistency
17. Abandoned Delete
18. Strict Replication Consistency
19. At least 3 of the following:
o Resolve replication failures within TSL
o Ensure Strict Replication Consistency is enabled
o Ensure large jumps in system time are blocked via registry key or policy
o Don't remove replication quarantine with "allowDivergent" setting without
removing LOs first
o Don't restore system backups that are near TSL number of days old
o Don't bring DCs back online that haven't replicated within TSL
20. LONCONTOSODC: DomainDNSZones, Configuration
5THWARDCORPDC: Configuration
DALCORPDC: Configuration
FOURTHDC1: Configuration, ForestDNSZones
NYCORPDC: Configuration
CONTOSOROOTDC1: Configuration
FOURTHDC2: Configuration
21.
Repadmin /removelingeringobjects 5thwardCorpDC ba9bcfb2-7445-2cd9-8c66-9b27d534a4b3 cn=configuration,dc=contoso,dc=com /advisory_mode
Troubleshooting Lingering Objects DRAFT V9.3 5.0 Assessment
14 © 2011 Microsoft Corporation. All rights reserved.
Repadmin /removelingeringobjects 5thwardCorpDC ba9bcfb2-7445-2cd9-8c66-
9b27d534a4b3 cn=configuration,dc=contoso,dc=com
5.4 Performance Assessment
Students take their performance assessment in the hands-on lab environment. The
performance assessment is a culmination of all prior lab tasks without the benefit of step-by-
step guidance. The lab environment is broken via several scripts. After the scripts run, both
lingering objects and abandoned objects are present. The students receive a handout with
intentionally vague problem descriptions. They are instructed to document the issue
thoroughly and then resolve the problems. Good documentation consists of symptoms,
cause, and resolution. The symptoms section should contain a list of all "problematic
objects." The resolution section should have a thoroughly documented action plan. Here is
the text they are prompted with:
You are the consultant for Adatum Corporation. Please help resolve the following
problems in our environment.
Changes are not propagated amongst DCs for the Adatum domain.
Unable to create the following user account in the West domain: Mike Miller
Ann Wallace's account in the East domain does not show up on any other domain's
GC
Users that send email to the CorpVP mail-enabled universal group receive NDRs on
occasion. Additionally, our Exchange 2010 mailbox server cannot generate an
Offline Address Book. This worked on our Exchange 2007 mailbox server.
Please ensure that you document each problem thoroughly. This documentation should
include forest and DC environment settings (tombstone lifetime and replication
consistency), symptom, cause and resolution sections. The symptoms section should
contain a list of all "problematic objects." The resolution section should have a thoroughly
documented action plan. Implement your action plan after documenting the issue.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 15
5.5 Performance Assessment Rubric
Hands-on Assessment Rubric: Troubleshooting Lingering Objects
Student Name: _____________________________________
Assessment Criteria Max.
Points Exceptional (all points) Average (65-85%) Poor (0 - 65%) Comments &
Points Earned
Documentation (objectives 1.4, 3.1, 3.2, 4.1, 4.2)
10 Symptom, cause and resolution sections
The symptoms section contains a list of all objects
The resolution section has a thoroughly documented action plan
Symptom, cause and resolution sections are mostly documented
The symptoms section contains a partial list of all objects
The action plan is missing one to two steps
Symptom, cause and resolution sections is inadequate
Less than 25% of all objects are listed
The action plan will not resolve the issue or will make things worse
AD Replication and Lingering object cleanup (5.1, 5.2)
25 All lingering objects are removed from the environment
AD Replication is successful
Most (greater than 75%) of lingering objects are removed.
Less than 25% of lingering objects are removed
AD Replication is not successful
Abandoned object cleanup (5.1, 5.2)
25 Abandoned object is no longer present on any DC
new object is created in its place
Abandoned object is no longer present on most DCs (greater than 75%)
Abandoned object is still present on most DCs
Abandoned delete resolution (5.1, 5.2)
25 Object completely removed from the environment
Object mostly removed from the environment
Object is still present on most DCs in the environment
Lingering Link cleanup (5.1, 5.2)
15 CorpVP group contains correct group membership on all DCs
Group still has the same objectSID
CorpVP group contains correct group membership on all DCs
Group does not have the same ObjectSID
CorpVP group has inconsistent group membership on most DCs
TOTAL: 100
Troubleshooting Lingering Objects DRAFT V9.3 6.0 Evaluation
16 © 2011 Microsoft Corporation. All rights reserved.
6.0 Evaluation Following the conclusion of the course, the students are emailed a link to a survey to take
online.
6.1 Survey Questions
# Question
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
1 I was provided with the information I needed
(logistics, pre-work) for the training in a
timely manner.
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
2 The classroom setup and hardware (if
supplied) functioned appropriately to
support face-to-face learning.
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
3 The instructor was knowledgeable about the
subject matter.
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
4 The instructor's presentation skills helped
me better understand the content.
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
5 The instructor consistently linked the course
content to Microsoft’s business and/or my
role.
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
6 The length of the course was appropriate Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
7 Overall, I was satisfied with this course. Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
8 This course builds skills improving how I sell,
market, and/or provide services to our
customers and partners.
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
9 This course was a valuable use of my time. Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 17
10 I would recommend this course. Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
11 The messaging in this course is relevant to
Microsoft's customers and/or partners.
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
12 If not, please provide additional feedback.
13 How soon will you be able to apply this
learning?
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
14 My manager and I have discussed how I will
apply this training to my job.
Strongly
Agree
Agree Neither
Agree
nor
Disagree
Disagree Strongly
Disagree
Don't
Know
15 What are you going to do differently as a
result of this course?
16 What was the most useful portion of this
course? (Please provide specifics, e.g.
instructor effectiveness, content quality,
materials usefulness).
17 What was the least useful portion of this
course? (Please provide specifics, e.g.
instructor effectiveness, content quality,
materials usefulness).
18 Please provide any additional comments (e.g. learning environment, instructor effectiveness, content/materials quality, content level, relevance, application).
Troubleshooting Lingering Objects DRAFT V9.3 7.0 Timeline
18 © 2011 Microsoft Corporation. All rights reserved.
7.0 Timeline The following proposed timeline should allow for sufficient coverage of the course material.
Time Objectives Activities / Training Methods
Materials
9:00 AM
15 minutes
Welcome and Instructor Introduction
1.2
Intro and Classroom discussion
Slide 1: Course Title and
Instructor Name
9:15 AM
20 minutes
Lingering Object Fundamentals
1.1
Lecture and discussion
Lesson 1 Slides
9:35 AM
15 minutes
Exploring Lingering Object Fundamentals
1.4
Lab 1exercise Lab 1 guide and lab environment
9:50 AM
20 minutes
Symptoms and Cause
1.2, 1.3, 2.1
Lecture and discussion
Lesson 2 Slides
Provide real-word scenarios
10:10 AM
20 minutes
Identification and Classification
3.1, 3.2
Lecture and discussion
Lesson 3 Slides
Show prior case action plans
10:30 AM
10 minutes
Break
10:40 AM
45 minutes
Lingering Object Diagnosis and Documentation
3.1, 3.2, 4.1
Lab 2 exercise Lab 2 guide and lab environment
11:25 AM
20 minutes
Lingering Object Removal
5.1, 5.2
Lecture and discussion
Lesson 4 Slides
11:45 AM
60 minutes
Lunch
1:00 PM
90 minutes
Lingering Object removal labs
5.1, 5.2
Lab exercise 4 - 6
LabLab documentation, Hyper-V images
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 19
2:30 PM
10 minutes
Break
2:40 PM
10 minutes
Real World Application
4.2, 4.3
Lecture and discussion
Lesson 5 Slides
2:50 PM
30 Minutes
Real-world case study
4.2, 4.3
Case Study Case data in instructor share
Case Details, Diagnostic Data
Present the high-level symptoms.
What data do you want to see?
Show the data
What is the action plan?
3:20 PM
10 minutes
Question Time Ask if there are any questions
3:30 PM
30 minutes
Assessment Post-course test Share assessment URL on-screen
4:00 PM
10 minutes
Break
4:10 PM
60 minutes
Performance assessment
Lab-based assessment
VMAS connection instructions for post-course performance assesment
10 minutes Summary and questions
1.1 - 5.2
Course Summary and wrap-up
Slide
Troubleshooting Lingering Objects DRAFT V9.3 8.0 Job Aid
20 © 2011 Microsoft Corporation. All rights reserved.
8.0 Job Aid
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 21
8.1 Instructor Job Aid
Course Parameters
Course title Troubleshooting Lingering Objects
Course Length 6 hours (1 day)
Course Objectives At the completion of this workshop, the engineer shall be able to:
1. Explain how a user's group membership is stored in Active Directory
2. Explain what happens during a user deletion
3. Understand why a special procedure is needed to restore users along with
their group membership.
4. Explain the three methods of recovery after deletion
5. Identify recommendations and considerations for a better recovery
experience
6. Perform the most common (and preferred) method of recovery for our
customers
Target Audience Microsoft Support Engineers (Platforms, Directory Services)
Prerequisites Trainee:
1. Knowledge of Active Directory replication
2. Familiarity with Active Directory concepts and terminology
3. Experience with Hyper-v for the lab session
Instructor:
1. Real world experience with Active Directory replication and Lingering
Object troubleshooting procedures
2. Hyper-V user experience for demonstration session
3. PowerPoint user experience
Room arrangement Classroom setting
Materials/equipment PowerPoint setup, whiteboard and markers, Computer for
demonstration, and one computer for each workshop participant.
On the computers: Microsoft Windows 7, Microsoft Office 2010,
Intranet access, PowerPoint presentation, and supporting reference
documentation
Evaluation/
Assignments
Learning exercises for participants and online Instructor/classroom
evaluation form
Instructor Justin Turner is a Sr. Support Escalation Engineer on the Microsoft
Platforms Directory Services Support team where he obtained his first-
hand knowledge of the material. He has been with Microsoft for over
ten years, and is currently pursuing his MS in Computed Education and
Cognitive Systems degree from the University of North Texas.
Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid
22 © 2011 Microsoft Corporation. All rights reserved.
Note to Trainers
Checklist of Supplies
Print out slides with “notes pages.” The notes pages provide the necessary material to help explain
the contents of each slide.
Alternatively, you can have the students copy the course materials to their computer and print out
the slides to a new Microsoft OneNote notebook.
The student lab guide is stored electronically on the hyper-v image: DC1
Room Arrangement
Standard Microsoft classroom configuration: Classroom style with whiteboard and projector screen
at the front of the room
Handouts / Visual Aids
Print out one copy of the slide deck in "Handouts" format for each student (or print to OneNote).
Course workbook and lab guide are available on the Instructor computer.
Lab Computer setup
Microsoft Windows 7
Office 2010
Connection to the corporate Intranet
Preparation
Before Class starts:
1. Have PowerPoint slide deck opened up
2. On instructor machine: Launch Hyper-V, and launch DC1s image
3. Ensure classroom has intranet connectivity
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 23
Obtaining Access to Virtual Machines
To access VMs provisioned for your use during this course, perform the following steps:
4. Log onto the physical computer using your Corpnet credentials.
5. Access the VMAS server that hosts your VMs using the link provided by your instructor.
6. Open the VMAS menu and select Manage VMAS VMs.
7. Use Manage My VMs to access virtual machines referenced in lab exercises.
Note: For more information, click links in the Documents section on the right to open course documents included in the VM package.
Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid
24 © 2011 Microsoft Corporation. All rights reserved.
Activities
Introduction
Welcome the students to the course. Ask them to share the following:
Name
Role
Time at Microsoft
Something that no one (at work) else knows about them or something unique
Classroom Discussion
After the introduction, lead a discussion to gauge student's prior knowledge. Ask probing
questions like:
What is a lingering object?
Why do I care about removing them from my environment?
What does tombstone lifetime have to do with this?
Who can explain the different between strict and loose replication consistency?
What is an abandoned object? How is that different from a lingering object?
What is a lingering linked value?
Who here has worked a lingering object issue? Were you able to resolve it? How long
did it take?
Who here has used repldiag? What did you think about it?
Real-world examples
Where appropriate, provide examples of actual cases worked. Highlight the successes and
failures (what went right and what went wrong).
Present new information in context in which it will be used
Case Study
The case study within the course includes real diagnostics data from an actual customer case.
The data was scrubbed to remove personally identifiable information (PII). Present the facts
of the case and encourage the students to play the role of engineer. There is an action plan
included in the case study. The action plan is intentionally poor in quality and if implemented
would result in disastrous results. Together come up with the appropriate action plan to
resolve the problem.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 25
Present case studies, role plays, or simulations in which learners demonstrate skills,
knowledge, attitudes
Present problems and demonstrate how to solve, explicitly stating the strategies that
were used.
Lab Activities
Students have access to their lab environment through the VMAS site. Each lab activity
corresponds to a lesson in the course. You may be tempted to do the entire lecture at once
and then all lab activities at the end of the course. It is important not to do this. Please have
the students complete the lab activities along with the appropriate lesson in the course.
Hands-on lab
If unfamiliar with the lab environment and lab material, you should work through each
lab activity at least one time prior to the course
Provide support and coaching as needed when learners are performing tasks
Ask learners to demonstrate skill; provide corrective feedback
8.2 Learner Job Aid
Lingering Object Terminology
Table 1: Lingering Object Terminology
Term Definition
Abandoned delete An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.
Abandoned object
An object created on one DC that never got replicated to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition.
Lingering link
A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale references are referred to as lingering links.
Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid
26 © 2011 Microsoft Corporation. All rights reserved.
Lingering Object
An object that is present on one replica, but has been deleted and garbage collected on another replica.
Loose Replication Consistency With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This undesirable behavior causes a lingering object to be “reanimated.”
Strict Replication Consistency With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, replication is blocked
with the source DC for the partition where the lingering object was detected
Tombstone An object that has been deleted but not yet garbage collected
Tombstone Lifetime (TSL)
The amount of time tombstones are retained in Active Directory before being garbage collected and permanently purged from the database.
Tombstone Lifetime Default Values
Table 2: Default TSL Values
OS Install Path Default TSL
Windows 2000 RTM 60 days
Windows 2003 RTM, 2003 R2 60 days
Windows 2000RTM upgrade to Windows 2003 SP1 60 days
Windows 2003SP1, 2003SP2, 2008, 2008R2 180 days
NT4 upgrade to Windows 2003 SP1 180 days
Replication Consistency Settings
Strict Replication Consistency
Defines how a destination DC behaves if a source DC sends updates to an object that does
not exist in the destination DC’s local copy of Active Directory.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 27
o Destination DCs should see USN for creates before object is modified
o Only modifies for lingering objects arrive for object not on destination DC
o Only destination DC’s enforce strict replication and log events
Destination DCs stop replicating from source DC’s partitions containing LO’s
Lingering objects are quarantined on source DCs where they can be detected
End-to-end replication may be impacted for partitions containing lingering objects
Administrators must remove lingering objects to restore replication
Enabling Strict Replication
Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command
prompt:
For all domain controllers, type:
repadmin /regkey * +strict
For all global catalog servers, type:
repadmin /regkey gc: +strict
You can also enable strict replication by manually setting the Strict Replication Consistency
registry value to 1.
HKLM\System\CurrentControlSet\Services\NTDS\Parameter Strict Replication Consistency
(Reg_DWORD) to 1
1 (enabled): Inbound replication of the specified directory partition from the source is
stopped on the destination.
Warning: Ensure you are prepared to deal with replication failures after enabling strict replication consistency due to the existence of lingering objects.
Loose Replication Consistency
If you enable Loose Replication Consistency, if a destination receives a change to an object
that it does not have, the entire object is replicated to the target for the sake of replication
consistency. This behavior causes a lingering object to be reapplied to all domain controllers
in the replication topology.
Enable Loose Replication
Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command
prompt:
For all domain controllers, type:
repadmin /regkey * -strict
Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid
28 © 2011 Microsoft Corporation. All rights reserved.
For all global catalog servers, type:
repadmin /regkey gc: -strict
You can also enable strict replication by manually setting the Strict Replication Consistency
registry value to 0.
HKLM\System\CurrentControlSet\Services\NTDS\Parameters Value: Strict Replication Consistency Type: (Reg_DWORD) Value Data: 0 0 (disabled): The destination requests the full object from the source domain controller, and the lingering object is revived in the directory as a new object.
Critical: The Loose Replication Consistency setting will cause the undesirable behavior of reanimation of lingering objects.
Default Settings for Strict Replication Consistency
Upgrade Path Default Notes
Windows NT 4.0 Loose
Windows 2000 RTM Root Loose A post-SP2 NTDSA.DLL defaulted to strict replication consistency but was quickly recalled. Windows 2000 Services 1 through 4 all default to loose replication consistency.
Windows NT 4.0 to Windows 2000 Root
Loose
Windows 2000 to Windows Server 2003 SP1
Loose Upgrading a Windows 2000 forest to Windows Server 2003 slipstreamed with SP1 does not enabled strict replication consistency.
Windows Server 2003 RTM Root
Strict DCPROMO creates an operational GUID that causes Windows Server 2003 domain controllers to inherit strict replication mode but is ignored by Windows 2000 domain controllers.
Windows Server 2003 SP1 root
Strict Same as above.
Windows NT 4.0 to Windows Server 2003 root
Strict DCPROMO creates an operational GUID that causes
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 29
Windows Server 2003 domain controllers to inherit strict replication mode but is ignored by Windows 2000 domain controllers.
The default value for the strict replication consistency registry entry is determined by the
conditions under which the domain controller was installed into the forest.
Note: Raising the domain or forest functional level does not change the replication
consistency setting on any domain controller.
More Information: For more information about this topic, see:
http://blogs.technet.com/b/askds/archive/2010/02/15/strict-replication-consistency-myth-versus-reality.aspx
Troubleshooting Overview
Common methods to remove lingering objects include: Repadmin /Removelingeringobjects Replfix Repldiag Manually through LDP or using script Rehost the partition:
Repadmin /rehost (or /unhost and /add) (only if the partition is not-writable on the DC containing lingering objects)
Un-GC (but you don’t really have control over who the DCs sources the partition from) Demote and Promote (DCPromo)
Repadmin /removelingeringobjects Quick Reference
Have the customer run the following command: repadmin /showrepl * /csv >showrepl.csv Once you have this, filter column K for 8606, so that you know exactly which DCs have lingering objects and in which partitions. The DCs in the SourceDC column contain lingering objects. You can use the repadmin /removelingeringobjects command to remove lingering objects. In some cases it may make sense to just rehost the partition with the repadmin /rehost command. In order to use the /removelingeringobjects command you need to know three things:
1. You need to know which dc's contain lingering objects 2. Which partition the lingering object resides in 3. A good reference DC that hosts that partition that does not contain lingering objects
Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid
30 © 2011 Microsoft Corporation. All rights reserved.
Repadmin RLO example usage:
The command is: repadmin /removelingeringobjects LingeringDC ReferenceDC_DSA_GUID Partition Where: LingeringDC: FQDN of DC that has the lingering objects ReferenceDC_DSA_GUID: The DSA GUID of a domain controller that hosts a writeable copy of the partition Partition: The distinguished name of the directory partition where the lingering objects exist So for example: We have a server named DC1.contoso.com that contains lingering objects. We know that the lingering object is in the childdomain.contoso.com partition. We know that DC3.childdomain.contoso.com hosts a writeable copy of the partition and doesn't contain any lingering objects. We need to find the DSA GUID of DC3 is, so we run: repadmin /showrepl DC3.childdomain.contoso.com At the top of the output, locate the DC Object GUID entry. This is the GUID you need to enter in the command for the reference DC. The command would be repadmin /removelingeringobjects DC1.contoso.com 5ed02b33-a6ab-4576-b109-bb688221e6e3 dc=childdomain,dc=contoso,dc=com
-------------------------------------------------------------------------------------------------
Detailed troubleshooting guidance is located here:
2028495 Troubleshooting Active Directory operations that fail with error 8606: Insufficient attributes were given to create an object. http://support.microsoft.com/default.aspx?scid=kb;en-US;2028495
Un-hosting a partition
It is sometimes necessary to remove a partition from the database of a DC temporarily.
Repadmin includes a /rehost option that allows you to do this, but the /unhost option allows
you to exercise more control over the procedure. Take note that /unhost only allows you to
remove a read-only copy of the partition. With the exception of application partitions, you
cannot remove a writable copy of a partition from a DC without using DCPROMO.
Repadmin /?:unhost Remove a specific read-only partition from a GC. [SYNTAX] /unhost DSA <Naming Context> Repadmin /unhost ContosoDC1 dc=corp,dc=contoso,dc=com
Event ID 1659 indicates the status of the un-host operation. Do not re-add the partition until
event ID 1660 is logged in the Directory Services event log. The re-host operation may fail
with error 8339 if you attempt to re-add the partition too soon after the un-host.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 31
Manually adding a replication connection using
repadmin.exe
The add command will create a RepsFrom attribute on the destination domain
controller for the specified naming context and initiate a replication request. During a
normal replication cycle, the destination domain controller will request updates from
the source domain controller. When creating temporary replication links between
replication partners, the process could fail if the KCC starts while you are performing
the procedure. The KCC will delete any replication links for which no corresponding
connection object exists. Since these commands can take a very long time to
complete as they trigger the replication of the corresponding naming context, it is
important to ensure that KCC do not disturb the process. This is where you would use
+DISABLE_NTDSCONN_XLATE which effectively disables KCC's capability to translate
connection objects to replication links.
Disable KCC connection translation so that KCC doesn’t remove our temporary
replication connection:
Repadmin /options ContosoDC1 +disable_ntdsconn_xlate
Then add a replication connection for the configuration partition of the server we
want to source the partition from:
Repadmin /add <Naming Context> <Dest DSA> <Source DSA> [/readonly] [/selsecrets] <Source DSA> The source DSA must be specified by fully qualified computername. repadmin /add cn=configuration,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.
Add a replication connection to the server for the domain partition that we
need to source from (/readonly is specified if the partition is a GC non-writable
partition /selsecrets needs to be specified if the destination DC is an RODC):
repadmin /add dc=emea,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com /readonly One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.
Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid
32 © 2011 Microsoft Corporation. All rights reserved.
If you need to replicate the other way, then just reverse the order of the server names
in the commands.
To begin a normal sync of the partition using the new replication connection:
Repadmin /replicate <Dest_DSA_LIST> <Source DSA_NAME> <Naming Context> [/force] [/async] [/full] [/addref] [/readonly] repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly
To begin a full sync of that partition using the new replication connection:
repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly /full Sync from LONEMEADC.Emea.contoso.com to ContosoDC1.contoso.com completed successfully.
Turn KCC connection translation back on when you no longer need the connection:
Repadmin /options ContosoDC1 -disable_ntdsconn_xlate
Repldiag quick reference
Removing lingering objects from a forest with repldiag is as simple as running repldiag
/removelingeringobjects. However, it is usually best to exercise some control over the
process in larger environments. The option /OverRideReferenceDC allows you to select
which DC is used for cleanup. The option /outputrepadmincommandlinesyntax allows you to
see what a forest-wide cleanup looks like using repadmin.
Repldiag /removelingeringobjects /outputrepadmincommandlinesyntax
This will give you output of corresponding repadmin /removelingeringobjects syntax. It will first select one DC per partition to be used as a reference DC. It will then clean the reference DCs up against all other DCs for the partition(s) it was selected to be used as a reference for. Finally it cleans up all other DCs in the forest with the new “cleaned up” reference DCs as sources. The /outputrepadmincommandlinesyntax option does not actually attempt object cleanup. You would need to leave this option off if you want to execute lingering object cleanup.
Number Complete,Status,Server Name,Naming Context,Reference DC,Duration,Error Code,Error Message repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=forestdnszones,dc=contoso,dc=com
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 33
repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=corp,dc=contoso,dc=com Reference NCs cleaned in 0h:0m:0s. Cleaning everything else against reference NCs. repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com
Troubleshooting Lingering Objects DRAFT V9.3 8.1 Instructor Job Aid
34 © 2011 Microsoft Corporation. All rights reserved.
repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com All NCs cleaned in 0h:0m:0s.
This output can also be viewed in Excel: Copy commands to a text file. Modify the text file to include only the command portion of the output. Then open up the text file in Exel. (space delimited)
More control: /OverRideReferenceDC
This option allows you to specify a DC that you want to be used as a reference DC for the partition specified. In a large distributed environment, take careful consideration when choosing the reference DC. Things to consider when choosing a suitable reference DC: Well connected: Fast WAN link. Performance: Excellent server class hardware: Disk, RAM, CPU and NIC Critical Network Applications / Services do not depend on this DC: Such as an Exchange
facing DC Other DCs don’t report replication failures with reference DC as the source: filter repadmin
/showrepl * /csv ouput, or use the topology report created by repldiag /save.
repldiag /removelingeringobjects /overridedefaultreferencedc:"cn=configuration,dc=contoso,dc=com":nycorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=corp,dc=contoso,dc=com":seacorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=forestdnszones,dc=contoso,dc=com":5thwardcorpdc.corp.contoso.com /outputrepadmincommandlinesyntax Replication topology analyzer. Written by [email protected] Version: 2.0.3397.24022 Command Line Switch: /removelingeringobjects Command Line Switch: /overridedefaultreferencedc:cn=configuration,dc=contoso,dc=com:nycorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=corp,dc=contoso,dc=com:seacorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=forestdnszones,dc=contoso,dc=com:5thwardcorpdc.corp.contoso.com Command Line Switch: /outputrepadmincommandlinesyntax Attempting to override NC cn=configuration,dc=contoso,dc=com with DC nycorpdc.corp.contoso.com... Overriden Attempting to override NC dc=corp,dc=contoso,dc=com with DC seacorpdc.corp.contoso.com... Overriden Attempting to override NC dc=forestdnszones,dc=contoso,dc=com with DC 5thwardcorpdc.corp.contoso.com... Overriden
/UseRobustDCLocation
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 35
Query each and every DC for a list of DCs in forest. Ensures replication instability does not cause any to be missed. We’ve had cases where we clean up lingering objects in the forest but do to an AD topology problem some DCs were not cleaned up. This option is almost always recommended if you want it to do a thorough job.
Troubleshooting Lingering Objects DRAFT V9.3 9.0 Course Workbook
36 © 2011 Microsoft Corporation. All rights reserved.
9.0 Course Workbook
Document Conventions The following conventions are used in the course materials:
Acronyms appear in all uppercase letters.
Path and file names may appear in a combination of uppercase and lowercase letters.
Unless otherwise indicated, paths and file names entered in dialog boxes or at a
command prompt are not case-senstitive
File extensions without a file name appear in all lower-case letters.
Book titles and URLs appear in Italic.
Window, dialog box, menu titles, menu items, and section titles appear in Bold.
Other document conventions are described below.
Program Code and Commands
Program code listings, diagnostic output, entries typed at a command prompt or in scripts or
initialization files, and other text mode content appear in a console font with a grey
background formatted as shown in the following example. Descriptive comments may be
inserted in line with the listing.
d:\%systemroot%>dir /ad
where:
d: is the drive letter where the operating system is installed.
%systemroot% is the folder where the operating system is installed.
Volume in drive C is Main Volume Serial Number is 000A-BCDE Directory of C:\Windows 12/19/2004 11:56 AM <DIR> . 12/19/2004 11:56 AM <DIR> .. 07/07/2003 06:57 AM <DIR> addins 11/17/2004 02:45 PM <DIR> Application Compatibility Scripts 11/17/2004 02:47 PM <DIR> AppPatch 11/17/2004 02:42 PM <DIR> Cache ...
The ellipsis (...) on the last line indicates a partial listing.
The following conventions apply to all commands and program code listings:
Type command statement elements that appear in Bold exactly as they appear in the
example, including quotation marks.
Italic elements in command statements indicate placeholders for variable information.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 37
Braces ({ }) enclose required items as shown by {parameter1, parameter2, “title”} in the
example. Commas separate multiple items. Type quotation marks as shown; do not
type the braces.
Square brackets ([ ]) enclose optional items as shown by [option1 | option2] in the
example. Pipe symbols ( | ) indicate alternate choices. If multiple options are listed,
only type one option. Do not type the brackets or pipe symbols.
Notes
Icons and labels call attention to informational notes and reader alerts as shown in the
following table.
Table 3. Note Icons and Labels
Icon Label Description
Note/Important Emphasizes content and provides additional information.
Important Strongly emphasizes key content.
Tip Highlights a best practice.
Critical Indicates strongly recommended actions.
Warning Indicates strongly recommended actions required to prevent
data loss or other undesirable results.
Do Not Warns against actions that may cause system failure or data loss.
More Information Link to reference material.
More Help Link to guides, white papers, or KB articles.
Trends Indicates industry trends, top support issue trends, etc.
Tables and Figures
Each table and figure is preceded by Caption. Captions are numbered sequentially
throughout each module.
Course Document and Slide Numbering
Modules may be numbered sequentially within a course. Lessons, demonstrations, and videos
may be numbered sequentially within a module. Topic and subtopic headings are not
numbered. Lab sessions may be numbered sequentially throughout the course. Individual
exercises are numbered sequentially within each lab session.
Troubleshooting Lingering Objects DRAFT V9.3 9.0 Course Workbook
38 © 2011 Microsoft Corporation. All rights reserved.
In each module, slide number paragraphs shown in the following figure identify the
presentation slide that accompanies the topic.
Figure 2. Slide Number Paragraph
Slide ##
The first slide in each presentation is unnumbered. Subsequent slides and slide indicator
paragraphs in each module are numbered sequentially starting with 1.
Note: Each presentation slide corresponds to a topic section in the module. Topic sections that include supplemental information may not be referenced on corresponding presentation slides.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 39
Lesson 1: Lingering Objects Fundamentals There is a lot of technical jargon associated with Lingering Object issues that you will need to
understand. The following section provides a definition for each term with context to enable
you to speak confidently when dealing with lingering object issues.
What You Will Learn
After completing this lesson, you will be able to:
Summarize seven terms commonly used in lingering object scenarios.
Terminology associated with Lingering Object issues
Lingering Objects
A lingering object is an object that is present on one replica, but has been deleted and garbage
collected on another replica.
Tombstone
When Active Directory deletes an object from the directory, it does not physically remove the
object from the database. Instead, Active Directory marks the object as deleted by setting the
object's isDeleted attribute to TRUE, stripping most of the attributes from the object,
renaming the object, and then moving * the object to a special container in the object's
naming context (NC) named CN=Deleted Objects. The object, now called a tombstone, is
invisible to normal directory operations.
Note: Some objects don’t get moved upon deletion and will therefore not be moved into the Deleted Objects container.
Tombstone Lifetime (TSL)
When an object is deleted, Active Directory replicates the deletion as a tombstone object. By
inbound-replicating this object, other domain controllers in the domain and forest become
aware of the deletion. The tombstone is retained in Active Directory for a specified period
called the tombstone lifetime. At the end of the tombstone lifetime, the tombstone is deleted
from the directory permanently.
More Help: For more help on this topic, see:
Determine the tombstone lifetime for the forest http://technet.microsoft.com/en-us/library/cc784932(WS.10).aspx
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 1: Lingering Objects Fundamentals
40 © 2011 Microsoft Corporation. All rights reserved.
In most cases, the default value is 60 days. If the forest was built on 2008 or later, it should
be 180. The minimum setting is 2 days.
Do Not: Do not reduce TSL to 2 days. (Unless directed to do so by a senior AD Replication SME)
Refer to the following table to determine TSL default values
Table 4: Default TSL Values
OS Install Path Default TSL
Windows 2000 RTM 60 days
Windows 2003 RTM, 2003 R2 60 days
Windows 2000RTM upgrade to Windows 2003 SP1 60 days
Windows 2003SP1, 2003SP2, 2008, 2008R2 180 days
NT4 upgrade to Windows 2003 SP1 180 days
Removing Outdated Objects Following Expiration of Tombstone Lifetime
If a domain controller fails to replicate for a number of days exceeding the tombstone
lifetime, replicas of objects that have been deleted from a writable partition might remain in
that domain controller's directory. Because the tombstones of the deleted objects are
permanently removed from the directory at the end of the tombstone lifetime, a domain
controller that fails to replicate changes for tombstoned objects never deletes or garbage
collects deleted objects.
This condition can occur for a variety of reasons, including the following:
Prolonged misconfigurations (such as those that cause 1311 events);
Prolonged errors in name resolution, authentication, or the replication engine, each of
which blocks inbound replication;
Turning on a domain controller that has been offline for more than 60 days;
and,
Advancing system time or reducing TSL values in an attempt to accelerate garbage
collection before end-to-end replication has occurred for all naming contexts in the forest.
To avoid such conditions, incorporate monitoring regimens that detect domain controller
replication problems.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 41
Outdated objects can also occur due to hardware and software problems that render the
domain controller unreachable. Regardless of the reason, a deleted object can remain on a
domain controller in either of the following circumstances.
A domain controller goes offline immediately before the deletion of an object on another
domain controller, and remains offline for a period that exceeds the tombstone lifetime.
A domain controller goes offline immediately after the deletion of an object on another
domain controller, but before receiving replication of the tombstone, and remains offline for
a period that exceeds the tombstone lifetime.
The following provides information for a legacy operating system but is included here as it is
still relevant. Additionally, some pre-Windows 2000 SP3 domain controllers experience a
replication error condition after a non-authoritative restore. A large number of objects
created after the restore may never be considered for replication.
More Information: For more information about this topic, see:
Microsoft Knowledge Base Article 316829, “Possible Active Directory Inconsistency after You Restore a Domain Controller.”
On domain controllers that are running Windows Server 2003 or later, you can use the
Repadmin support tool to analyze and remove lingering objects from a domain controller that
you suspect or know has not replicated for a tombstone lifetime. This tool includes the
RemoveLingeringObjects command. This command removes objects that are outdated (do
not exist in a replica of the same directory partition on the source domain controller).
Problems with Lingering Objects
In Windows 2000, if an attribute for a lingering object had been replicated, the inbound
domain controller that had previously processed the deletion would re-animate the entire
object. However, this is undesirable for a number of reasons.
The lingering object is holding a value on a unique attribute, such as samAccountName, that
another object wants to use. This commonly occurs when the lingering object exists in the
read-only naming context but not the domain naming context.
The lingering object is a security risk. For example, it might represent a user that should be
deleted.
The lingering object only exists in the read-only naming context (global catalog). This
behavior makes the object difficult to delete in Windows 2000.
Important: A deleted user or group account remains in the global address list (GAL) on Exchange
servers. Therefore, although the account name appears in the GAL, attempts to send e-mail messages result in errors.
Multiple copies of an object appear in the object picker or GAL for an object that should be unique in the forest. Duplicate objects sometimes appear with altered
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 1: Lingering Objects Fundamentals
42 © 2011 Microsoft Corporation. All rights reserved.
names, causing confusion on directory searches. For example, if the relative distinguished name of two objects cannot be resolved, conflict resolution appends "*CNF:GUID" to the name, where * represents a reserved character, CNF is a constant that indicates a conflict resolution, and GUID represents the objectGUID attribute value.
E-mail messages are not delivered to a user whose Active Directory account appears to be current. After an outdated domain controller or global catalog server becomes reconnected, both instances of the user object appear in the global catalog. Because both objects have the same e-mail address, e-mail messages cannot be delivered.
A universal group that no longer exists continues to appear in a user’s access token. Although the group no longer exists, if a user account still has the group in its security token, the user might have access to a resource that you intended to be unavailable to that user.
A new object or Exchange mailbox cannot be created, but you do not see the object in Active Directory. An error message reports that the object already exists.
Searches that use attributes of an existing object incorrectly find multiple copies of an object of the same name. One object has been deleted from the domain, but it remains in an isolated global catalog server.
Strict and Loose Replication Constancy
If the attributes on a lingering object never change, the object is never considered for
replication. However, if an attribute changes, the attribute is considered for outbound
replication. The problem is that the receiving domain controller does not hold the object for
the attribute being replicated. An update cannot be performed because the entire object does
not exist on the partner domain controller. What happens next depends on the replication
consistency set on the domain controller.
Loose Replication Consistency
When replication consistency is set to loose, the receiving domain controller detects that it
does not have the object for the attribute that is being replicated. The inbound partner
requests the entire object from the outbound partner, and reanimates the object on its copy
of the directory. The same process repeats on all domain controllers that do not have a copy
of the object.
This mechanism can be used to “reanimate” lingering objects across the entire forest. If a
lingering object is discovered and its presence is appropriate, then you may perform any
update to that object. As long as replication consistency is set to loose on all domain
controllers, the object will be reanimated as it replicates around the forest.
“Loose replication consistency” is the default for Windows 2000 domain controllers (except
on domain controllers that have the Security Rollup Package installed from November 2001).
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 43
Strict Replication Consistency
Because of the issues outlined above in the Problems section, the default behavior for
Windows Server 2003 (and upgraded Windows NT 4.0 domain controllers) is to block
inbound replication per naming context when a domain controller receives an update to an
object that it does not have. Replication is halted in the naming context for the object until the
lingering object is removed or the replication mode is set to loose.
Storage for Consistency Setting
The setting for replication consistency is in the registry on each domain controller.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
Strict Replication Consistency
Value: 1 (Set to 0 to disable)
Data type: REG_DWORD
Note
A post-SP2 hot fix (also included in the SRP) from November of 2001 used a different registry
value. A setting of 0 will not recreate the missing object (strict), and a setting of 1 will create
the missing object. This value is only needed with the November version of the hot fix.
Value Name: Correct Missing Objects
Data type: REG_DWORD
Value: 1
Defines how a destination DC behaves if a source DC sends updates to an object that does
not exist in the destination DC’s local copy of Active Directory.
o Destination DCs should see USN for creates before object is modified
o Only modifies for lingering objects arrive for object not on destination DC
o Only destination DC’s enforce strict replication and log events
Destination DCs stop replicating from source DC’s partitions containing LO’s
Lingering objects are quarantined on source DCs where they can be detected
End-to-end replication may be impacted for partitions containing lingering objects
Administrators must remove lingering objects to restore replication
Enabling Strict Replication
Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command
prompt:
For all domain controllers, type:
repadmin /regkey * +strict
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 1: Lingering Objects Fundamentals
44 © 2011 Microsoft Corporation. All rights reserved.
For all global catalog servers, type:
repadmin /regkey gc: +strict
You can also enable strict replication by manually setting the Strict Replication Consistency
registry value to 1.
HKLM\System\CurrentControlSet\Services\NTDS\Parameter Strict Replication Consistency
(Reg_DWORD) to 1
1 (enabled): Inbound replication of the specified directory partition from the source is
stopped on the destination.
Warning: Ensure you are prepared to deal with replication failures after enabling strict replication consistency.
Loose Replication Consistency
If you enable Loose Replication Consistency, if a destination receives a change to an object
that it does not have, the entire object is replicated to the target for the sake of replication
consistency. This behavior causes a lingering object to be reapplied to all domain controllers
in the replication topology.
Enable Loose Replication
Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command
prompt:
For all domain controllers, type:
repadmin /regkey * -strict
For all global catalog servers, type:
repadmin /regkey gc: -strict
You can also enable strict replication by manually setting the Strict Replication Consistency
registry value to 0.
HKLM\System\CurrentControlSet\Services\NTDS\Parameters Value: Strict Replication Consistency Type: (Reg_DWORD) Value Data: 0 0 (disabled): The destination requests the full object from the source domain controller, and the lingering object is revived in the directory as a new object.
Critical: The Loose Replication Consistency setting will cause the undesirable behavior of reanimation of lingering objects.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 45
Ensure Strict Replication Consistency Is Enabled On Newly Promoted
Domain Controllers
If you are upgrading a forest that was originally created using a computer running Windows
2000 Server, you should ensure that the forest is configured to enable strict replication
consistency on newly promoted domain controllers to help avoid lingering objects. After you
update the forest, all new domain controllers that you subsequently add to the forest are
created with strict replication consistency disabled. However, you can implement a forest
configuration change that causes new domain controllers to have strict replication
consistency enabled. To ensure that new domain controllers that you add to the forest have
strict replication consistency enabled, you can use Ldifde.exe to create an object in the
configuration directory partition of the forest. This object is responsible for enabling strict
replication consistency on any Windows Server 2003 domain controller that is promoted into
the forest.
The object that you create is an operational GUID with the following name:
CN=94fdebc6-8eeb-4640-80de-ec52b9ca17fa,CN=Operations,CN=ForestUpdates,CN=Configuration,DC=<ForestRootDomain>
Perform the following procedure on any domain controller in the forest to add this object to
the configuration directory partition.
Requirements:
Administrative credentials: To complete this procedure, you must be a member of the
Domain Admins group.
Tools: Ldifde.exe, Notepad
To create the object that ensures strict replication consistency on new domain
controllers
1. In a text editor such as Notepad, create the following text file:
dn: CN=94fdebc6-8eeb-4640-80de-ec52b9ca17fa,CN=Operations,CN=ForestUpdates,CN=Configuration,DC=<ForestRootDomain> changetype: add objectClass: container showInAdvancedViewOnly: TRUE name: 94fdebc6-8eeb-4640-80de-ec52b9ca17fa objectCategory: CN=Container,CN=Schema,CN=Configuration,DC=<ForestRootDomain>
Where <ForestRootDomain> contains all domain components (DC=) of the forest root
domain. For example, for the contoso.com forest, DC=contoso,DC=com; for the
fineartschool.net forest, DC=fineartschool,DC=net.
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 1: Lingering Objects Fundamentals
46 © 2011 Microsoft Corporation. All rights reserved.
2. Open a Command Prompt as an administrator: On the Start menu, right-click Command
Prompt, and then click Run as administrator. If the User Account Control dialog box
appears, provide Enterprise Admins credentials, if required, and then click Continue.
3. At the command prompt, type the following command and then press ENTER:
ldife -i –f <Path\FileName>
Value Description
-i Specifies import mode. If not specified, the default mode is
export.
-f Identifies the import or export file name.
<Path\FileName> The path and name of the import file that you created in step 1.
For example, C:\ldifde.txt.
More Information: For more information about this topic, see:
http://technet.microsoft.com/en-us/library/cc780362(WS.10).aspx
Abandoned object
An object created on one DC that is not replicated to other DCs hosting a writable copy of the NC but is replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition. The net effect is the object exists only in read-only copies of the partition. The object is present on RODCs or GCs hosting a read-only copy of the partition.
Abandoned delete
An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.
Table 5: Lingering Object Terminology
Term Definition
Abandoned delete An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.
Abandoned object An object created on one DC that never got replicated
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 47
to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition.
Lingering link
A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale references are referred to as lingering links.
Lingering Object
An object that is present on one replica, but has been deleted and garbage collected on another replica.
Loose Replication Consistency With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This undesirable behavior causes a lingering object to be “reanimated.”
Strict Replication Consistency With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, replication is blocked
with the source DC for the partition where the lingering object was detected
Tombstone An object that has been deleted but not yet garbage collected
Tombstone Lifetime (TSL)
The amount of time tombstones are retained in Active Directory before being garbage collected and permanently purged from the database.
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 2: Symptoms and Cause
48 © 2011 Microsoft Corporation. All rights reserved.
Lesson 2: Symptoms and Cause It is uncommon for an Administrator to be aware of and want to resolve a lingering object
problem without first experiencing some other problem in their environment that leads them
to discover the lingering object issue. This lesson will present common symptoms and causes
of lingering objects.
What You Will Learn
After completing this lesson, you will be able to:
Identify four symptoms of lingering object issues
Explain three ways in which lingering objects are created
List at least three methods to prevent lingering objects.
Symptoms of Lingering Objects
Detection of Domain Controllers That Have Not Replicated in
the Tombstone Lifetime
Windows Server 2003 records the last time a domain controller has replicated (directly or
transitively). Each domain controller will periodically compare the last time a domain
controller replicated with the forest’s tombstone lifetime. If a domain controller does not
replicate within the tombstone lifetime, event 1864 is posted to the directory service (DS)
log.
Event ID: 1864 NTDS Replication This is the replication status for the following directory
partition on the local domain controller. The local domain controller has not recently
received replication information from a number of domain controllers. The count of domain
controllers is shown, divided into the following intervals.
More than 24 hours: 1
More than a week: 1
More than one month: 1
More than two months: 1
More than a tombstone lifetime: 1
Tombstone lifetime (days): 60
If a domain controller in this state attempts to replicate, the inbound domain controller will
block replication and alert the administrator with the message below (event 2042). In this
case, the administrator has the following options.
1. Forcefully demote or reinstall the domain controllers that have not replicated, and then
perform a metadata cleanup.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 49
2. Remove any lingering objects on the non-replicating domain controller, and then enable
replication with divergent or corrupt partners (as follows).
a. Run repadmin /removelingeringobjects (see “Removing Lingering Objects with
Repadmin” for instructions).
b. Enable replication with divergent or corrupt partners by adding the following
registry key.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
Allow Replication With Divergent and Corrupt Partner
Value: 1 (Set to 0 to disable)
Data type: REG_DWORD
Important
Before using the above-mentioned key to override this replication safeguard, be sure to use
repadmin /removelingeringobjects command to prevent the spread of unwanted lingering
objects. Once replication has succeeded, be sure to remove the “Replication With Divergent
and Corrupt Partner” value, or set it to zero.
Event Source: NTDS Replication
Event Type: Error
Event Category: Replication
Event ID: 2042
Description:
It has been too long since this machine last replicated with the named source machine. The time
between replications with this source has exceeded the tombstone lifetime. Replication has been
stopped with this source.
The reason that replication is not allowed to continue is that the two machine's views of deleted
objects may now be different. The source machine may still have copies of objects that have been
deleted (and garbage collected) on this machine. If they were allowed to replicate, the source machine
might return objects which have already been deleted.
Time of last successful replication:
<date and time of last replication>
Invocation ID of source:
<invocation ID of the source DC>
Name of source:
<replication guid._msdcs.forest.root of source DC>
Tombstone lifetime (days):
60
The replication operation has failed.
User Action:
Determine which of the two machines was disconnected from the forest and is now out of date. You
have three options:
1. Demote or reinstall the machine(s) that were disconnected.
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 2: Symptoms and Cause
50 © 2011 Microsoft Corporation. All rights reserved.
2. Use the "repadmin /removelingeringobjects" tool to remove inconsistent deleted objects and then
resume replication.
3. Resume replication. Inconsistent deleted objects may be introduced. You can continue replication by
using the following registry key. Once the systems replicate once, it is recommended that you remove
the key to reinstate the protection.
Registry Key:
HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Allow Replication With Divergent and
Corrupt Partner
Replication Errors Caused by Lingering Objects
If a domain controller has replicated within the tombstone lifetime and replication
consistency is set to loose, administrators cannot be alerted to the presence or replication of
lingering objects. If an attribute is changed, the object is re-animated by all participating
domain controllers without notification. If strict replication is enabled in the domain,
replication of the partition hosting the object is halted on all inbound domain controllers.
Replication for the partition stops until the object is removed or replication consistency is set
to loose. When the replication is halted, the following error message is reported in the DS log
on the inbound domain controller.
Event ID: 1988
Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Description:
Another domain controller has attempted to replicate into this domain controller an object which is
not present on this domain controller. The object may have been deleted and already garbage collected
(a tombstone lifetime or more has past since the object was deleted) on this domain controller.
Replication will not continue with the source domain controller until the situation has been resolved.
Source DC:<DC guid>._msdcs.<forestroot>
Object:<dn of object>
Object GUID: <guid of object>
User Action:
Verify that the object was deleted on this domain controller or in the forest. If object restoration is
desired, authoritatively restore the object on the source domain controller. If restoration isn't desired,
install the support tools included on the installation CD and use "repadmin /removelingeringobjects"
on the source domain controller to remove the object from the forest and continue replication. To
allow automatic restoration of this object and future similar objects on this domain controller, the
following registry key can be deleted.
Registry Key:
HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Strict Replication Consistency
Repadmin /showreps
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 51
In addition to the above-mentioned event, repadmin reports the following.
Sitename\<DC sending lingering object> via RPC
DC object GUID: <dsa guid of dc>
Last attempt @ 2002-07-19 19:14:43 failed, result 8606
(0x219e):
Insufficient attributes were given to create an object. This object may not exist because it may have
been deleted and already garbage collected.
Cause of Lingering Objects
How lingering objects occur
When a domain controller is disconnected for a period that is longer than the TSL, one or
more objects that are deleted from Active Directory on all other domain controllers may
remain on the disconnected domain controller. Such objects are called lingering objects.
Because the domain controller is offline during the time that the tombstone is alive, the
domain controller never receives replication of the tombstone.
When this domain controller is reconnected to the replication topology, it acts as a source
replication partner that has an object that its destination partner does not have.
Replication problems occur when the object on the source domain controller is updated. In
this case, when the destination partner tries to inbound-replicate the update, the destination
domain controller responds in one of two ways:
If the destination domain controller has Strict Replication Consistency enabled, the
controller recognizes that it cannot update the object. The controller locally stops
inbound replication of the directory partition from the source domain controller.
If the destination domain controller has Strict Replication Consistency disabled, the
controller requests the full replica of the updated object. In this case, the object is
reintroduced into the directory.
Five Causes of Lingering Objects
Cause 1: The source DC sends updates to object that have already been garbage collected on the destination Dc either because the source DC has been offline or has failed replicati An object deleted on
one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 2: Symptoms and Cause
52 © 2011 Microsoft Corporation. All rights reserved.
An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.
on for TSL elapsed # of days
The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC2 experiences a motherboard failure. Meanwhile, DC1 makes originating deletes for stale security groups over each of the next 90 days. After being offline for 90 days , DC2 gets its a replacement motherboard, powers up then originates an ACL change on all user accounts before it inbound replicates knowledge of originating deletes from DC1. DC1 logs 8606 errors for updates security groups purged on DC1 for the 1st 30 days that DC2 was offline.
Cause 2: The Source DC sends updates to objects @ the cusp of TSL expiration
that have already been garbage collected by a strict mode destination DC
The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. DC1 is in-place upgraded to W2K8 R2 which stamps new attributes on all objects in the configuration and writable domain partitions, including objects currently in the deleted objects container, some of which were deleted 60 days ago and now at the cusp of tombstone expiration. DC2 garbage collects some of the objects deleted TSL days ago before the replication schedule opens with DC2. Error 8606 is logged until DC1 garbage collects the blocking objects. Any updates to the partial attribute set can cause temporary lingering objects that, like the addition of the 1st W2K8 R2 DC to an existing forest, will clear themselves up once source DCs garbage collect deleted objects @ the cusp of TSL expiration.
Cause 3: A time jump on a destination DC prematurely accelerates the garbage
collection of deleted objects on a destination DC
The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. The reference time source used by DC1 (but not DC2) rolls forward to calendar year 2039, causing DC2 to also adopt a system time in CY2039 which causes DC1 to prematurely purge objects deleted today from its deleted objects container. DC2 meanwhile originates changes to attributes on users, computers and groups that are live on DC2 but deleted and now prematurely garbage collected on DC1. DC1 will log error 8606 when it next inbound-replicates changes for the premature deleted objects.
Cause 4: An object is reanimated at the cusp of TSL expiration
The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. An OU containing users, computers and groups is
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 53
accidentally deleted. A system state backup made at the cusp of TSL in the past is auth restored on DC2. The backup contains objects that are live on DC2 but already deleted and garbage collected DC1.
Cause 5: A USN bubble is triggered the logging of the 8606
Say you create an object in a USN bubble, such that it doesn’t outbound replicate because the destination DC "thinks" it has the object due to the bubble. Now, after the bubble closes and new changes start replicating again, a change is created for that object on the source DC and appears as a lingering object to the destination DC which logs the 8606 event.
Lingering Object Prevention
It's easy to come up with methods to prevent lingering objects, now that you know how they
are caused. Keep the following in mind the next time someone asks you what they need to do
to ensure they hit this issue again.
Important: o Resolve replication failures within TSL
o Ensure Strict Replication Consistency is enabled
o Ensure large jumps in system time are blocked via registry key or policy
o Don't remove replication quarantine with "allowDivergent" setting without removing LOs first
o Don't restore system backups that are near TSL number of days old
o Don't bring DCs back online that haven't replicated within TSL
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 3: Identification and Classification
54 © 2011 Microsoft Corporation. All rights reserved.
Lesson 3: Identification and Classification
What You Will Learn
After completing this lesson, you will be able to:
Use repadmin.exe to generate diagnostic data for analysis
Use diagnostic data to determine the scope of the problem by listing all partitions
and all servers containing lingering objects.
Create a replication health report A good first step in tracking down the cause of Active Directory replication failures is to get a
list of the replication errors encountered. This is a very simple procedure using repadmin
/showrepl with the /csv option. For every domain controller in the forest, the spreadsheet
shows the source replication partner, the time that replication last occurred, and the time
that the last replication failure occurred for each naming context (directory partition). By
using Autofilter in Excel, you can view the replication health for working domain controllers
only, failing domain controllers only, or domain controllers that are the least or most current,
and you can see the replication partners that are replicating successfully.
To generate a forest-wide replication status spreadsheet for domain controllers:
1. Open a Command Prompt as an administrator: On the Start menu, right-click
Command Prompt, and then click Run as administrator. If the User Account Control
dialog box appears, provide Enterprise Admins credentials, if required, and then click
Continue.
2. At the command prompt, type the following command, and then press ENTER
repadmin /showrepl * /csv >showrepl.csv
3. Open Microsoft Excel.
4. Click the Office button (File menu for versions prior to Excel 2010), click Open,
navigate to showrepl.csv, and then click Open.
5. Hide or delete column A and column G, as follows:
To hide a column, right click the column header then click Hide
To delete a column, right click the column header then click Delete
6. Select a column that you want to hide or delete.
7. Select row 1 beneath the column heading row. On the View tab, click Freeze Panes, and
then click Freeze Top Row.
8. Select any cell. On the Data tab, click Filter.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 55
9. In the Last Failure Status column, click the filter down arrow, deselect the value 0.
You now have a filtered report showing only the replication failures. Deselect all values
except value 8606 to display just the replication failures caused by lingering objects..
Try This: Generate an AD Replication report using repadmin
Take what you have learned and try to use repadmin.exe to generate a forest-wide AD
Replication report
1. Connect to DC1 in your lab environment.
2. Use the steps documented above to generate a filtered report.
3. Save the report to the desktop as showrepltimestamp.xls
Use AD Replication report and repadmin to determine the scope of the problem
The list of DCs in the Source DC column contain lingering objects when the replication report
is filtered on value 8606 in column K. This display gives you the following information:
DC containing lingering objects
Partition where lingering objects exist
This is two of the three data points needed for repadmin /removelingeringobjects.
Important: Repadmin /RemoveLingeringObjects DestinationDC SourceDC_Guid DirectoryPartition (Optional switch /advisory_mode)
DC containing lingering objects = DestinationDC
Partition where lingering objects exist = DirectoryPartition
A common misconception is that the list you have just generated is comprehensive and once
you remove lingering objects from the DCs in the Source DC column your job is done.
However, that may not be the case as this is only a list of DCs where replication is currently
blocked. It is entirely possible that once you remove lingering objects from these DCs,
replication will begin failing with these now-clean DCs as the destination and a new list of
DCs as the source. Once you have a list of DCs containing lingering objects
Tip: To save time, act as if all DC / GCs contain lingering objects for the partition in question.
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 3: Identification and Classification
56 © 2011 Microsoft Corporation. All rights reserved.
Run repadmin /removelingeringobjects in /advisory_mode first to see what objects are
considered lingering on the DC. Event ID 1946 is logged once per lingering objects on the
destination DCs Directory Services event log.
Tip: Increase the size of the Directory Services event log prior to running repadmin /removelingeringobjects with the /advisory_mode option. It is common to see the event log wrap when this command is run and the event log is the default size.
You can also use ldifde and replfix.exe to generate a list of lingering objects. This process is describe in Lesson 4.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 57
Lesson 4: Lingering Object Removal
What You Will Learn
After completing this lesson, you will be able to:
Execute the steps in an action plan in order to remove lingering objects
Remove lingering objects using five different methods
Methods to Remove Lingering Objects Common methods to remove lingering objects include:
Repadmin /Removelingeringobjects
Repldiag
Replfix
Manually through LDP or using script
Rehost the partition: o Repadmin /rehost (or /unhost and /add) (only if the partition is not-writable on the
DC containing lingering objects) o Un-GC (but you don’t really have control over who the DCs sources the partition
from) o Demote and Promote (DCPromo)
Removing Lingering Objects with Repadmin
Repadmin includes an advanced switch (view using /experthelp) to remove lingering objects
from a specific server.
To remove outdated (lingering) objects from a directory partition on a domain
controller that has not replicated for a tombstone lifetime, perform the following.
1. Using Repadmin, type the following at the command line:
Repadmin /RemoveLingeringObjects DestinationDC SourceDC_Guid
DirectoryPartition (Optional switch /advisory_mode)
where
DestinationDC is the DNS name or IP address of the domain controller that has
outdated objects; and,
SourceDC_Guid is the domain controller’s object GUID.
To obtain the object’s GUID, do one of the following.
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal
58 © 2011 Microsoft Corporation. All rights reserved.
o Use Repadmin /showrepl SourceDCName. The domain controller’s object GUID is
listed as “domain controller object GUID.”
-or-
o In Active Directory Sites and Services, find the Source domain controller under
Sites\<the domain controller’s Site>\ Servers\ DCname\ NTDS Settings\ Properties.
Look in the DNS Alias box. The GUID prior to _msdcs.forestrootname.com is the
domain controllers Object GUID. Repadmin only needs the GUID. Omit
_msdcs.forestrootname.com from the Repadmin syntax.
DirectoryPartition is the distinguished name of the directory partition from which to
remove outdated objects.
2. Repeat the procedure for the following partitions, as needed.
Domain directory partition dc=DomainName…,dc=ForestRootDomainName
Configuration directory partition cn=configuration,dc=DomainName…,dc=ForestRootDomainName
Application directory partition or partitions cn=ApplicationDirectoryPartitionName,dc=DomainName…,dc=ForestRootDomainName
Schema directory partition cn=schema,cn=configuration,dc=ForestRootDomainName
The following is an example of the command syntax.
C:\>repadmin /removelingeringobjects lonemeadc.emea.contoso.com B0AE6093-15F5-
4DB8-836B-4495F3B19493 dc=contoso,dc=com /advisory_mode
RemoveLingeringObjects successful on lonemeadc.emea.contoso.com
Events Associated with Lingering Object Removal
When removing lingering objects, the target domain controller (the domain controller with
the lingering objects) will record all removal information, including source domain controller,
objects removed, and a total count of all objects removed.
Event ID 1937: NTDS Replication. Lingering Object Removal has been initiated on this
domain controller. All objects on this DC will have their existence verified on the following
source domain controller. Objects that have been deleted and garbage collected from the
source domain controller will be DELETED from this domain controller if they still exist.
Subsequent event logs will list all deleted objects.
Source DC: <source DC guid ._msdcs.<forest root>
Event ID 1945: NTDS Replication. Lingering Object Removal will DELETE the following
object. Its deletion and garbage collection was detected on the source domain controller
without replicating the deletion to this domain controller.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 59
Object:DC= <dn of lingering object>
Object GUID:<objectGUID>
Source DC: <dc guid> ._msdcs.<forest root>
Event ID 1939: NTDS Replication. Lingering Object Removal has executed successfully on
this domain controller. All objects on this domain controller have had their existence
verified on the source domain controller. Objects that had been deleted and garbage
collected from the source domain controller were DELETED from this domain controller.
Previous event logs list all such objects.
Source DC: <source DC guid> ._msdcs.<forest root>
Lingering Objects Deleted 23
Details of Repadmin’s Lingering Object Removal Mechanism
To be added after external reviews are complete.
Remove Lingering Objects Using Repldiag
Removing lingering objects from a forest with repldiag is as simple as running repldiag
/removelingeringobjects. However, it is usually best to exercise some control over the
process in larger environments. The option /OverRideReferenceDC allows you to select
which DC is used for cleanup. The option /outputrepadmincommandlinesyntax allows you to
see what a forest-wide cleanup looks like using repadmin.
Tip: Repldiag is by far the easiest and fastest way to remove lingering objects. The other methods are important to know when repldiag is not an option.
Help
Replication topology analyzer. Written by [email protected] Version: 2.0.3397.24022 Command Line Options: ReplDiag [/Save] [/CheckForStableReplTopology] [/RemoveLingeringObjects] [/ImportData:<FileName.XML>] [/ShowTestCases] [/OverrideDefaultReferenceDC:"dc=namingcontext,dc=com":domainController.namingcontext.com] /UseRobustDCLocation -Query each and every DC for a list of DCs in
forest. Ensures replication instability does not cause any to be missed.
/Save -Save out the data from the current environment to XML. File is named "ReplicationData.xml" and is located in the current directory.
/ImportData -Import the XML that was saved during a prior execution of this utility. Run one of the other options to do something with the data.
/ShowTestCases -Show detail about test cases. Lingering Object Cleanup: /RemoveLingeringObjects -Use the current forest topology to clean all the
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal
60 © 2011 Microsoft Corporation. All rights reserved.
NCs in the forest. WILL NOT CLEAN WINDOWS 2000 SYSTEMS!!!
/AdvisoryMode -Check for lingering objects only, do not clean. Must be used with /RemoveLingeringObjects.
/OverrideDefaultReferenceDC -Specify reference DC for a naming context when when removing lingering objects, can be used multiple times for different NCs. Only functional if using /RemoveLingeringObjects.
/OutputRepadminCommandLineSyntax -Output the command line syntax for repadmin. Only active in conjunction with /RemoveLingeringObjects.
Example syntax: ReplDiag /Save - Collect the AD replication topology from the environment and save it. ReplDiag /ImportData:"ReplicationData.xml" - Load in previously collected data and check replication status. ReplDiag /RemoveLingeringObjects /OverrideDefaultReferenceDC:"cn=Configuration,dc=forestroot,dc=com":dc1.forestroot.com /OverrideDefaultReferenceDC:"dc=forestroot,dc=com":dc2.forestroot.com
Sample output
Repldiag.exe /save
Open ReplicationData.xml in Excel
Repldiag /removelingeringobjects /outputrepadmincommandlinesyntax This will give you output of corresponding repadmin /removelingeringobjects syntax. It will first select one DC per partition to be used as a reference DC. It will then clean the reference DCs up against all other DCs for the partition(s) it was selected to be used as a reference for. Finally it cleans up all other DCs in the forest with the new “cleaned up” reference DCs as sources. The /outputrepadmincommandlinesyntax option does not actually attempt object cleanup. You would need to leave this option off if you want to execute lingering object cleanup.
Number Complete,Status,Server Name,Naming Context,Reference DC,Duration,Error Code,Error Message repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=forestdnszones,dc=contoso,dc=com
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 61
repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed6-79b5626db9fd dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f6-4298f02505e8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b-54b0431a374a dc=corp,dc=contoso,dc=com Reference NCs cleaned in 0h:0m:0s. Cleaning everything else against reference NCs. repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c66-8e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 dc=corp,dc=contoso,dc=com All NCs cleaned in 0h:0m:0s.
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal
62 © 2011 Microsoft Corporation. All rights reserved.
This output can also be viewed in Excel: Copy commands to a text file. Modify the text file to include only the command portion of the output. Then open up the text file in Exel. (space delimited)
More control: /OverRideReferenceDC This option allows you to specify a DC that you want to be used as a reference DC for the partition specified. In a large distributed environment, take careful consideration when choosing the reference DC. Things to consider when choosing a suitable reference DC: Well connected: Fast WAN link. Performance: Excellent server class hardware: Disk, RAM, CPU and NIC Critical Network Applications / Services do not depend on this DC: Such as an Exchange
facing DC Other DCs don’t report replication failures with reference DC as the source: filter repadmin
/showrepl * /csv ouput, or use the topology report created by repldiag /save. repldiag /removelingeringobjects /overridedefaultreferencedc:"cn=configuration,dc=contoso,dc=com":nycorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=corp,dc=contoso,dc=com":seacorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=forestdnszones,dc=contoso,dc=com":5thwardcorpdc.corp.contoso.com /outputrepadmincommandlinesyntax Replication topology analyzer. Written by [email protected] Version: 2.0.3397.24022 Command Line Switch: /removelingeringobjects Command Line Switch: /overridedefaultreferencedc:cn=configuration,dc=contoso,dc=com:nycorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=corp,dc=contoso,dc=com:seacorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=forestdnszones,dc=contoso,dc=com:5thwardcorpdc.corp.contoso.com Command Line Switch: /outputrepadmincommandlinesyntax Attempting to override NC cn=configuration,dc=contoso,dc=com with DC nycorpdc.corp.contoso.com... Overriden Attempting to override NC dc=corp,dc=contoso,dc=com with DC seacorpdc.corp.contoso.com... Overriden Attempting to override NC dc=forestdnszones,dc=contoso,dc=com with DC 5thwardcorpdc.corp.contoso.com... Overriden
/UseRobustDCLocation Query each and every DC for a list of DCs in forest. Ensures replication instability does not cause any to be missed. We’ve had cases where we clean up lingering objects in the forest but do to an AD topology problem some DCs were not cleaned up. This option is almost always recommended if you want it to do a thorough job.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 63
Remove Lingering Objects Using Replfix
Remove Lingering Object using LDP or Script
Removing Lingering Objects in Windows 2000
Unfortunately, Windows 2000 provides no easy way to detect and remove lingering objects. A
supported method to delete these objects is documented in MSKB 314282: “Lingering Objects
May Remain After You Bring an Out-of-Date Global Catalog Server Back Online”
In Windows 2000 SP3 (and in the post-SP2 hot fix), enhancements were made that allow an
administrator to enable strict replication. This will help identify lingering objects and prevent
them from replicating. However, lingering objects will not be detected unless an attribute on
the object is changed.
Note: Even though this method was first used for Windows 2000, it is still sometimes needed in certain scenarios.
Remove Lingering Objects by partition re-host operation
When one of the other methods is not an option, it is sometimes necessary to re-host the
partition from a DC containing a good clean writable copy of the partition. This may be a
temporary solution if the problem is widespread since the DC may later replicate with a DC
that is not clean.
Tip: If re-host is necessary, it is usually best to identify all GCs needing the procedure and clean them up at the same time to prevent recurrence.
Un-hosting a partition
It is sometimes necessary to remove a partition from the database of a DC temporarily.
Repadmin includes a /rehost option that allows you to do this, but the /unhost option allows
you to exercise more control over the procedure. Take note that /unhost only allows you to
remove a read-only copy of the partition. With the exception of application partitions, you
cannot remove a writable copy of a partition from a DC without using DCPROMO.
Repadmin /?:unhost Remove a specific read-only partition from a GC. [SYNTAX] /unhost DSA <Naming Context> Repadmin /unhost ContosoDC1 dc=corp,dc=contoso,dc=com
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal
64 © 2011 Microsoft Corporation. All rights reserved.
Event ID 1659 indicates the status of the un-host operation. Do not re-add the partition until
event ID 1660 is logged in the Directory Services event log.
Warning: The re-host operation may fail with error 8339 if you attempt to re-add the partition too soon after the un-host.
Manually adding a replication connection using repadmin.exe
The add command will create a RepsFrom attribute on the destination domain
controller for the specified naming context and initiate a replication request. During a
normal replication cycle, the destination domain controller will request updates from
the source domain controller. When creating temporary replication links between
replication partners, the process could fail if the KCC starts while you are performing
the procedure. The KCC will delete any replication links for which no corresponding
connection object exists. Since these commands can take a very long time to
complete as they trigger the replication of the corresponding naming context, it is
important to ensure that KCC do not disturb the process. This is where you would use
+DISABLE_NTDSCONN_XLATE which effectively disables KCC's capability to translate
connection objects to replication links.
Disable KCC connection translation so that KCC doesn’t remove our temporary
replication connection:
Repadmin /options ContosoDC1 +disable_ntdsconn_xlate
Then add a replication connection for the configuration partition of the server we
want to source the partition from:
Repadmin /add <Naming Context> <Dest DSA> <Source DSA> [/readonly] [/selsecrets] <Source DSA> The source DSA must be specified by fully qualified computername. repadmin /add cn=configuration,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.
Add a replication connection to the server for the domain partition that we
need to source from (/readonly is specified if the partition is a GC non-writable
partition /selsecrets needs to be specified if the destination DC is an RODC):
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 65
repadmin /add dc=emea,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com /readonly One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.
If you need to replicate the other way, then just reverse the order of the server names
in the commands.
To begin a normal sync of the partition using the new replication connection:
Repadmin /replicate <Dest_DSA_LIST> <Source DSA_NAME> <Naming Context> [/force] [/async] [/full] [/addref] [/readonly] repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly
To begin a full sync of that partition using the new replication connection:
repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly /full Sync from LONEMEADC.Emea.contoso.com to ContosoDC1.contoso.com completed successfully.
Turn KCC connection translation back on when you no longer need the connection:
Repadmin /options ContosoDC1 -disable_ntdsconn_xlate
DSASTAT Dsastat can be used to compare the number of objects that exist on two domain controllers.
However, it cannot report on which objects exist on one and not the other. Likewise, it cannot
make an intelligent determination about the differences. Replication latency or other factors
might result in valid cases where an object exists but has not replicated out. Some objects are
set to not replicate (like the Universal group membership cache). For this reason, DSASTAT
can only be used as a guideline for comparisons between naming contexts hosted on different
domain controllers.
Read-only Naming Context (Global Catalogs) The global catalog is particularly susceptible to problems caused by lingering objects. This is
because an object can exist on a read-only naming context, but not in the domain naming
context from which it originally replicated. If it still existed in the domain naming context, it
could be deleted there, and the tombstone could remove it from the global catalog. The other
problem is that global catalogs can replicate from each other. The global catalog function
might be removed from a computer, and then reinstated in an attempt to re-replicate the
partial attribute set from a domain controller hosting the writable copy of the naming
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 4: Lingering Object Removal
66 © 2011 Microsoft Corporation. All rights reserved.
context. In this case, the global catalog might replicate from another global catalog. This
would return the object you were trying to delete. A better solution is to determine whether
the object exists on all global catalogs. If it does not, remove the global catalog function from
all servers that contain the object. Then reinstate the global catalog function on all of them,
and let a clean copy of the directory replicate in. In larger environments, removing and
reinstating the global catalog function might be undesirable and prohibited. Applications
such as Microsoft Exchange Server depend on the global catalog to operate. Moreover, the
additional traffic incurred as the domains re-replicate into the global catalog might be
undesirable. In this case, use the post-SP2 hot fix and process described in the following
article.
MSKB 314282: “Lingering Objects May Remain After You Bring an Out-of-Date Global Catalog
Server Back Online”
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 67
Lesson 5: Real World Application
What You Will Learn
After completing this lesson, you will be able to:
Create a detailed action plan that will remove the lingering objects in all partitions on
all servers
Recommend the correct method to remove lingering objects given five different
scenarios
Recommend changes that will result in a better solution given a subpar action plan
Determining What to Do with a Lingering Object
In most cases, a lingering object results from a missed tombstone. In other words, the object
was intentionally deleted because one or more domain controllers received the instruction to
delete the object. In rare cases, the object was not actually deleted. In such cases, the
existence of the object may be intended. Determining what to do with a deleted object
depends on whether or not it was intentionally deleted. First, let’s look at some common
lingering object scenarios, and then discuss the recommended corrective action.
Known Cause: Domain controller has not replicated beyond the tombstone lifetime. If a
domain controller has not replicated within the tombstone lifetime, it will likely have
missed the deletions of some attributes (or inbound replicated tombstones). Left alone, the
objects will persist without any replication. If any of the attributes are changed, or if
another domain controller is doing a full sync (as when a global catalog is populating its
copy of the domain partition), the objects will attempt to replicate out and cause problems.
These lingering objects are unintended, and should be removed using repadmin (see
below).
Unknown Cause: Security principal is attempting to replicate in. In some cases, a user or
computer object has become a lingering object without any known cause. These are almost
always undesirable. However, before removing them, check the event log and see what
object is being replicated in. If the object is desired, enable loose consistency. (Refer to the
section Intended Objects below.)
Unknown Cause: Deletion is replicating in (tombstone replicating in). If the inbound object is
a deletion (the object will include DEL in the name), it is probably harmless and not needed.
However, if the deleted object still exists on another domain controller somewhere else in
the forest, removing this lingering object will actually turn the other .live. copy into a
lingering object.
The next section examines what to do with intended and unintended objects.
Unintended Objects
Use Repadmin to delete these lingering objects (see below).
Troubleshooting Lingering Objects DRAFT V9.3 Lesson 5: Real World Application
68 © 2011 Microsoft Corporation. All rights reserved.
Intended Objects
Change the replication consistency on the inbound domain controller. The object will be re-
animated on this domain controller. When using this method, the following things should be
considered.
After the object has been reanimated and replicated into the domain controller, it will
replicate out to the domain controller’s other partners. It is not likely that the other
partners will have the object, and inbound replication will be blocked until the consistency
setting is changed. This might result in the lingering object or re-animation moving
throughout the domain. To animate the object fully, you might have to .chase. the replication
failures throughout the forest. Use Eventcomb to monitor for the lingering object detection
event.
While the idea of chasing a lingering object around a forest might not seem like much fun,
there is a good reason to do it. It is possible to turn off replication consistency in a domain
or forest (using scripts or custom ADM files with Group Policy). However, this could have
some unwanted side effects: for example, replication would be blocked for the first lingering
object.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 69
10.0 Lab Guide This lab manual describes the environment required to perform practice exercises in this
course and lab sessions included in this manual.
Before You Begin
Before starting this course you should:
Complete... course prerequisite.
Review... course prerequisite.
Practice exercises are performed on physical and virtual machines on one computer per
participant. To complete the exercises, your computer hardware and software must be
configured as described in this section.
For additional details, refer to the Classroom Setup Guide that accompanies this course.
Critical: Lab sessions that accompany this course use a preconfigured virtual machine environment. If you start or modify VMs in any way prior to use in lab exercises, exercise tasks and steps will not work as intended. DO NOT start or modify any VM until instructed to do so in the lab exercises.
Preconfigured VMs use lab environment scripts to complete certain steps at first launch based on the computer name entered in mini-setup. Failure to enter computer names specified in the lab exercises exactly as shown will incorrectly configure VMs, which will cause lab exercise tasks and steps to fail.
What You Will Learn
After completing the labs in this course, you will be able to:
Describe | Explain… course objective.
Install | Configure… course objective.
Analyze | Troubleshoot... course objective.
Troubleshooting Lingering Objects DRAFT V9.3 Lab Sessions
70 © 2011 Microsoft Corporation. All rights reserved.
Lab Sessions This manual includes the following lab sessions. Each lab includes step-by-step instructions
to complete the exercises. You can use the problem solving lab exercises in your workbook to
challenge your understanding of course material and refer to the Lab Manual for detailed
steps if needed.
Lab 1: Exploring Lingering Object Fundamentals
During this lab, you will identify the forest's configured tombstone lifetime and replication
consistency settings
Estimated time to complete this lab: 15 minutes
Lab 2: Lingering Object Diagnosis and Documentation
During this lab, you will generate diagnostic data via repadmin, ldifde and replfix. You will
then analyze that data and document all lingering objects in the environment.
Estimated time to complete this lab: 30 minutes
Lab 3: Lingering Object removal using repadmin
During this lab, you will remove lingering objects from the environment using repadmin
/removelingeringobjects.
Estimated time to complete this lab: 30 minutes
Lab 4: Lingering Object removal using ldp and repldiag
During this lab, you will remove a single lingering object using ldp. You will then remove the
remaining lingering objects using repldiag.
Estimated time to complete this lab: 30 minutes
Lab 5: Abandoned Object and Abandoned Deleted object remediation
During this lab, you will identify and remove an abandoned object. You will then remediate
and abandoned deleted object scenario.
Estimated time to complete this lab: 30 minutes
Lab 6: Lingering Link identification and cleanup
During this lab, you will identify all lingering-linked values in the environment. You will them
remove them in order to ensure group membership consistency.
Estimated time to complete this lab: 45 minutes
Setting Up Your Lab Environment To complete this lab, you will need the hardware and software configuration described in this
section.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 71
Hardware
Practice exercises assume that all lab hardware is listed on the Hardware Compatibility List
(HCL) as compatible with operating systems and applications described later in this section.
The following table describes minimum hardware requirements for practice exercises.
Table 6: Minimum Hardware Requirements
Minimum System Requirements
Computer/Processor Computer with a 2.4 GHz processor or higher
(If available, disable hyperthreading and enable hardware virtualization)
Operating System <Host OS>; see Classroom Setup Guide for details
Memory 4 GB RAM
Storage 160 GB hard drive
CD or DVD drive (DVD drive recommended)
Display Super VGA (800 x 600) or higher-resolution monitor with 256 color
(Recommended: 1024 x 768 with 16-bit or higher color)
Peripherals Microsoft Mouse or compatible pointing device
Microsoft or compatible keyboard
Software
Operating systems and applications listed in the following table must be installed on all
computers.
Table 7: Lab Computer requirements
Software Version tested and notes
Microsoft® Windows® 7, Enterprise Edition Service Pack 1
Current Microsoft® Windows® 7, Enterprise Edition Service Pack 1 and Critical Updates
Office 2010 Professional Service Pack 1
Microsoft Office 2010 OneNote Service Pack 1
Current Office 2010 Service Pack 1 and Critical Updates
Microsoft .NET Framework Version 2.0 Retail
Current .NET Framework 2.0 Service Pack and Critical Updates
Microsoft .NET Framework Version 3.0 Retail
Current .NET Framework 3.0 Service Pack and Critical Updates
Troubleshooting Lingering Objects DRAFT V9.3 Lab Sessions
72 © 2011 Microsoft Corporation. All rights reserved.
Software Version tested and notes
Current Adobe Reader Version Retail
Current Adobe Reader Critical Updates
Network Layout
The following figure illustrates the lab network. The lab network must be isolated from
production networks.
Figure 3: Network Layout
Individual computer configurations are described in detail in the next section.
Computer Names and IP Addresses
Table 8: Lab Computer Names and IP Addresses on page 73 lists computer configurations
for the classroom lab network.
Replace <Host> in Computer Name with <site>-<room> as follows:
<site> Site name abbreviation (Example: For Las Colinas, use LC1 or LC2)
<room> Room number (Example: For Rio Grande classroom, use 1693)
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 73
For example, participant computer 1 in the Rio Grande classroom in Las Colinas
Building 2 would be named LC2-1693-1 or LC2-1693-1A (see table).
Replace x in IP address with the classroom number or any representative number that
is unique on the overall classroom subnet and reference this number in all lab
exercises.
Important: This computer naming convention eliminates potential issues when multiple classrooms are connected to the same subnet during classroom configuration or course delivery.
Table 8: Lab Computer Names and IP Addresses
Computer Name IP Address Preferred DNS Server Role
<Host>- Instr-1 172.168.1.200 192.168.x.200 Stand-alone Server
<Host>-Instr-2 172.168.x.201 192.168.x.200 Stand-alone Server
<Host>-1 172.168.x.101 192.168.x.200 Stand-alone Server
<Host>-2 172.168.x.102 192.168.x.200 Stand-alone Server
<Host>-3 172.168.x.103 192.168.x.200 Stand-alone Server
<Host>-4 172.168.x.104 192.168.x.200 Stand-alone Server
<Host>-5 172.168.x.105 192.168.x.200 Stand-alone Server
<Host>-6 172.168.x.106 192.168.x.200 Stand-alone Server
<Host>-7 172.168.x.107 192.168.x.200 Stand-alone Server
<Host>-8 172.168.x.108 192.168.x.200 Stand-alone Server
<Host>-9 172.168.x.109 192.168.x.200 Stand-alone Server
<Host>-10 172.168.x.110 192.168.x.200 Stand-alone Server
<Host>-11 172.168.x.111 192.168.x.200 Stand-alone Server
<Host>-12 172.168.x.112 192.168.x.200 Stand-alone Server
<Host>-13 172.168.x.113 192.168.x.200 Stand-alone Server
<Host>-14 172.168.x.114 192.168.x.200 Stand-alone Server
<Host>-15 172.168.x.115 192.168.x.200 Stand-alone Server
<Host>-16 172.168.x.116 192.168.x.200 Stand-alone Server
Configuring Your Computer(s)
Each student requires one physical machine with a fully configured virtual machine
environment. Before starting this lab, make sure your computer is configured as follows:
<Operating System and Version> installed and started
Troubleshooting Lingering Objects DRAFT V9.3 Lab Sessions
74 © 2011 Microsoft Corporation. All rights reserved.
Virtual Server 2005 R2 SP1 (may be preinstalled on classroom computers)
Note: Virtual Server 2005 R2 SP1 may already be installed on your computer. If these applications are not installed, you may obtain a free download of the installation files from:
http://www.microsoft.com/downloads/details.aspx?FamilyID=bc49c7c8-4840-4e67-8dc4-1e6e218acce4&DisplayLang=en
Windows Server 2008 DVD media or installation ISO file in <path>.
Virtual machines installed or created on the computer:
o <VMName>: <OS | Role | description>
o <VMName>: <OS | Role | description>
o <VMName>: <OS | Role | description>
Course files located in the C:\Labfiles and C:\VS folders on your computer or
accessible from a network share on the instructor computer.
Accounts and Group Membership
Important: You must log on as an administrative user in order to perform some of the tasks in this lab.
The following user accounts and passwords must be configured on the physical computer and
in all virtual machines:
Administrative username and password
Username: Administrator Password: LS1setup!
Member of: Local Administrators
Normal username and password
Username: Studentn Password: LS1setup!
Member of: Local Users
Replace n in Studentn with the number assigned to your classroom computer by the
instructor.
Domain Membership
Your physical computer is not joined to a domain.
Lab exercises may require you to join the following virtual domain(s):
Contoso.com
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 75
Virtual machines joined to a virtual domain require group and account configurations as
shown in the following table.
Table 9: Groups and Accounts
Group Members
Domain Groups
Domain Administrators Administrator
Domain Users Studentn
Local Groups
Administrators Administrator, Domain Administrators
Users Studentn, Domain Users
Shares on Instructor Computer(s)
During lab exercises, you may be required to access the following shares on the instructor
computer(s):
\\<Host>-Instr-1\Labfiles
Includes lab files installed on participant host computers
\\<Host>-Instr-1\VS
Includes files required for virtual machine environment
Using the Keyboard and Mouse in a Virtual Machine This course includes virtual machines for lab exercises. You use the keyboard and a mouse to
control a virtual machine much as you would a physical computer. This section explains how
to use the keyboard and mouse in virtual machines and describes special keys and menu
items.
Using the Keyboard
In general, the keyboard works the same for a virtual machine as it does for a physical
computer. However, some keyboard shortcuts such as Ctrl+Alt+Delete do not work within a
virtual machine because of the interaction between the host operating system and the guest
operating system. Virtual Server 2005 provides much of the required keyboard functionality
with a Host key and keyboard shortcuts. By default, the Host key is the Right-Alt key. You
can use the Host key in two ways:
If a virtual machine has captured the pointer, press the Host key to return control of the
mouse to the host operating system.
Troubleshooting Lingering Objects DRAFT V9.3 Lab Sessions
76 © 2011 Microsoft Corporation. All rights reserved.
Use the Host key in combination with other keys for specific functions as described in
the following table.
Table 10. Keyboard Shortcuts for Virtual Machines
Key Combination Description
Host Key+Delete Sends Ctrl+Alt+Delete functionality to the virtual machine operating system.
Host Key+C Connects the Remote Control or VMRC to the VMRC server.
Host Key+A Switches the Remote Control or VMRC to the Administrator Display.
Host Key+I Displays connection information.
Host Key+V Sets the virtual machine so that the guest operating system cannot be manipulated. You can only view the virtual machine window.
Host Key+H Displays the control to set the Host key.
Host Key+Enter Switches the virtual machine window to full-screen display. This option is available only when you connect to a virtual machine using the VMRC client.
Host Key+Left Arrow Switches to the previous virtual machine. This option is available only when you connect to a virtual machine using the VMRC client.
Host Key+Right Arrow Switches to the next virtual machine. This option is available only when you connect to a virtual machine using the VMRC client.
Tip: As shown in the preceding table, you can use Host Key+Delete to send the functionality of the Ctrl+Alt+Delete keyboard shortcut to a guest operating system running in a virtual machine. You can also use Send Ctrl+Alt+Del from the Remote Control menu of either the VMRC or Remote View page.
Using the Mouse
The way you use the mouse depends on whether Virtual Machine Additions is installed.
If Virtual Machine Additions is installed on the virtual machine, you can move the
pointer freely between the virtual machine window and the host operating system.
This simplifies switching among virtual machines and the host operating system.
If Virtual Machine Additions is not installed on the virtual machine, the virtual machine
must capture the pointer before the mouse can be used within the virtual machine
window. The virtual machine captures the pointer when you click the pointer inside
the virtual machine window.
If a pointer is captured by a virtual machine on which Virtual Machine Additions is not
installed, the virtual machine must release it before you can use the mouse on the host
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 77
operating system or in another virtual machine window. You can use the Host key to return
the use of the mouse to the host operating system.
Troubleshooting Lingering Objects DRAFT V9.3 Lab 1: Exploring Lingering Object Fundamentals
78 © 2011 Microsoft Corporation. All rights reserved.
Lab 1: Exploring Lingering Object Fundamentals During this lab, you will identify the forest's configured tombstone lifetime and replication
consistency settings
Estimated time to complete this lab: 15 minutes
Before You Begin
To complete this lab:
Complete Lesson 1
What You Will Learn
After completing this lab, you will be able to determine the Active Directory settings that
govern how it handles tombstones and lingering objects.
Exercise 1 Determine tombstone lifetime.
Exercise 2 Determine DC replication consistency setting.
Scenario
You are assisting a customer who is having issues with…
Configuring Your Computer(s) Each student requires at least one physical computer and a fully configured local or remote
hosted virtual machine environment. Before starting this lab, make sure your computer is
configured as described in About This Lab.
Configuring Your Virtual Machine Environment
Exercises in this Lab require the following virtual machines:
VMname: DC1
Exercises may also require files located in the C:\Labfiles folder on your computer or
accessible from a network share on the instructor computer.
Accounts and Group Membership
Important: You must log on as an administrative user in order to perform some of the tasks in this lab.
The following user accounts and passwords must be configured on the physical computer and
in all virtual machines:
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 79
Administrative username and password
Username: Administrator Password: LS1Setup!
Member of: Local Administrators
Normal username and password
Username: Usern Password: LS1Setup!
Member of: Local Users
Replace n in Usern with the number assigned to your classroom computer by the instructor.
Domain Membership
Lab exercises may require virtual machines to be joined to the following virtual domain(s):
Contoso.com
Virtual machines joined to a virtual domain require group and account configurations as
shown in the following table.
Table 11. Groups and Accounts
Group Members
Domain Groups
Domain Administrators Administrator
Domain Users Usern
Local Groups
Administrators Administrator, Domain Administrators
Users Usern, Domain Users
Exercise 1: Determine Tombstone Lifetime Setting In this exercise, you will attempt to determine the tombstone lifetime setting of the forest.
Scenario
You are assisting a customer that is having issues
Task Detailed Steps
Complete these steps by connecting to DC1
Troubleshooting Lingering Objects DRAFT V9.3 Lab 1: Exploring Lingering Object Fundamentals
80 © 2011 Microsoft Corporation. All rights reserved.
Task Detailed Steps
Task Description 1. Step.
a. Sub-step.
Setting | Parameter Value
Item 1
Item 2
b. Sub-step.
c. Sub-step.
2. Step.
Task Description 1. Step.
a. Sub-step.
b. Edit the registry as shown below:
Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1
c. Sub-step
2. Step.
Task Description 1. Step.
a. Sub-step.
b. Sub-step.
2. Step.
Task Description 1. Step.
a. Sub-step.
b. Sub-step.
2. Step.
Review
1. <Question>
Answer
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 81
2. <Question>
Answer
Exercise 2: Determine forest and DC replication consistency settings
In this exercise, you will identify the replication consistency settings for each DC in the environment and will determine if the f
Scenario You are assisting a customer that is having issues <add scenario here>.
Task Detailed Steps
Complete these steps by connecting to <VM name>
Task Description 3. Step.
a. Sub-step.
Setting | Parameter Value
Item 1
Item 2
b. Sub-step.
c. Sub-step.
4. Step.
Task Description 3. Step.
a. Sub-step.
b. Edit the registry as shown below:
Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1
c. Sub-step
4. Step.
Task Description 3. Step.
a. Sub-step.
b. Sub-step.
4. Step.
Troubleshooting Lingering Objects DRAFT V9.3 Lab 1: Exploring Lingering Object Fundamentals
82 © 2011 Microsoft Corporation. All rights reserved.
Task Description 3. Step.
a. Sub-step.
b. Sub-step.
4. Step.
Review 3. <Question>
Answer
4. <Question>
Answer
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 83
Lab 2: Lingering Object Diagnosis and Documentation
During this lab, you will generate diagnostic data via repadmin, ldifde and replfix. You will
then analyze that data and document all lingering objects in the environment.
Estimated time to complete this lab: 30 minutes
Before You Begin
To complete this lab:
Complete Lesson 1 and Lesson 2
What You Will Learn
After completing this lab you will be able to <Lab terminal objective>.
After completing the exercises you will be able to:
Exercise 1 enabling objective.
Exercise 2 enabling objective.
Scenario
You are assisting a customer who is having issues with…
Exercise 1: Lingering Object Diagnosis <Briefly describe the goal of the exercise>
Scenario
You are assisting a customer that is having issues <add scenario here>.
Tasks
<Define starting conditions, including virtual machines and lab files required>.
1. <Task>.
a. <Step>.
i. <Sub-step>.
ii. <Sub-step>.
b. <Step>.
2. <Task>.
Troubleshooting Lingering Objects DRAFT V9.3 Lab 2: Lingering Object Diagnosis and Documentation
84 © 2011 Microsoft Corporation. All rights reserved.
Setting | Parameter Value
Item 1
Item 2
3. <Task>.
a. <Step>.
Edit the registry as shown below:
Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1
b. <Step>.
Review
1. <Question>
Answer
2. <Question>
Answer
Exercise 2: Lingering Object Documentation <Briefly describe the goal of the exercise>
Scenario
You are assisting a customer that is having issues <add scenario here>.
Tasks
<Define starting conditions, including virtual machines and lab files required>.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 85
4. <Task>.
a. <Step>.
i. <Sub-step>.
ii. <Sub-step>.
b. <Step>.
5. <Task>.
Setting | Parameter Value
Item 1
Item 2
6. <Task>.
a. <Step>.
Edit the registry as shown below:
Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1
b. <Step>.
Review
3. <Question>
Answer
4. <Question>
Answer
Troubleshooting Lingering Objects DRAFT V9.3 Lab 3: Lingering Object removal using repadmin
86 © 2011 Microsoft Corporation. All rights reserved.
Lab 3: Lingering Object removal using repadmin During this lab, you will remove lingering objects from the environment using repadmin
/removelingeringobjects.
Estimated time to complete this lab: 30 minutes
Before You Begin
To complete this lab:
Complete lessons 1-4
What You Will Learn
After completing this lab you will be able to remove lingering objects using repadmin.
After completing the exercises you will be able to:
Exercise 1 enabling objective.
Exercise 2 enabling objective.
Scenario
You are assisting a customer who is having issues with…
Exercise 1: <Problem Solving Exercise Title> <Briefly describe the goal of the exercise>
Scenario
You have completed recovering files from a back up and now need to restore the files.
Tasks
<Define starting conditions, including virtual machines and lab files required>.
1. <Task>.
a. <Step>.
b. <Step>.
Setting | Parameter Value
Item 1
Item 2
2. <Task>.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 87
Sample solution
Your result should look something like the Sample in <Lab Title>, <Exercise Title> in the Lab
Manual that accompanies this course.
For step by step instructions, see <Lab Title>, <Exercise Title> in the Lab Manual that
accompanies this course.
Review
1. <Question>
Answer
2. <Question>
Answer
Exercise 2: <Simulation Exercise Title> <Briefly describe the goal of the exercise>
Scenario
You have received email from your manager requesting a maintenance action.
Tasks
1. Read Email from your manager explaining the situation.
<Add email text here>
2. Review supporting documents in <local path>:
a. Company organization chart.
b. Company ____ data.
c. Report on problems with the ____ system.
Troubleshooting Lingering Objects DRAFT V9.3 Lab 3: Lingering Object removal using repadmin
88 © 2011 Microsoft Corporation. All rights reserved.
3. Open the VM containing the company system and resolve the issues.
Sample solution
Your result should look something like the Sample in <Lab Title>, <Exercise Title> in the Lab
Manual that accompanies this course.
For step by step instructions, see <Lab Title>, <Exercise Title> in the Lab Manual that
accompanies this course.
Review
1. <Question>
Answer
2. <Question>
Answer
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 89
Lab 4: Lingering Object removal using ldp and repldiag
During this lab, you will remove a single lingering object using ldp. You will then remove the
remaining lingering objects using repldiag.
Estimated time to complete this lab: 30 minutes
Before You Begin
To complete this lab:
Complete <list lesson(s) etc.>.
What You Will Learn
After completing this lab, you will be able to <Lab terminal objective>.
Exercise 1 enabling objective.
Exercise 2 enabling objective.
Scenario
You are assisting a customer who is having issues with…
Troubleshooting Lingering Objects DRAFT V9.3 Lab 5: Abandoned Object and Abandoned Deleted object remediation
90 © 2011 Microsoft Corporation. All rights reserved.
Lab 5: Abandoned Object and Abandoned Deleted object remediation
During this lab, you will identify and remove an abandoned object. You will then remediate
and abandoned deleted object scenario.
Estimated time to complete this lab: 30 minutes
Before You Begin
To complete this lab:
Complete 1-4
Configure | verify your lab environment:
o Virtual machines <VM name(s)> installed and configured.
o <Application name> installed and configured.
o <List and link to specific lab files if needed>.
What You Will Learn
After completing this lab, you will be able to <Lab terminal objective>.
Exercise 1 enabling objective.
Exercise 2 enabling objective.
Scenario
You are assisting a customer who is having issues with…
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 91
Lab 6: Lingering Link identification and cleanup During this lab, you will identify all lingering-linked values in the environment. You will them
remove them in order to ensure group membership consistency.
Estimated time to complete this lab: 45 minutes
Before You Begin
To complete this lab:
Complete lessons 1-4
Configure | verify your lab environment:
o Virtual machines <VM name(s)> installed and configured.
o <Application name> installed and configured.
o <List and link to specific lab files if needed>.
What You Will Learn
After completing this lab, you will be able to <Lab terminal objective>.
Exercise 1 enabling objective.
Exercise 2 enabling objective.
Scenario
You are assisting a customer who is having issues with…
Troubleshooting Lingering Objects DRAFT V9.3 10.0 Presentation Slides
92 © 2011 Microsoft Corporation. All rights reserved.
10.0 Presentation Slides
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 93
Troubleshooting Lingering Objects DRAFT V9.3 10.0 Presentation Slides
94 © 2011 Microsoft Corporation. All rights reserved.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 95
Troubleshooting Lingering Objects DRAFT V9.3 10.0 Presentation Slides
96 © 2011 Microsoft Corporation. All rights reserved.
DRAFT V9.3 Active Directory Replication Troubleshooting
[email protected] Microsoft Corporation 97
Troubleshooting Lingering Objects DRAFT V9.3 10.0 Presentation Slides
98 © 2011 Microsoft Corporation. All rights reserved.