A Mixed Reality Virtual Cloths Try-On SystemPegah hamidkhani Student no:92131562
ECS PresentationThought by: Dr. Alireza Hashemi
Intro• Physical try-on of cloths is a time consuming procedure in retail shopping• Virtual try-on can help to speed-up the process by narrowing down selections• Enhancement of user experience through new features• Side-by-side comparison of various cloths• Simultaneous viewing of outfits from different angles
• It can also be an interesting feature of digital signage for advertisement and/or attracting crowds
Ray Ban Virtual mirror
Overview
In this presentation we will:• Describe challenges in virtual trying-on• Describe 3 virtual try on scenarios of the system• Automatic avatar customization and skin tone mapping
algorithms• A novel method for alignment of a 3D avatar with the user’s
2D image• The implementation details with experimental results• User study on this concept
Challenges
• Accurate alignment and scaling of cloths• Different Body styles• Fast rotating of user and following user movements• Fast algorithm is needed in Real-time environments• The cloths worn by the user remain visible• 3D modeling of cloths is time and effort consuming• Simulation of various garment types is almost impossible• This is a modern technology and customer acceptance
needs time
Virtual Try-On Scenarios• Avatar Only (AO): Virtual cloths on an avatar• Dress Only (DO): Virtual cloths on a user’s image• Hybrid Version (HV): Virtual cloths on an avatar blended with a
user’s face image
Avatar Only(AO) Scenario• A generic 3D avatar is customized
based on user’s body size and its skin color is matched to the user’s face skin color
• Use a novel algorithm to align the 3D customized avatar with user’s image in real-time
• Use simulation for animating cloth (Virtual garment)
• Remove the user’s image from screen and replace with clothed avatar
• This follows the user’s movement
Dress Only(DO) Scenario• A generic 3D avatar is customized
based on user’s body size and its skin color is matched to the user’s face skin color
• Use a novel algorithm to align the 3D customized avatar with user’s image in real-time
• Use simulation for animating cloth (Virtual garment)
• 3D virtual cloths are augmented on the user’s image without displaying the avatar
• This follows the user’s movement
Hybrid Version(HV) Scenario• A generic 3D avatar is customized
based on user’s body size and its skin color is matched to the user’s face skin color
• Use a novel algorithm to align the 3D customized avatar with user’s image in real-time
• Use simulation for animating cloth (Virtual garment)
• We segment out the user image below the neck and replace it by a reconstructed background
• This follows the user’s movement
Body Customization• Why ?• It is much economical in terms of time and effort instead of creating
model from scratch• How?• An accurate avatar can be created based on twelve key human body
measurements• Height, shoulder width, bust girth, waist girth, hip girth, thigh girth, ankle
girth, waist height, crotch height, knee height, upper arm length and forearm length
• Algorithm• Scale the model globally according to the user’s height• Scale the torso and the legs along the y-axis based on the user’s waist,
crotch and knee height.• Modify the torso and legs based on the user’s shoulder width, bust, waist,
hip, thigh and ankle girths• Modify the arm based on the user’s upper arm and the forearm length
Skin-Tone Matching• We use the user’s actual face skin color to adaptively change
the avatar’s body skin color• Steps:• Facial features are located using the active shape model (ASM)• Use linear curves to represent the cheek areas and extract cheek
patches• Apply a global color transfer method to shift the color of the face
patches to the avatar body• Problems• Different viewing and lighting conditions• Cloths or hair with similar color to face• Misclassified as skin area (lips, eyebrows,… )• Highlights in the forehead, nose and chin areas
Skin-Tone Matching• Cheek area is the largest flat skin area on the face and is least
affected by shadows• To detect cheek:• To detect face 76 landmarks are marked• Here we use 20 landmarks and their connection lines to enclose
right and left cheek
Align 3D avatar with 2D user image
• In a virtual try-on system, accurate alignment between a 3D clothed avatar with a 2D user image stream is of crucial importance
• One way is to use the transformation matrix but is prone to misalignment errors for other body parts
• So we ask the user to stand in a standard pose at the beginning for scaling and alignment (Key Frame)
• To map 2D image point (m) and 3D avatar point(M) we have
ProjectionMatrix
ArbitraryFactor
Robust 3D-2D alignment• Being real-time needs good performance• To improve robustness and smoothness we should we should
have more 3D-2D point correspondence • So we need to establish the 2D-2D correspondence in real-time
between the current frame j and the key frame• We also use Learning-based matching method which is fast and
have good Performance
Key frame Key frame
Current Try-On System Overview
• Automatically alignment of avatar with the user’s pose
• Skin-Tone is Matched• Three scenarios are experimented• Avatar Only (AO)• Dress Only (DO)• Hybrid Version (HV)
• Alignment based on shoulders• Because the method is based on the
information from the current frame so there will be no accumulated errors over time
• It works well as long as the RGB-D camera is able to detect the user poses
Measuring the accuracy• 10 female experimented for each of 10 Virtual garments• Compared mean average error for different garments in x and
y axis• Compared standard deviation for different garments in x and y
axis• We can see average error in X axis is more than Y because we aligned Y axis based on shoulders 5.2>>0.30• So By using this algorithm we can align more accurate in Y direction
Implementation Details• Visual Studio 2010• 2.53GHz Intel Xeon(R) with 24 GB RAM• A Kinect Camera for pose detection, body measurements, user
segmentation and face skin color detection
Try-On System In Action• First the user stands in front of a display• The system establish relevant 3D-2D correspondences based
on a key frame• The user’s body size and the user’s face skin color are
extracted using the Kinect camera• The User can key-in more body size for a more accurate avatar
customization• The user can select her favorite virtual cloths for virtual try-on• The selected virtual clothes will be aligned on the user’s
image, simulated and rendered in real-time• The user can see the virtual trying-on results with various
clothes from different angles based on movements
Try-On System In Action• Average computation time for each frame is about 110
milliseconds• Time consuming stages
• But the rendering time largely depend on the complexity of cloth and avatar
• The background reconstruction algorithms utilizes the user detection results of RGB-D camera and replaces the detected user image by pre-captured background image
Background reconstruction 3D-2D Alignment Rendering
47 0.6 63
User Study Design
Evaluates the effectiveness of 3 virtual try-on solution about:• Quality Attributes (QA): assist a
purchase decision• Reliability, Accuracy, User centric
issues• Cognitive Attributes (CA): attributes
concerning the mental processes• Attention, learning, decision making,
and emotive elements• Attributes Toward Using (ATU):
attributes resulting from a presence of perceived ease of use
Questionnaire
User Study ResultsThe results show that:• The user responded positively to the DO and HV versions• All three versions were perceived positively• ATU is most poplar for DO version• People prefer to see their face and body while trying-on and think its more
realistic and gives them the sense of shopping• The average score for DO is the most• The average score for AO is the least• But the dislike for DO version was mainly because they can see what they wear
underneath
Discussion• The HV solution provides more realistic than others• The DO version is most preferred by the user• Some factors affect the performance• Large Body Rotation: People like to rotate and see the result from
different angles. But large rotation can not be detected reliably• Solution: Use multiple RGB-D cameras
• Body Customization: RGB-D sensor is not good enough for body measurement• Solution: A fast method for measurement is still an open problem
• Skin Tone Mapping: A one-time procedure is used to change the avatar’s skin color. But the participants detected the difference• Solution: Despite the existence of some techniques it is still an open
problem on how to connect the user’s face to the avatar’s neck without notice
Conclusion• A mixed reality based virtual clothes try-on system described• Series of novel techniques for virtual try-on was proposed• Three scenarios of virtualization displayed• Virtual cloths on the avatar• Virtual cloths on the actual user’s image• Virtual cloths on the avatar blended with the user’s face image
• The major contribution• Automatically customized an invisible (or partially visible) avatar
based on the user’s body size• A user study was also conducted to evaluate effectiveness• The result showed that it can help in customer’s purchase
decision
Reference• Miaolong Yuan; Khan, I.R.; Farbiz, F.; Susu Yao; Niswar, A.; Min-
Hui Foo, "A Mixed Reality Virtual Clothes Try-On System," Multimedia, IEEE Transactions on , vol.15, no.8, pp.1958,1968, Dec. 2013
Top Related