Sensor Selection and Stage & Result Classifications forAutomated Miniature Screwdriving

Xianyi Cheng, Zhenzhong Jia, Ankit Bhatia, Reuben M. Aronson, and Matthew T. Mason Fellow, IEEE

Abstract— Hundreds of billions of small screws are assem-bled in consumer electronics industry every year, yet reliablyautomating the screwdriving process remains one of the mostchallenging tasks. Two barriers to further adoption of roboticthreaded fastening systems are system cost and technicalchallenges, especially for small screws. An affordable intelligentscrewdriving system that can support online stage and resultclassification is the first step to bridge the gap. To this end,starting from a state transition graph of screwdriving processesand a labeled screwdriving dataset (1862 runs of M1.4 screws)on multiple sensor signals, we develop classification algorithmsand perform sensor reduction. Fast and accurate result classi-fiers are developed using linear discriminant analysis, while awrapper method for feature subset selection is used to identifythe optimal feature subset and corresponding sensor signals toreduce cost. A stage classifier based on decision tree is developedusing the optimal sensor subset. The stage classifier achieveshigh accuracy in realtime prediction of various stages whenaugmented with the state transition graph.

I. INTRODUCTION

Threaded fastening is one of the most commonly usedmethods in industrial assembly [1]. Around 1/4 to 1/3of typical assembly operations can be classified as boltand nut insertions [2] [3] [4]. Unfortunately, screwdrivingremains one of the most difficult tasks to automate, despitesubstantial research in this field. One reason might be dueto our incomplete understanding of the underlying process,particularly the initial mating step [5]. Our survey paper [1]summaries various open problems and barriers that confrontautomated screwdriving systems. Four major improvementsneed to be made: (1) fast and reliable ways to feed screwswith smaller length-to-diameter aspect ratios; (2) strategiesfor fast and reliable initial thread mating and early faultdetection; (3) interactions of multiple objects (screw, driverbit, vaccum adapter, and target); (4) online failure predictionand fault recovery algorithms.

Automated screwdriving becomes even more challengingwhen it comes to the consumer electronics industry (e.g.,laptops, tablets, and smartphones), where hundreds of bil-lions of small screws (≤ #4 or ≤M3) are assembled everyyear [6]. In fact, it is one of the most challenging operationsthat prevent manufacturing enterprises from further adoptingautomated robotic systems [6]. Many errors can occur duringscrewdriving. Even a very small fraction of assembly failurescan cause serious consequences. For example, a loose screwcan damage the battery inside a laptop, causing overheat-ing and posing a fire hazard [7]. Small screws introduce

The authors are with the Robotics Institute, Carnegie Mellon University,Pittsburgh, PA 15213, USA. {xianyic, zhenzjia, ankitb,rmaronson, mattmason}@cmu.edu

additional challenges and design considerations [8] [1]: (1)tighter tolerances for screw feeding and acquisition; (2)higher positioning accuracy and improved locating strategiesfor misalignment correction; (3) accurate and affordablescrewdrivers with online fault prediction and recovery

Automated screw fastening involves multiple steps, oftenincluding screw feeding and acquisition, alignment, screw-driving, and post-fastening steps [1]. Comprehensive reviewsof threaded fastening, including theoretical fundamentals,tools, control strategies, failure detection and industrial ap-plications, can be found in [9], [10], and [1]. Among varioussteps, most works focus on the screwdriving process, wherea properly acquired screw is driven into the target hole. Oneway to understand the screwdriving process is by plottingapplied torque against the total rotation angle to producethe torque-angle curve [1]. This curve has been used forISO rotary tool evaluation standards [11], control strategies,and failure detection [12]. The screwdriving process canbe further divided into three major sub-steps: initial threadmating, rundown, and tightening with some variations orextra steps for self-tapping screws [1].

While most of the literature focuses on big screws (seeSection II), this paper focuses on the miniature screwdrivingprocess. Besides aforementioned technical challenges andreliability issues, one factor that prevents further adoptions ofautomated screwdriving systems is cost, as will be discussedin Section II. Hence, one goal of this paper is to select propersensor signals to produce affordable intelligent screwdriversthat can be deployed to the actual assembly line for consumerelectronics products. The screwdriver should perform bothonline result (see Section IV) and stage (see Section V)classifications, the very foundations to build fault predictionand recovery system for reliable screwdriving [12].

To summarize, our contributions are:

• A systematic way to identify optimal sensor readings toproduce low-cost intelligent screwdriving system.

• Fast algorithms for online stage and result classificationsfor threaded fastening. To our knowledge, this is the firstattempt for screwdriving stage classification.

• Compared with the literature, our evaluations are per-formed on a much larger screwdriving dataset with richresult and stage information.

II. RELATED WORK

One can refer to our survey paper [1] for a complete reviewof automated threaded fastening. In the following, we brieflydescribe some work that are directly related to this paper.

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)Madrid, Spain, October 1-5, 2018

978-1-5386-8093-3/18/$31.00 ©2018 IEEE 6078

A. Sensor Selection for Screwdriving Systems

In previous screwdriving literature, the sensor selectionand control strategies are mainly empirical, i.e., based onyears of engineering practice and expert experiences. For ex-ample, many COTS (commercial off-the-shelf) screwdrivers[13] are equipped with sensors to measure or estimate thefastening torque and rotational angle, because these two pa-rameters can provide valuable information for screwdriving,especially for fault detection. Many control strategies alsoinvolve in monitoring the torque and/or angle information.These strategies can be divided into three broad categories:torque-only control, torque-angle monitoring and control,and torque-rate control [10] [1].

However, one problem associated with these COTS sys-tems is that their sensor signals (torque and angle) may notbe rich enough for controls and fault detection in roboticscrewdriving. For example, robust detection of initial matingis critical for torque-angle control because it triggers theclamping angle count — an important parameter for qual-ity control. Many high-end COTS screwdrivers (e.g., [13])compare the driving torque against user-specified thresholdto detect the initial engagement. However, as shown in Fig.2,the driving torque (Tz) is not a robust feature because itis almost constant during initial mating. In comparison,the insertion force (Fz) might be a much better candidate.Another limitation is that fault detection algorithms based ontorque-angle signals alone cannot detect less-common failuremodes, as shown in [12] [14].

There are some robotic screwdriving systems that canprovide richer sensor readings. In [15], the insertion force(Fz) can be calculated by measuring the spring displacement.In [16] and [12], 6-axis force/torque (F/T) sensors are used toprovide much richer information, at the expense of significantincrease in system cost. In fact, besides reliability issues, cost(especially the sensor cost) is another factor that preventsfurther adoption of robotic screwdriving. For example, high-end screwdrivers [13] can cost more than $10,000, becausea typical 6-axis F/T sensor [17] cost around $7,000 or evenmore. In this paper, starting from a much richer dataset, wefollow a systematic feature reduction approach to identify theoptimal subset of sensors for both online stage and resultclassifications (the building blocks for fault prediction andrecovery system) to minimize system cost.

B. Fault Detection and Quality Monitoring

Reliable fault detection and error recovery are requiredby autonomous screwdriving, because even well-engineeringsystems can be tripped up by factors like part tolerance is-sues, bad material, and tool wear. One of the most commonlyused method is the teaching method, in which faults typicallyshow up as major deviations from the correct torque-anglefastening signature curve. To overcome the inflexibility andgenerality issues of the teaching method, fault diagnosismethods based on artificial intelligence, soft computing,model-based fault detection, and fuzzy systems have alsobeen developed [1]. For example, artifical neural networks(ANNs) and support vector machines (SVMs) have been

Fig. 1. Screwdriving experiment setup and the instrumented screwdriver.

investigated in [16]. Most methods mentioned above canachieve around 90% or better accuracy in detecting differentfailure modes. However, many things need to be improvedbefore deploying to the actual assembly line.

Fault detection alone cannot satisfy the stringent require-ments for high-volume production [1] in consumer elec-tronics industry. To improve the overall success rate, faultprediction and recovery algorithms that can detect precedingfailures earlier and take proper correcting actions uponpredicted failure types are desired for future assembly lines[1] [12]; unfortunately, they are still missing in the literature.Our stage classification algorithm developed in Section V isthe first step to bridge the gap.

III. DATA COLLECTION AND ANALYSIS

Fig. 1 shows the robotic screwdriving system we buildfor data collection. An instrumented screwdriver is installedon a 6-dof industrial robot to perform the following oper-ations repetitively: picking up screws (M1.4× 4 Phillips-head screws for cellphones) from the feeder (shaker tray)using vacuum suction, moving towards the threaded plate,aligning with the precalibrated target, and inserting screwsinto the target holes. Each run terminates when either themotor current or motor angle reaches a specified threshold.The system then proceeds to the next run. Another robotholds a high-speed camera with manual-focusing macro lensto record the operation.

A 6-axis force/torque (F/T) sensor (model: ATI mini-40,calibration: SI-20-1) is integrated into the screwdriver. Therobot frame of the F/T sensor is connected to the robotflange through a linear compliance unit, while other parts ofthe screwdriver, including the motor, vacuum head, and thedriver-bit, are essentially connected to the F/T tool frame.Compared with the wrist-mount F/T design in [16], our“floating structure” design is more suitable for miniaturescrewdriving because the F/T sensor is much closer to thescrew and thus can provide more accurate data. Detaileddescriptions and discussions of our data collection systemand experiment procedures can be found in [12] [14].

In [12], we collected a total of 1862 screwdrving runs,each of which consists of 6-axis force and torque (see Fig. 2),

6079

Fig. 2. The 6-axis force and torque signatures of (a) successful case withhole finding stage and (b) an unsuccessful (crossthread) case.

motor current and speed, and video data (see Fig. 3), allsampled at 100 Hz. After data analysis, we empirically cameup with a list of stages (through which the screwdrivingoperation progresses) and result classes, as shown in Fig. 3.A complete description of the stages and result classes canbe found in [12]. Our dataset has also been hand-labeled withcorresponding stage (see Fig. 2) and result information foreach run, forming the largest screwdrving dataset we knowof [12]. All of our subsequent work is performed on thisdataset.

Our result classes contain outcomes common in the litera-ture and less-studied cases, such as those related to strippedscrews. The stages shown in Fig. 3, though, represent the firstdivision of screwdriving into stages, to our knowledge. Asdiscussed in [12], “the stages provide a deeper understandingof the underlying operation, which can be applied to identifyprocess failures that would be missed through simple resultclassification.” For example, the hole finding stage, definedas “screw has touched plate but not yet fallen into hole” [12],might damage crucial parts (e.g., PCB boards) in electronicsproducts. But this stage cannot be detected by a resultclassifier. On the other hand, a complete list of stages that theoperation passes is sufficient to predict the result. Moreover,as explained in [12], it is possible to build a failure predictionand recovery system based on the stage information, therebyimproving the overall performance of the assembly line.

IV. RESULT TYPE CLASSIFICATIONS

Classification of the result types is important to fully au-tomated screwdriving system [16]. This helps to identify thecause of failures. And based on correct predictions, propercontrol/recovery strategies can be developed to improvethe performance. Meanwhile, we should choose affordablesensors instead of expensive ones (e.g., 6-axis F/T sensor)to develop low-cost systems that can be deployed to theassembly line.

To this end, fast and accurate result classification is per-formed by linear discriminant analysis (LDA) model [18] inSection IV-B. A wrapper method for feature subset selection

[19] is used to reduce the number of required featuresand sensor signals in Section IV-C. Finally, reduced featuresubsets (see Table I and Table II) from less sensor signals areselected to produce a highly accurate yet affordable roboticscrewdriving system.

A. Preprocessing of Time Series Data

In Section III, we collected and labeled multivariate timeseries data. In this paper, data from force-torque sensor andmotor is used for classification. The video data (see Fig. 3)only serves as the ground truth. In our dataset, the time lengthof data varies among different runs. Classifying them can betricky and computationally expensive [20], thus we use someglobal statistic characteristics of the captured data as featuresto perform result classification.

For each screwdriving run, 85 global features are extractedfrom the whole length of time series data from full sensorsignals (8 channels in total). The angle feature (θ ) is providedby the last motor encoder reading, which is commonly usedin industry as an important criterion for failure detection, asdiscussed in Section II-B. The other 84 features are based onsignals from the 6-axis F/T sensor and motor current, i.e., 7channels in total. For each channel, 12 statistic features areextracted, including the range, mean and standard deviationof the following time series data: the original data, its firstorder differences (∆), its second order differences (∆2), andits successive ratio (∆dd). Finally, each feature is normalizedto have a standard normal distribution.

B. Linear Discriminant Analysis for Result Classification

Given 1862 labeled samples and 85 normalized features,a multi-class linear discriminant analysis (LDA) model isapplied to predict the result classes shown in Fig. 3. Theclassification result is also used as a benchmark for the sensorreduction work described in Section IV-C.

Multi-class LDA is a simple but powerful classificationalgorithm that can be used to separate multiple classes. TheLDA has the advantages of giving linear decision boundariesand requiring less computations. In this method, a set ofobservation x is classified as the class y that has the largestposterior probability P(Y = y|X = x) among all K classes.By Bayes’ theorem, this can be written as

y = argmaxy=1,...,K

P(X = x|Y = y)P(Y = y)

where P(X = x|Y = y) is probability density function of xand P(Y = y) is prior probability of class y. For densityfunctions of all the classes, LDA assumes that they arenormally distributed with the same covariance Σ but differentmeans µ1, . . . ,µK , where µ1, . . . ,µK , and Σ can be directlyestimated from the training data. By maxmizing the posteriorprobability, a linear decision boundary can be found for eachpair of the classes.

Wit 85 global features, this multi-class LDA methodachieves an average classificaton accuracy of 98.93% in 10-fold cross-validation. In comparison, our previous GTC-DFmodel [12] (Graph of Temporal Constraint Decision Forest)

6080

Fig. 3. Left: The state transition graph summarizing all the stage and result classes. The vertices represent stages through which the screwdriving passes,with the terminal stages corresponding to the result classes. Colors and sizes are scaled logarithmically with the number of runs in each transition, stage,or result (modified from [12]). Right: A successful screwdriving run example including a hole finding stage.

Fig. 4. Confusion matrix for the LDA model trained with 85 features.

trained on the same dataset with 144 features achieves a99.03% accuracy. But this LDA model is much simpler thanthe GTC model. The confusion matrix of the LDA modelis shown in Fig. 4. We see that the LDA method achieveshigh accuracy in predicting success, no screw, crossthread,and stripped result classes. However, it performance is notideal for the other result classes. For stripped no engageand partial, the reason is that we have very few runs fromthese classes. For the no hole found class, there are severaltypes of signals that are quite similar to those of the otherclasses, as shown in Fig. 5 and Fig. 6. The overall trends aresimilar, while there exist local differences such as different

Fig. 5. Similar F/T signals might cause a misclassification between a nohole found run and a stripped no engage run.

oscillation periods. Since the result classifiers are trainedwith global features, some meaningful local characteristicsmight be ignored. To improve the performance, more datafor the rare failure types and better understanding of no holefound are required.

The results in this part can serve as the baseline fordeveloping more complex algorithm. Note that other clas-sical machine learning methods are also applicable for theresult classification in this paper, such as logistic regression,quadratic discriminant analysis and SVMs. All these clas-sifiers yield similar results, except that the accuracy of theLDA is slightly better.

C. Sensor Reduction through Feature Selection

In this part, sensor reduction for more economical screw-driving system is achieved by feature reduction. We followa standard feature selection approach to reduce the numberof required features and sensor signals. Feature selectionrequires a search algorithm to select candidate feature subsetsand an objective function to evaluate these candidates [21].

6081

Fig. 6. Similar F/T signals might cause a misclassification between a nohole found run and a stripped run.

In our study, the sequential backward selection (SBS) [19]is chosen as the search algorithm. Starting from the fullfeature set, SBS sequentially removes the feature that leastreduces the value of the objective function. SBS providesa systematic way to sort the original features according totheir importance. Note that alternatives such as bidirectionalsearch might give better results; SBS is chosen here due toits simplicity. For subset evaluation, we use the wrapper ap-proach. In the wrapper approach, a feature subset is evaluatedby the performance of the chosen learning algorithm. In thispaper, for a given feature subset, a multi-class LDA modelis trained as in IV-B. Based on the LDA model, the average10-fold cross-validation accuracy is used as the criterionfor evaluation. The overall feature reduction algorithm is asfollows:

1) Start with the full feature set X0 = {x1,x2, ...,xN}.2) For every feature xi ∈ Xk, train a LDA model on Xk−{xi}.

3) Remove the feature x = argmaxx

J(Xk−{x}), where J is

the 10-fold classification accuracy.4) Update the feature set as Xk+1 = Xk−{x}.5) Goto step 2 and repeat until meeting a certain stop

condition (the accuracy and the number of remainingfeatures meet user-specified requirements).

This algorithm helps to determine the optimal featuresubset and provides a sensor selection guideline to reducecost for the final implementation. In addition, fewer featuresresult in reduced complexity and faster algorithm.

To show the whole picture, we set the stopping conditionto be k = N − 1, i.e., only one feature is left in the finalsubset. Fig. 7 shows the 10-fold cross-validation accuracyof result classification for each optimal subset. The curve isnon-monotonic; the accuracy initially increases when somefeatures are removed and drops significantly when very fewfeatures are left. Fig. 8 shows the variation of the optimalfeature set (grouped according to the sensor signals) duringthe reduction process. We see that most features from Fx andFy are removed at the first few steps, while features from Mc,Fz and Tz can still achieve quite good accuracy when mostfeatures from other signals are removed. The angle feature θ

remains during the entire reduction process (not shown in thefigure); this explains the importance of angle measurement.

This feature selection process indicates that, instead ofusing all the signals, accurate result type classification can

Fig. 7. Prediction accuracy for optimal feature subsets of different sizes.

Fig. 8. Stacked bar graph during the feature reduction process.

be achieved with only four signals: Mc, Fz, Tz, and θ (angle).Note that we do not need both Mc and Tz. In fact, we canfurther reduce the required sensor signals and cost based onthe gear ratio. For low gear ratios (5.8:1 in our system), Mcis almost proportional to Tz, thus we can select Mc (motorcurrent) because it costs less to measure. For high gear ratios,it is necessary to measure Tz because the fastening torquecannot be estimated from Mc due to high gear loss.

For our system, we can remove Tz and reduce the signalsto {Mc, Fz, θ}. To evaluate the performance, we train twoLDA classifiers on the selected signals with and without Tz.We obtain an optimal feature subset consisting of 20 featuresfor {Tz, Mc, Fz, θ} signals and 18 features for {Mc, Fz,θ} signals, as shown in Table I and Table II, respectively.As shown in Fig. 9, even with fewer sensor signals andfeatures, both classifiers achieve very high accuracy (99.03%and 98.66%, respectively). Their performance is quite similarto the baseline case shown in Fig. 4, where 85 global featuresare used. Compared with the baseline, the reduced LDAmodels perform better in predicting the partial and strippedno engage classes; they perform slightly worse for the nohole found class.

We see that a linear classifier based on reduced sensorsignals can successfully predict the results of screwdrivingtasks with very high accuracy. This method can be viewed asan extension of current industrial method, which only usesthe maximum torque and angle to predict success or failure.

V. STAGE CLASSIFICATIONS

As discussed in Section III, identifying stages in screw-driving tasks is the first step towards building online failureprediction and recovery for advanced robotic screwdrivingsystem. This is particular important for large-volume pro-ductions in consumer electronics industry [1]. In this section,we train decision tree models [22] to discriminate differentstages during screwdriving process. The reason of using

6082

Fig. 9. Confusion matrices for LDA models trained with 20 (left figure)and 18 (right figure) global features, respectively. Result type numbering:1.success; 2.no screw; 3.no hole found; 4.crossthread; 5.stripped; 6.strippedno engage; and 7.partial.

TABLE IOPTIMAL FEATURES FOR {Tz , Mc , Fz , θ} SIGNALS

Order Feature Order Feature Order Feature1 Tz std 8 ∆2Tz std 15 ∆Tz std2 ∆Fz std 9 Mc std 16 Tz range3 ∆2Fz range 10 ∆2Fz std 17 ∆Fz mean4 Fz mean 11 Fz range 18 ∆2Mc mean5 Fz std 12 ∆2Mc range 19 ∆2Tz mean6 ∆Tz mean 13 Mc range 20 Tz mean7 θ 14 ∆2Mc std

decision tree models is that they give the best results over allthe models we have tested (such as SVM, LR and LSTM, seeSection V-B). The stage classifiers are trained respectivelyon the full signal set and the reduced subset {Mc,Fz,Tz},the latter is selected based on the sensor reduction resultin Section IV. It is shown that the stage classification stillachieves high accuracy with reduced sensor signals.

A. Stage Data Extraction

Labeled stage samples are extracted at the rate of 10Hz from the original time series data (sampled at 100Hz). This means that our stage prediction algorithm makesprediction every 0.1s. This is a reasonable value given thatour screwdriver is operated at a very low speed; a typical runmight take 10 ∼ 18s (see Fig. 2). With a shift of 0.01s for0.1s windows, a total of 1866238 stage samples are extracted.Each sample is represented as a vector of length 10∗N fromN signals. For stage classifier trained on all the signals, thelength of each sample is 80, while for stage classifier trainedon the subset {Mc,Fz,Tz}, the length of each sample is 30.

B. Decision Tree

We shuffle the dataset and split it into the training set(80%) and the test set (20%). Decision trees are trained topredict realtime stages [23]: one is trained on the full sensorset; the other is trained on reduced sensor set {Mc,Fz,Tz}.The classifier using all signals provides a benchmark to theclassifier using only three signals {Mc,Fz,Tz}.

Table III compares these two decision trees. The corre-sponding confusion matrices are shown in Fig. 10 and Fig.11, respectively. It can be seen that the reduced sensor setachieves slightly higher accuracy compared to the full sensor

TABLE IIOPTIMAL FEATURES FOR {Mc , Fz , θ} SIGNALS

Order Feature Order Feature Order Feature1 Fz std 7 ∆2Fz range 13 ∆Fz range2 ∆2Mc range 8 Fz range 14 ∆Fz mean3 Fz mean 9 ∆Mc range 15 ∆ddMc std4 ∆2Fz std 10 ∆2Mc std 16 ∆ddMc range5 ∆Fz std 11 θ 17 ∆Mc mean6 Mc mean 11 Mc range 18 ∆ddMc mean

TABLE IIICOMPARISON OF TWO DECISION TREES

Signals Nodes Training Accuracy Test Accuracy8 26983 99.55% 97.55%3 74911 98.79% 97.78%

set. This result suggests that three signals ({Mc,Fz,Tz}) aregood enough for the screwdriving stage classification. Thistable also shows that the decision tree models work wellfor all stage types, including those with fewer samples. Forexample, the testing accuracy for stripped engaging andstripped tightening are 93.1% and 93.0% respectively, eventhough their total percentage are less than 1%.

We trained other models, including SVM, logistic regres-sion (LR) and Recurrent Neural Network using the LongShort Term Memory (LSTM). The SVM and LR both yield aless than 91% accuracy with nearly zero accuracy on the fourstages (screw fallen, stripped engaging, stripped rundown,and stripped tightening) with fewer data. The performance ofthe LSTM model is slightly better (96.51%), but still cannotbeat the decision tree; moreover, the training is significantlymore computationally intensive.

Our stage classifier not only reveals the underlying processof the ongoing tasks, but also helps improving the robustnessand end-user experience. First, our classifier can improvethe performance of the result classifier. As mentioned in IV-B, our result classifier has lower accuracy at distinguish-ing no hole found against stripped or stripped no engage.However, this problem can be easily solved when combinedwith the stage classifier. Moreover, similar to the GTC-DFmodel in [14], our models are also easy to interpret dueto the decision tree structure. After simplification, we canprovide interpretable logical statement interface to the endusers. Meanwhile, the probability of each stage class can beobtained at each terminal node, which can help human tomonitor the screwdriving process.

C. Realtime Stage Prediction using Decision Tree

Using the 3-signal ({Mc,Fz,Tz}) decision tree in SectionV-B, realtime stage prediction can be performed with fastspeed. On a 4-core i7-6560 2.20GHz computer with 8GBmemory, prediction for each 3-signal stage sample takes10.1ms in average, even though our MATLAB code has notbeen optimized for speed yet. Prediction for each 8-signalstage sample takes 4.1ms in average, due to reduced modelsize, as shown in Table III.

The realtime prediction results for typical runs with dif-ferent result types and stage patterns are shown in Fig. 12-Fig. 14. Compared with the ground truth, our algorithms have

6083

Fig. 10. Stage classification confusion matrix for the full sensor set.

Fig. 11. Stage classification confusion matrix for {Mc,Fz,Tz}.

good performance in predicting the overall trends for stages.There exist some stage misclassifications; most of them areeither insignificant or correctable.

The first type of misclassification appears at the transitionof two stages. This will not be considered as a fault pre-diction, since a transition can be classified as either of itsneighboring stages. For example, as shown in Fig. 12, thetransition from hole finding to initial mating is classified asthe latter stage, this should also be considered as correctlyclassified. Note that our algorithm generates a misclassifi-cation, a initial mating stage at the end, because Tz almostequals to zero in this period. This misclassification actuallybelongs to the second type (see below).

The second type of misclassifications are those can becorrected by the stage transition graph (see Fig. 3). Forexample, the screw fallen stage in Fig. 13 (a) is not connectedwith the initial mating stage in Fig. 3, so this misclassifi-

Fig. 12. Realtime stage prediction for a crossthread run. The misclassifiedinitial mating stage is highlighted by a red block.

Fig. 13. Realtime stage prediction for a success run. The misclassifiedinitial mating stage is highlighted a red block.

cation can be corrected by the stage transition graph. Thecorrected result will match the actual classification shownin Fig. 13 (b). Fig. 14 (b) shows the stage prediction resultafter augmenting with the stage transition graph. Comparedwith Fig. 14 (b), there is only one stage misclassification —the tightening stage that only lasts 0.1s right after rundown.Close examination reveals that our algorithm actually worksvery well because Tz rapidly increases over this 0.1s windowright after the rundown phase. A reasonable classifier willpredict this 0.1s window as tightening stage. However, ahuman cannot do such precise classification (this may notbe necessary) during hand labeling. Instead, a human simplylabels a stripped tightening right after rundown by consider-ing the entire signal profile, including future signals.

VI. DISCUSSIONS AND FUTURE WORK

The stage and result classification algorithms we de-veloped are essentially data-driven, which requires largeamounts of training data. This data-driven approach suffersfrom generality issues. There are many different types ofscrews and nuts used in consumer electronics industry. Theforce and torque signatures will change when the part (e.g.,buttons and PCBs) to be bolted varies in geometry ormaterial. Therefore, one would need to rerun the experimentsfor data collection once the customers modify their product

6084

Fig. 14. Realtime stage prediction for a stripped run. Misclassified stagesare highlighted by red blocks.

designs. Although the retraining process can be quite fastusing our algorithms, the data collection process takes timeand resources. This process might be worthwhile for verylarge volume production. However, it is always beneficial toinvestigate techniques (through modeling or simulation) thatcan significantly reduce the data size requirement. This canbe quite difficult since exact modeling of the interactionsand contact forces between different objects and among thescrew threads is very challenging.

There is some information which could be inferred fromthe removed sensor signals. For example, the authors of [16]essentially use the auto-correlation function of {Fx,Fy} toestimate the insertion length. (Note that they actually usethe auto-correlation of Tz; however, there is a large offsetbetween the F/T z-axis and the center axis of the screwdriverin their setup, so Tz is dominated by Fx and Fy.) However, thisinsertion length can be easily measured by an encoder — anapproach adopted by industry. In our study, we find that theoscillating amplitude of the {Fx,Fy} (equivalent to {Tx,Ty}after transformation) is strongly correlated with the position-ing error. Therefore, one could estimate the misalignmentusing these signals. However, this will significantly increasethe system cost and complexity.

In the future, a low-cost screwdriver will be designedbased on our sensor reduction analysis. An online stage andresult prediction system will be developed through fasterand improved implementation of our algorithms. Recoveringstrategies corresponding to different failure types will alsobe designed to improve the overall success rate.

ACKNOWLEDGMENTS

The authors would like to thank Foxconn for providing thefinancial support. The views and opinions expressed in thispaper do not necessarily state or reflect those of Foxconn.

REFERENCES

[1] Z. Jia, A. Bhatia, R. Aronson, D. Bourne, and M. T. Mason, “A surveyof automated threaded fastening,” IEEE Transactions on AutomationScience and Engineering, Conditionally accepted, 2018.

[2] J. L. Nevins and D. E. Whitney, “Computer-controlled assembly,”Scientific American, vol. 238, pp. 62–74, 1978.

[3] J. Nevins and D. Whitney, “Assembly research,” Automatica, vol. 16,no. 6, pp. 595–613, 1980.

[4] L. A. Martin-Vega, H. K. Brown, W. H. Shaw, and T. J. Sanders,“Industrial perspective on research needs and opportunities in manu-facturing assembly,” Journal of manufacturing systems, vol. 14, no. 1,pp. 45–58, 1995.

[5] D. E. Whitney, Mechanical assemblies: their design, manufacture, androle in product development. Oxford university press, 2004, vol. 1.

[6] Z. Li. Robotics research for 3C assembly automation. [Online].Available: https://app.box.com/s/zcg8qqxt6fw6v4xz22h6

[7] U. S. C. P. S. Commission. (2018) Lenovo re-calls thinkpad laptops due to fire hazard. [On-line]. Available: https://www.cpsc.gov/Recalls/2018/lenovo-recalls-thinkpad-laptops-due-to-fire-hazard

[8] A. Weber, “Automation for small screws,” Assembly, vol. 55, no. 2,pp. 40–43, 2012.

[9] J. H. Bickford, Handbook of bolts and bolted joints. CRC press,1998.

[10] ——, Introduction to the design and behavior of bolted joints: non-gasketed joints. CRC Press, 2007.

[11] ISO, “ISO 5393: Rotary tools for threaded fasteners – performancetest method,” ISO, Tech. Rep., 2013.

[12] R. M. Aronson, A. Bhatia, Z. Jia, M. Guillame-Bert, D. Bourne,A. Dubrawski, and M. T. Mason, “Data-driven classification ofscrewdriving operations,” in International Symposium on ExperimentalRobotics. Springer, 2016, pp. 244–253.

[13] MicroTorque-ToolsTalk MT User Guide, Atlas Copco.[14] R. M. Aronson, A. Bhatia, Z. Jia, and M. T. Mason, “Data collection

for screwdriving,” in Robotics Science and Systems, Workshop on(Empirically) Data-Driven Manipulation, 2017.

[15] J.-Y. Hwang, D.-H. Jung, Y.-J. Roh, K.-J. Nam, and D.-Y. Hwang,“Low-cost automatic screw machine using a commercial electricscrewdriver,” in Control, Automation and Systems (ICCAS), 2012 12thInternational Conference on. IEEE, 2012, pp. 1055–1060.

[16] T. Matsuno, J. Huang, and T. Fukuda, “Fault detection algorithm forexternal thread fastening by robotic manipulator using linear supportvector machine classifier,” in Robotics and Automation (ICRA), 2013IEEE International Conference on. IEEE, 2013, pp. 3443–3450.

[17] A. I. Automation, “Multi-axis force/torque sensors.” [Online].Available: http://www.ati-ia.com/products/ft/sensors.aspx

[18] G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction tostatistical learning. Springer, 2013, vol. 112.

[19] H. Liu and H. Motoda, Feature selection for knowledge discovery anddata mining. Springer Science & Business Media, 2012, vol. 454.

[20] A. Nanopoulos, R. Alcock, and Y. Manolopoulos, “Feature-basedclassification of time-series data,” International Journal of ComputerResearch, vol. 10, no. 3, pp. 49–61, 2001.

[21] V. Kumar and S. Minz, “Feature selection,” SmartCR, vol. 4, no. 3,pp. 211–229, 2014.

[22] L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen, Classificationand regression trees. CRC press, 1984.

[23] L. Rokach and O. Maimon, “Top-down induction of decision treesclassifiers-a survey,” IEEE Transactions on Systems, Man, and Cyber-netics, Part C (Applications and Reviews), vol. 35, no. 4, pp. 476–487,2005.

6085