Discriminative Feature Selection via A Structured SparseSubspace Learning Module

Zheng Wang1 , Feiping Nie1∗ , Lai Tian1 , Rong Wang1,2 and Xuelong Li11School of Computer Science and Center for OPTical IMagery Analysis and Learning (OPTIMAL),

Northwestern Polytechnical University, Xi’an, 710072, P. R. China2School of Cybersecurity, Northwestern Polytechnical University, Xi’an, 710072, P. R. China

zhengwangml, feipingnie, tianlai.cs@gmail.com, wangrong07@tsinghua.org.cn, li@nwpu.edu.cn

AbstractIn this paper, we first propose a novel StructuredSparse Subspace Learning (S3L) module to ad-dress the long-standing subspace sparsity issue.Elicited by proposed module, we design a newdiscriminative feature selection method, namedSubspace Sparsity Discriminant Feature Selection(S2DFS) which enables the following new func-tionalities: 1) Proposed S2DFS method directlyjoints trace ratio objective and structured sparsesubspace constraint via `2,0-norm to learn a row-sparsity subspace, which improves the discrim-inability of model and overcomes the parameter-tuning trouble with comparison to the methodsused `2,1-norm regularization; 2) An alternative it-erative optimization algorithm based on the pro-posed S3L module is presented to explicitly solvethe proposed problem with a closed-form solu-tion and strict convergence proof. To our bestknowledge, such objective function and solver arefirst proposed in this paper, which provides a newthough for the development of feature selectionmethods. Extensive experiments conducted on sev-eral high-dimensional datasets demonstrate the dis-criminability of selected features via S2DFS withcomparison to several related SOTA feature selec-tion methods. Source matlab code: https://github.com/StevenWangNPU/L20-FS.

1 IntroductionData in many areas are represented by high-dimensional fea-tures, and a natural question emerges out: what are themost discriminative features in the high-dimensional data?Feature selection plays a crucial role in machine learningwhich endeavors to select a subset of features from the high-dimensional data for improving data compactness and reduc-ing noisy features so that the over-fitting, high computationalconsumption and low performance issues can be alleviated inmany real-world applications.

Feature selection approaches can be roughly divided intothree categories, i.e., filter methods [Gu et al., 2012], wrap-

∗Corresponding Author

per methods [Maldonado and Weber, 2009] and embed-ded methods [Xiang et al., 2012; Ming and Ding, 2019;Tian et al., 2019]. Wherein, filter methods focus on eval-uating the correlation of features with respect to class labelof data, which results in that the correlated features are re-dundant. Wrapper methods measure the importance of fea-tures according to classification performance, so as to thecomputational cost of wapper model is very high and the rep-resentability of learned features may be poor in other tasks.In contrast, embedded methods gain more attentions since itincorporates feature selection and classification model intoa unified optimization framework by learning sparse struc-tural projections. Besides, some feature selection approachesbased on neural network [Zhang et al., 2020] and auto-encoder model [Han et al., 2018; Abid et al., 2019] pay over-much attentions on minimizing the reconstruct errors, whichpossible not benefit for classification task and tuning param-eters may affect its efficiency of practical applications.

Most of embedded methods commonly employ `2,1-normregularization to improve the row-sparsity in learned sub-space. Concretely, Nie et al. propose robust feature selectionmodel (RFS) [Nie et al., 2010] which simultaneously incor-porates `2,1-norm into loss function and regularization termto overcome outliers issues and make feature selection cometrue. Similar, [He et al., 2012] proposes to use CorrentropyRobust loss function in Feature Selection (CRFS) to enhancemodel’s robustness. Additionally, [Xiang et al., 2012] devel-ops a feature selection effort joints discriminative least squareregression (DLSRFS) model and sparse `2,1-norm regulariza-tion into an unified framework. Recently, RJFWL [Yan et al.,2016] aims to learn the ranking of all features by imposing`2,1-norm constraint and non-negative constraint on learnedfeature weights matrix. Regrettably, all aforementioned ap-proaches are always unable to escape the dilemma of tuningthe trade-off parameter between loss term and sparse regu-larization term. Moreover, the compactness of sparse row issensitive to the trade-off parameter, namely, the uniquenessof optimum is difficult to guarantee.

An illustrative example is depicted in Figure 1, from whichwe can figure out that `2,1-norm calculates the score of fea-tures integrally, then ranks them resorting to the learnedscores. By doing so, the feature ranking results may bechanged dramatically when the feature score vary subtly.In contrast, `2,0-norm obtains the score matrix only depend

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

3009

1

2

3

4

5

6

7

8

9

10

(b). L2,0 structured sparse learning

1.0

Feat

ure

scor

es

(a). L2,1 sparse learning

1.89

1.902.181.361.251.682.282.530.48

1

2

3

4

5

6

7

8

9

10

0.58

1.01.01.0

1.0

Feat

ure

scor

es

Score Score

Figure 1: Visualization of learned sparse feature score matric, (a).`2,1 sparse matrix and (b). `2,0 structured sparse matrix, respec-tively. Navy blue background denotes the scores that close to zero.

upon the k features, and no further ranking operation is re-quired. Based on these, Cai et al. [Cai et al., 2013] question-ing that whether the method based on convex problem is al-ways better than that based on non-convex problem. They ex-ploit top-k features selection method (RPMFS) based on leastsquare regression model with `2,0-norm equality constraint.Pang et al. [Pang et al., 2018] develop an Efficient SparseFeature Selection (ESFS) via `2,0-norm constraint based onleast square regression model as well. In [Du et al., 2018],Unsupervised Group Feature Selection (UGFS) algorithm se-lects a group of features initially then update the selectionuntil a better group appears by using `2,0-norm constraint.Nevertheless, both of above methods optimize the `2,0-normconstraint problem by using Augmented Lagrangian Multi-plier (ALM) and gradient descent optimization algorithms,which may result in that the solution is sensitive to the initial-ization and easy to stuck in local optimum. Ultimately, thesubsequent performance of classification is fluctuant.

Facing these obstacles that hinder the development of fea-ture selection, we entail solving `2,0-norm constraint problemalong with rigorous theoretical guarantee firstly in this pa-per. Then, we propose a new discriminative feature selectionmodel via orthogonal `2,0-norm constraint. Finally, we ana-lyze the performance of proposed method from two aspects,i.e., discriminability and efficiency. Our contributions canbe summarized as follows:• We propose a Structured Sparse Subspace Learning

(S3L) module to solve the subspace sparsity issue withtheoretical guarantee, which is first presented in our pa-per according to our best knowledge.• We elaborately design a discriminative feature selection

model, named Subspace Sparsity Discriminant FeatureSelection (S2DFS) that integrates trace ratio formulatedobjective and structured sparse subspace constraint foracquiring more informative feature subset.• Optimizing the proposed S2DFS model is a NP-hard

problem. We provide an alternative iterative optimiza-tion algorithm to solve trace ratio problem, in which theproposed S3L module is employed to acquire a closed-formed solution rather than an approximate one, so as toensure the stability of selection results.• Experimental results show the effectiveness of proposed

optimization algorithm in two perspectives, i.e., perfor-mance: our method outperforms other related SOTA

feature selection methods in terms of classification onseveral real-world datasets; convergent speed: our algo-rithm reaches convergence within few iterations.

2 Related WorksIn this section, we briefly review constrained Linear Discrim-inative Analysis (LDA) in trace ratio formulation, then thesubspace sparsity issue will be introduced.

2.1 Constrained Trace Ratio LDAGiven the training data X = x1,x2, ...,xn ∈ Rd×n, wherexi ∈ Rd×1 and the label vector is y = y1, y2, ..., yn,where yi ∈ 1, 2, ..., c, and c denotes the total class num-ber. LDA aims to find a transformation matrix W ∈ Rd×m

to map original high-dimensional sample x ∈ Rd×1 into low-dimensional subspace with m dimensions where m << d.For improving the discriminative power of model, the sam-ples within the same class are pulled together as well asthose points between different classes are pushed far away inlearned subspace ideally. There are many kinds of objectivefunction of LDA [Bishop, 2006], one of the most discrimi-native objectives is the following constrained trace ratio one

maxWTW=I

Tr(WTSbW)

Tr(WTSwW), (1)

where Sb =∑c

i=1 ni (µi − µ) (µi − µ)T and Sw =∑c

i=1

∑xj∈πi

(xj − µi) (xj − µi)T denote between-class

and within-class scatter matrix respectively. ni is the num-ber of samples belong to i-th class, µi denotes the mean ofsamples in i-th class, µ is the mean of total samples and πi

denotes the set of i-th class points. Orthogonal constraint isused to avoid trivial solution [Nie et al., 2019]. Note that, op-timizing such model in Eq.(1) is a non-convex optimizationproblem that does not have a closed-form solution, and sev-eral attempts [Wang et al., 2007; Nie et al., 2019] have triedto solve it. In what follows, we will provide an alternativeiterative optimization algorithm to get the solution.

2.2 Subspace Sparsity ProblemSubspace sparsity issue is originally derived from the sparseprincipal subspace estimation [Vu et al., 2013] which can begenerally formulated as

maxWTW=Im×m,‖W‖2,0=k

Tr(WTAW), (2)

where m ≤ k ≤ d and A ∈ Rd×d is a positive semi-definite matrix. ‖W‖2,0 =

∑di=1

∥∥∥∑mj=1 w

2ij

∥∥∥0

denotes thenumber of non-zero rows in matrix W, and the constraint‖W‖2,0 = k forces the learned subspace is row sparse, inwhich the index of non-zero rows equals to the index of se-lected features. Since both the orthogonal and row sparsityconstraints are the non-convex constraint, optimizing prob-lem (2) is challenging. In the following Section 3.1, we willpropose the S3L module to solve it with theoretical guarantee.

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

3010

3 Proposed Method3.1 Structured Sparse Subspace Learning ModuleIn this section, we propose a novel S3L module to solve prob-lem (2) that has been tentatively solved in [Wang et al., 2014;Yang and Xu, 2015], but none of them provides deterministicguarantee on global convergence. We first develop a straight-forward strategy for solving the case that rank (A) ≤ m.Then, an iterative optimization Algorithm 1 is derived tosolve the general case that rank (A) > m in which a low-rank proxy covariance P is designed to approximate A. Forreadability, we define two definitions that will be used in fol-lowing deducing:

Definition 3.1 Given an indices I whose elements span from1 to d. Then, the row extraction matrix Qk

d ∈ Rd×k is definedas:

qij =

1, if i = I(j)0, otherwise.

Define Qd,k(I) = Qkd as an operator that inputs indices I

and outputs row extraction matrix Qkd ∈ Rd×k.

Case 1: rank (A) ≤ mWe first consider the simplest problem, i.e., k = m, thus,

the problem (2) reduces to

maxWTW=Im×m,‖W‖2,0=m

Tr(WTAW

). (3)

Due to the constraint WTW = Im×m and ‖W‖2,0 = m,W can be rewritten as W = Qm

d V, where V ∈ Rm×m andsatisfies VTV = Im×m. Additionally, Qm

d = Qd,m(I) ∈Rd×m is the row extraction matrix that has been defined.Then, problem (3) has been reduced to solve

maxV∈Rm×m,VTV=Im×m

Tr(VT AV

), (4)

where A = Qmd

TAQmd . So far, the row extraction matrix

Qmd is still unknown. As a consequence, considering V ∈

Rm×m is a square matrix and VTV = VVT = I, we caninfer the fact Tr(VT AV) = Tr(AVVT ). Thus, problem(4) can be reduced to

maxTr(A), (5)

which has a globally optimal solution that m largest diagonalelements of A, and Qm

d is generated by operator Qd,m(I)with input I that equals to the indices of m largest diagonalelements of A.

Next, we consider the case that k 6= m, and problem inCase 1 becomes to

maxW∈Rd×m

Tr(WTAW

)s.t. WTW =Im×m, ‖W‖2,0 = k, rank(A) ≤ m.

(6)

We use similar technique to solve problem (6) as

maxTr(Qk

d

TAQk

d

), (7)

which can be easily solved globally by selecting k largest di-agonal elements of A, and Qk

d = Qd,k(I) where I denotes

the indices of k largest diagonal elements of A. Through-out above analysis, our S3L module is able to efficiently(non-iteratively) obtain globally optimal solution of NP-hard problem (2) when rank(A) ≤ m.

Nevertheless, the Case 1 is not usually occurred in prac-tice, since PCA often used as a data-preprocessing techniqueto remove null space of data, which cause a general Case 2,i.e., rank(A) = d > m. Next, we develop an iterative S3Lmodule to solve the problem (2) in the following Case 2.

Case 2: rank (A) > mWe solve the problem (2) in case 2 based on the

Minimization-Majorization framework (MM) [Sun et al.,2016] whose key insight is to maximize a lowerbound func-tion of target problem. Therefore, how to design a suitablelowerbound function is the key to the success of MM. For-tunately, through a lot of attempts, we elaborately design anideal lowerbound function of problem (2) as follow:

g (W|Wt) = Tr(WTΓtW)

s.t. W ∈ K,(8)

where K = W ∈ Rd×m|WTW = I, ‖W‖2,0 = k isthe structured sparse subspace constraint, the surrogate vari-able Γt = AWt

(WT

t AWt

)†WT

t A and (·)† denotes theMoore-Penrose pseudoinverse. As a lowerbound function, adeserved property should be invariably held:

Tr(WTΓtW) ≤ Tr(WTAW) ∀W ∈ K, (9)

which will be proved in Section 3.2.According to MM framework, solving problem (2) in Case

2 becomes to solve

maxW∈K

Tr(WTΓtW

), (10)

which is still a difficult problem that can not be optimizeddirectly. Now, we discuss some properties of designed surro-gate variable Γt through the following Theorem 1.

Theorem 1 In the t-th iteration (t ≥ 1), the conditionsrank (Γt) ≤ m and Γt 0 will always hold.

Proof 1 According to the property of matrix rankrank(AB) ≤ minrank(A), rank(B), we haverank(Γt) ≤ rank(Wt) = m, then the first condi-tion has been proved. Subsequently, we suppose thatΩ = AWt

(WT

t AWt

)†WT

t A12 , then according to the fact

that A 0, A is a symmetric matrix and B† = B†BB†, wecan obtain

ΩΩT = AWt

(WT

t AWt

)†WT

t AWt

(WT

t AWt

)†WT

t A

= Γt 0,(11)

which completes the proof of Theorem 1.

According to Theorem 1, we wondrously discover an in-teresting phenomenon that the designed surrogate variable Γt

satisfies all conditions of A in case 1, i.e, rank(A) ≤ m andΓt 0. Those facts enlighten a thought that problem (10)can be solved by using S3L module iteratively, which is sum-marized in the following Algorithm 1.

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

3011

Algorithm 1 Solve problem (2), when rank(A) > m

Input: A, k, m, d and W0;Initialization: W0 = rand(d,m), t = 0;Repeat:1. Γt ←AWt

(WT

t AWt

)†WT

t A;2. It ← sort ( diag(Γt), ’descend’, k );3. Qt

kd ←Qd,k(It);

4. Vt ← eigs((

Qtkd

)TAQt

kd, ’descend’,m

);

5. Wt+1 ←QtkdVt;

6. t = t+ 1;Until convergenceOutput: W∗ ∈ Rd×m.

3.2 Algorithm AnalysisWe prove the convergence of the proposed Algorithm 1through the following Theorem 2.Theorem 2 Algorithm 1 increases the objective functionvalue of Eq.(2) in each iteration, when rank(A) > m.Proof 2 Note the fact that B = BB†B, we can infer that

Tr(WT

t AWt

)= Tr

(WT

t AWt

(WT

t AWt

)†WT

t AWt

)= Tr

(WT

t ΓtWt

),

(12)where rank (Γt) ≤ m. Then, we can use the S3L module tomaximize the above Eq.(12) as

Tr(WT

t ΓtWt

)=Tr

(WT

t+1AWt

(WT

t AWt

)†WT

t AWt+1

),

(13)

where WTt+1 is generated according to Eq.(4) with input A =

Γt. Thanks to the commutative law in algebraic operation ofmatrix trace and A is a symmetric matrix, Eq.(13) can bedecomposed as

Tr (ΦΘ) , (14)where two symmetric matrices are respectively defined asΦ = A

12 Wt

(WT

t AWt

)†WT

t A12 ∈ Rd×d, Θ =

A12 Wt+1W

Tt+1A

12 ∈ Rd×d. Considering Theorem 4.3.53

in [Horn and Johnson, 2012], we have

Tr (ΦΘ) ≤d∑

i=1

λi (Φ)λi (Θ) ≤m∑i=1

λi (Θ) , (15)

which provides an evidence of correctness for Eq.(9). More-over, note that rank(Θ) ≤ rank(Wt+1) = m, then we have∑m

i=1 λi (Θ) = Tr (Θ). Combining Eq.(12), Eq.(13) andEq.(15), we can conclude that

Tr(WT

t AWt

)≤ Tr

(WT

t+1AWt+1

). (16)

According to the previous definitions, i.e., Wt+1 = QtkdVt

and Wt+1 = QtkdVt, we can finally obtain that

Tr(WT

t AWt

)= Tr

((Vt

)T (Qt

kd

)TAQt

kdVt

)≤ Tr

(WT

t+1AWt+1

),

(17)

then the proof of Theorem 2 has been completed.

3.3 Subspace Sparsity Discriminant FeatureSelection

In this part, we propose to introduce a novel feature selectionmethod named S2DFS, and optimize it by using S3L module.

Model Formulation. In order to improve model’s discrim-inative power, S2DFS leverages trace ratio LDA model inEq.(1) for pulling samples within same class together andpushing those samples between different classes far away inlearned subspace. Besides, we use the structured sparse sub-space constraint to directly facilitate the exploration and in-ference of important features of data. In light of these points,the objective function of S2DFS model can be formulated as

maxW∈Rd×m

Tr(WTSbW

)Tr (WTSwW)

s.t. WTW = Im×m, ‖W‖2,0 = k,

(18)

where m ≤ k < d, m is the number of selected featuresand k means the number of non-zero rows in W. Due tothe nonlinear constraints and trace ratio objective, optimizingsuch a maximization ratio problem in Eq.(18) is challenging.

Optimization Algorithm Before solving problem (18), wefirst consider to solve a general maximization ratio problemas follow:

maxx∈C

f(x)

g(x), (19)

where x ∈ C is arbitrary constraint on x and as the denomina-tor, g(x) > 0. In Algorithm 2, we summarize an alternativeoptimization algorithm to solve the general problem (19), anda series of theorems and proofs are presented to guarantee itsconvergence.

Algorithm 2 Algorithm to solve the general maximizationratio problem (19).

1. Initialization: x ∈ C, t = 1.While not converge do

2. Calculate λt =f(xt)

g(xt).

3. Calculate xt+1 = argmaxx∈C

f(x)− λtg(x).4. t = t+ 1.end while

Theorem 3 The global solution of the general maximizationratio problem (19) is equivalent to the root of following func-tion:

h(λ) = argmaxx∈C

f(x)− λtg(x). (20)

Proof 3 Supposing x∗ is the global solution of problem (19)and its corresponding maximal objective function value is λ∗,

then the following equation holds:f(x∗)

g(x∗)= λ∗. As a result,

∀ x ∈ C, we always havef(x)

g(x)≤ λ∗. As the condition

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

3012

g(x) > 0, we can infer that f(x)−λ∗g(x) ≤ 0 which means:

maxx∈C

f(x)− λ∗g(x) = 0 ⇒ h(λ∗) = 0. (21)

Consequently, the global maximal objective function value λ∗of problem (19) is the root of function h(λ). Thus, the proofof Theorem 3 is completed.

Theorem 4 Algorithm 2 increases the objective functionvalue of problem (19) in each iteration until it reaches con-vergence.Proof 4 According to the step 2 in Algorithm 2, we havef(xt) − λtg(xt) = 0, and from step 3, we can infer thatf(xt+1)− λtg(xt+1) ≥ f(xt)− λtg(xt). Combining abovetwo inequalities, we can obtain f(xt+1) − λtg(xt+1) ≥ 0

that equals tof(xt+1)

g(xt+1)≥ λt =

f(xt)

g(xt). That is, Algorithm

2 increases the objective function value in Eq.(19) in eachiteration and the proof of Theorem 4 has been finished.

Systematically, it is worth noting that Theorem 3–4 providea complete framework to optimize the general maximizationratio problem in Eq.(19) with rigorously convergent proof.

According to the Algorithm 2, we can infer that the key ofsolving proposed maximization ratio problem in Eq.(18) is tosolve the following problem:

maxW∈Rd×m

Tr(WTBW

)s.t. WTW = I, ‖W‖2,0 = k,

(22)

where B = Sb − λSw + ηI is a positive semi-definite matrixif η is large enough. In theory, the relaxation parameter η iseasily to be calculated as the largest eigenvalue of Sb − λSw.The first way that can be thought to acquire η is eigenvaluedecomposition, which however is time-consuming in dealingwith high-dimensional data. Here, we employ a power iter-ation method [Nie et al., 2017] to solve above issue, whichcan obtain the relaxation parameter η faster than eigenvaluedecomposition and improves the applicability of our method.

Note that problem (22) is a special case of problem (2),which can be directly addressed by using proposed S3Lmodule and obtain a closed-form solution.

4 ExperimentsIn this section, for verifying the effectiveness of proposedmethod, we first present an visualization experiment on Mnistdataset. Then, we evaluate its classification performance onnine publicly available high-dimensional datasets. Finally,we exhibit the convergent speed of our method.

4.1 PreliminaryDatasets. We evaluate the performance of proposed methodon several high-dimensional real-world datasets, and moredetails about them are shown in Table ??. For the color imagedatasets, i.e., Pubfig [Xu et al., 2018], we firstly downsampleeach image into a suitable scale and extract 100 LOMO fea-tures [Liao et al., 2015] for data representation.

1http://qwone.com/∼jason/20Newsgroups/

Datasets Dim. Class Num. Type20News1 8,014 4 3,970 TextWebKB2 4,165 7 1,166 WebBinalpha3 320 36 1,404 LetterPixraw10P3 10,000 10 100 Face imageText13 7,511 2 1,946 TextPubfig 65,536 8 772 Face image

Table 1: Descriptions of datasets.

Experimental setting. We compare our method to severalSOTA feature selection methods including DLSRFS [Xianget al., 2012], RFS [Nie et al., 2010], CRFS [He et al., 2012],RJFWL [Yan et al., 2016], infFS [Roffo et al., 2015] andESFS [Pang et al., 2018] in the classification task. We usethe k-nearest neighbor algorithm as the classifier, all exper-iments are repeatedly conducted 10 times and the averagerecognition rate and standard deviation are recorded as themeasurement of performance for all competitors.

4.2 Visualization ExperimentIn order to intuitively show the performance of feature selec-tion algorithm, we provide an visualization experiment con-ducted on Mnist digits dataset. Figure 2b exhibits the twotypes of selected features, i.e., digital pixels (white dots) andbackground pixels (red dots) respectively. In the Figure 2c,we show the selected features lied on the Mnist digits, fromwhich we can observe that the shape of digits can be dis-tinctly reconstructed by selected features generated by pro-posed method. Experimental results verify the discriminabil-ity of selected features that learned from our method.

(a) (b) (c)

Figure 2: The results of using proposed S2DFS to select 20 mostinformative pixels of Mnist digits. (a) The selected 20 features oneach MNIST digit that are shown in white pixels. (b) The selectedfeatures in training samples which are in red and white dots. (c) Theselected features (red points) lied on the Mnist digits.

4.3 Classification ExperimentFigure 3 shows the statistical estimation of classificationresults on six high-dimensional real-world datasets, whichdemonstrates the discriminability of feature subsets selectedby our method. Concretely, in general, comparing to allcompetitors, our method always achieves comparable or even

2http://www.cs.cmu.edu/afs/cs/project/theo-20/www/data/3http://www.escience.cn/system/file?fileId=82035

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

3013

(a) 20News (b) Binalpha (c) Pixraw10P

(d) Pubfig (e) WebKB (f) Text1

Figure 3: The error bar figure of classification accuracy with different numbers of selected features on six real-world datasets.

highest mean accuracies on almost all datasets with differ-ent number of selected features. Specifically, due to the non-parameter property of our method, it achieves stable perfor-mance on all datasets. Furthermore, our method outperformsESFS in different degrees on various datasets, which resultsfrom that ESFS applies approximate gradient descent algo-rithm to obtain a suboptimal solution, while our method ob-tains a closed-form solution, which guarantees stability of ourmethod’s performance.

4.4 Convergence AnalysisFigure 4 plots the convergence curves of our iterative al-gorithm with different number of selected features m andmodel’s sparsity k on 20News and WebKB datasets. Over-all, our algorithm reaches convergent within 15 iterations inmost cases or even less than 5 iterations on 20News dataset.In detail, a subfigure embedded in each figure is used to facil-itate observation of the curve convergence when k = m. Wediscover the fact that the curve reaches convergent with oneiteration on all datasets, which verifies the high-efficiency ofproposed optimization algorithm.

5 ConclusionIn this paper, we have proposed explicit discriminative fea-ture selection algorithm via employing structured sparse sub-space constraint, which is a NP-hard optimization problem.To our best knowledge, there are no techniques able to obtainclosed-form solution and proves convergence. As the majorcontribution of this paper, we provide an ideal iterative opti-mization algorithm that can directly solve proposed NP-hard

0 5 10 15 20 25 30 35 40 45 50

Iterations0

0.05

0.1

0.15

0.2

0.25

Obj

ectiv

e fu

nctio

n va

lue

m = 40, k = 80m = 30, k = 50m = 20, k = 30m = 10, k = 10

2 3 4

0.18

0.185

0.19

(a) 20News

0 5 10 15 20 25 30 35 40 45 50

Iterations0

0.1

0.2

0.3

0.4

0.5

0.6

Obj

ectiv

e fu

nctio

n va

lue

m = 40, k = 80m = 30, k = 50m = 20, k = 30m = 10, k = 101.5 2 2.5 3

0.2650.27

0.2750.28

(b) WebKB

Figure 4: Convergence curve of objective function value in Eq.(18)with different number of selected features m and model sparsity kon 20News (a) and WebKB (b) datasets respectively.

problem and reaches convergent very fast in both theory andpractice. Experimental results demonstrate the effectivenessof proposed method. In future works, we intend to developnonlinear feature selection model via proposed S3L modulefor future improving discriminability of selected features.

AcknowledgementsThis work was supported in part by the National Key Re-search and Development Program of China under Grant2018AAA0101902, in part by the National Natural Sci-ence Foundation of China under Grant 61936014, Grant61772427 and Grant 61751202, and in part by the Fundamen-tal Research Funds for the Central Universities under GrantG2019KY0501.

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

3014

References[Abid et al., 2019] Abubakar Abid, Muhammad Fatih Balin,

and James Zou. Concrete autoencoders for differen-tiable feature selection and reconstruction. arXiv preprintarXiv:1901.09346, 2019.

[Bishop, 2006] Christopher M Bishop. Pattern recognitionand machine learning. springer, 2006.

[Cai et al., 2013] Xiao Cai, Feiping Nie, and Heng Huang.Exact top-k feature selection via l2, 0-norm constraint. InProceedings of the Twenty-Third international joint con-ference on Artificial Intelligence, pages 1240–1246, 2013.

[Du et al., 2018] Xingzhong Du, Feiping Nie, WeiqingWang, Yi Yang, and Xiaofang Zhou. Exploiting combina-tion effect for unsupervised feature selection by `2,0-norm.IEEE transactions on neural networks and learning sys-tems, 30(1):201–214, 2018.

[Gu et al., 2012] Quanquan Gu, Zhenhui Li, and Jiawei Han.Generalized fisher score for feature selection. arXivpreprint arXiv:1202.3725, 2012.

[Han et al., 2018] Kai Han, Yunhe Wang, Chao Zhang, ChaoLi, and Chao Xu. Autoencoder inspired unsupervised fea-ture selection. In 2018 IEEE International Conference onAcoustics, Speech and Signal Processing (ICASSP), pages2941–2945. IEEE, 2018.

[He et al., 2012] Ran He, Tieniu Tan, Liang Wang, and Wei-Shi Zheng. `2,1 regularized correntropy for robust featureselection. In 2012 IEEE Conference on Computer Visionand Pattern Recognition, pages 2504–2511. IEEE, 2012.

[Horn and Johnson, 2012] Roger A Horn and Charles RJohnson. Matrix analysis. Cambridge university press,2012.

[Liao et al., 2015] Shengcai Liao, Yang Hu, Xiangyu Zhu,and Stan Z Li. Person re-identification by local maximaloccurrence representation and metric learning. In Proceed-ings of the IEEE conference on computer vision and pat-tern recognition, pages 2197–2206, 2015.

[Maldonado and Weber, 2009] Sebastian Maldonado andRichard Weber. A wrapper method for feature selectionusing support vector machines. Information Sciences,179(13):2208–2217, 2009.

[Ming and Ding, 2019] Di Ming and Chris Ding. Robustflexible feature selection via exclusive l21 regularization.In Proceedings of the 28th International Joint Conferenceon Artificial Intelligence, pages 3158–3164. AAAI Press,2019.

[Nie et al., 2010] Feiping Nie, Heng Huang, Xiao Cai, andChris H Ding. Efficient and robust feature selection viajoint `2,1-norms minimization. In Advances in neural in-formation processing systems, pages 1813–1821, 2010.

[Nie et al., 2017] Feiping Nie, Rui Zhang, and XuelongLi. A generalized power iteration method for solvingquadratic problem on the stiefel manifold. Science ChinaInformation Sciences, 60(11):112101, 2017.

[Nie et al., 2019] Feiping Nie, Zheng Wang, Rong Wang,Zhen Wang, and Xuelong Li. Towards robust discrimi-native projections learning via non-greedy l2,1-norm min-max. IEEE Transactions on Pattern Analysis and MachineIntelligence, 2019, DOI: 10.1109/TPAMI.2019.2961877.

[Pang et al., 2018] Tianji Pang, Feiping Nie, Junwei Han,and Xuelong Li. Efficient feature selection via `2,0-norm constrained sparse regression. IEEE Transactions onKnowledge and Data Engineering, 31(5):880–893, 2018.

[Roffo et al., 2015] Giorgio Roffo, Simone Melzi, andMarco Cristani. Infinite feature selection. In Proceedingsof the IEEE International Conference on Computer Vision,pages 4202–4210, 2015.

[Sun et al., 2016] Ying Sun, Prabhu Babu, and Daniel PPalomar. Majorization-minimization algorithms in signalprocessing, communications, and machine learning. IEEETransactions on Signal Processing, 65(3):794–816, 2016.

[Tian et al., 2019] Lai Tian, Feiping Nie, and Xuelong Li.Learning feature sparse principal components. arXivpreprint arXiv:1904.10155, 2019.

[Vu et al., 2013] Vincent Q Vu, Jing Lei, et al. Minimaxsparse principal subspace estimation in high dimensions.The Annals of Statistics, 41(6):2905–2947, 2013.

[Wang et al., 2007] Huan Wang, Shuicheng Yan, Dong Xu,Xiaoou Tang, and Thomas Huang. Trace ratio vs. ratiotrace for dimensionality reduction. In 2007 IEEE Confer-ence on Computer Vision and Pattern Recognition, pages1–8. IEEE, 2007.

[Wang et al., 2014] Zhaoran Wang, Huanran Lu, and HanLiu. Tighten after relax: Minimax-optimal sparse pca inpolynomial time. In Advances in neural information pro-cessing systems, pages 3383–3391, 2014.

[Xiang et al., 2012] Shiming Xiang, Feiping Nie, GaofengMeng, Chunhong Pan, and Changshui Zhang. Discrimi-native least squares regression for multiclass classificationand feature selection. IEEE transactions on neural net-works and learning systems, 23(11):1738–1754, 2012.

[Xu et al., 2018] Jie Xu, Lei Luo, Cheng Deng, and HengHuang. Bilevel distance metric learning for robust imagerecognition. In Advances in Neural Information Process-ing Systems, pages 4198–4207, 2018.

[Yan et al., 2016] Hui Yan, Jian Yang, and Jingyu Yang.Robust joint feature weights learning framework. IEEETransactions on Knowledge and Data Engineering,28(5):1327–1339, 2016.

[Yang and Xu, 2015] Wenzhuo Yang and Huan Xu. Stream-ing sparse principal component analysis. In InternationalConference on Machine Learning, pages 494–503, 2015.

[Zhang et al., 2020] H. Zhang, J. Wang, Z. Sun, J. M. Zu-rada, and N. R. Pal. Feature selection for neural networksusing group lasso regularization. IEEE Transactions onKnowledge and Data Engineering, 32(4):659–673, 2020.

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

3015