WHY ARE SKILLS ASSESSMENTS REQUIRED IN TODAY’S IT ... · change how we develop and assess skills....

WHY ARE SKILLS ASSESSMENTS REQUIRED IN TODAY’S IT ORGANIZATIONS?Carol Gravel, EdD, MEd, SHRM-CP

www.learnondemandsystems.com | 2© Learn on Demand Systems. All Rights Reserved.

WHY ARE SKILLS ASSESSMENTS REQUIRED IN TODAY’S IT ORGANIZATIONS?

By: Carol Gravel, EdD, MEd, SHRM-CP

Over the past few years, we have seen the mainstreaming of cloud computing as well as the increased use

of disruptive technology such as artificial intelligence (AI). For example, a 2017 study by Forbes showed

that hybrid cloud adoption has increased threefold since 2016, and an organization is using, on average,

29 different cloud solutions. These technologies make our lives easier and more productive while also

contributing to the globalization of trade and commerce.

Going forward, we will continue to see continuous changes in technology that will drive the need to

change how we develop and assess skills. Gartner’s Top 10 Strategic Predictions for 2018 and Beyond

shows that the digital transformation will be even more disruptive as the relationship between people and

machines will be blurred. Gartner suggests that to cope, organizations must develop practices to that will

allow them to keep pace. For IT organizations, coping requires creating a “New IT Organization” that has

new approaches to skills development and leverages new organizational models. In essence, keeping pace

requires a workforce that is nimble and can quickly define the right skills and the right people to quickly

respond to change on top of ever-changing technologies, especially those hosted in the “cloud.” These new

approaches will provide IT the ability to do rapid skilling or reskilling of the IT workforce as well as the

ability to quickly respond to business needs.

The need to skill or reskill the IT workforce is urgent. A 2018 McKinsey Global Institute report revealed

that by 2030 as many as 375 million workers, approximately 14 percent of the global workforce, will

change because of digital transformation, automation and advancements in artificial intelligence (AI). Six-

ty-six percent stated that “addressing potential skills gaps related to automation/digitization…is a top-ten

priority” while almost 30 percent stated it was in the top-five. (See graphic on next page.)

http://www.learnondemandsystems.com


Many innovative IT organizations are using agile teams as a new organizational model. These teams are

dedicated, small, cross-functional groups of people who have many or all the skills the team needs to pro-

duce a working, tested increment of a product or service and subsequently a fully functioning product or

service. It is important to note that there is no one model of an agile organization. Each organization must

define what is appropriate based on their internal and external needs. However, the agile workforce must

be deliberately designed to gather and deploy quickly and efficiently in response to requirements. There

are two key definitions of an agile organization:

• Agile: Demonstrating the ready ability “to move with quick, easy grace” and “having a quick, re-

sourceful and adaptable character.” (Source: Webster’s Dictionary)

• Agility: The ability of an organization to sense environmental change and respond efficiently and

effectively to that change. (Source: Gartner)

There are two key attributes of great agile teams; 1) continuous mentoring and 2) continuous skills de-

velopment. One of the largest benefits in working on a true agile team is that members learn from and

mentor each other. As a result, the agile team is greater than the impact made by individual members.

However, it is still vital that members have formal structured learning because it is critical these teams

have the scope and depth of skills needed to support each other’s work. It also guards against groupthink

that can lead the team down the wrong path.



Skill assessments provide an inventory of validated skills

Finding or building the skills needed in the New IT Organization can be challenging. Recent research from

CompTIA suggests that 8 in 10 US businesses are being negatively impacted by the lack of technology

talent. In addition, an EU Commission report revealed a shortage of 900,000 IT professionals by 2020.

Many organizations are creating skills inventory systems to track and analyze individual skills to not only

properly staff agile teams, but also to identify skills gaps. Building a skills inventory can be broken down

into three stages.

Stage 1: Define the competencies (knowledge, skills and abilities) needed for each role and/or job

family. It is vital that you look beyond the job description to define competencies. Consider

using focus groups of high performing agile teams as input as to what competencies are need-

ed. Then, for example, define competency levels from novice to expert.

Stage 2: Define a skills inventory “system.” Once you have the competencies defined, you will need to

organize them to search and analyze the data to determine staff and skills gaps. It is import-

ant to define standard terminology and organize the data to be able to produce effective que-

ries. Several enterprise LMS’s provide a competency/skills inventory module or you can build

your own. For example, PwC has the Digital Fitness Assessment app. Employees assess their

own technology skills and create personal learning paths to increase the scope and depth of

their skills.

Stage 3: Assess the skills. There are two common approaches to assessing skills. The first is self-as-

sessment, and the second is formal assessment. Self-assessment involves having an employee

or their manager assess their level of proficiency in each competency. With self-assessment,

it is important to communicate the assessment’s objective to staff because if employees are

thinking this will be a performance assessment instead of a competency assessment for staff-

ing, there may be challenges with getting accurate data.

A formal assessment is a systematic preplanned method of “testing” to see if the employee has the

competency or skills. Formal assessments can be tests using a set of questions or real-world situations

or challenges. Two common types of assessments, activity-based and performance- based, are de-

scribed in greater detail below.

It is important to note that each stage is not a one-time activity. Technology is always changing, so it is

important your skills inventory also is adaptive to change. I recommend learning leaders use an iterative

approach to maintaining the skills inventory. For example, in tandem with performance review cycles,

learning leaders collaborate with IT to review competencies to ensure they are up to date. Then, adjust

the skill assessments as needed.



Skill assessments allow organizations the ability to:

Assign individuals to teams or projects based on knowledge and/or necessary skills

With the skills inventory in place, you can perform simple or complex analyses of the skills data. From

a simple query of who has Java skills to a more complex query of who has Java skills that are available

to start in one week on a project that requires them to work in Northern Ireland. Beyond queries of

who has what skills you can also see where there are skills gaps in terms of current or future technol-

ogy. For example, IBM has a tool called Talent Match that allows leaders to identify employees with

specific verified skills. Employees register to be part of the Talent Match system. Their verified creden-

tials/assessment results are then available, so a leader can search the system to locate individuals with

specific skills.

Create a learning culture

It is important to note that IT professionals in the New IT Organization must possess the willingness

and desire to continuously learn new skills. With the skills inventory in place, you can support a contin-

uous learning culture in your organization. In a learning culture, the employee, mangers and teams are

responsible for seeking out the knowledge or skills they need to best perform their work. In addition,

knowledge and skills are shared among teams and, thus, creates a more agile organization. By having a

skills inventory, individuals, managers and teams can see who in the organization they can reach out to

for support and/or coaching.

HOW TO GET STARTED

Define a skills framework

Having an inventory of skills is vital data for the New IT Organization. However, the key first step is to

define a skills framework that you can use to classify and organize those skills. As a former Director of IT

Training, it was important for me to answer two key questions associated with assessing skills to deter-

mine what training may be needed:

1. How can we objectively measure achievement of non-technical skills?

2. How can we effectively assess the quality and output of IT technical skills?



The “T-Shaped” skills framework is an ideal framework. The vertical line on the T represents the depth of

skills needed. The horizontal line on the T represents the breath of skills needed.

The T-Skills Framework suggested includes 3 skill domains.

1. Technical

2. Interpersonal

3. Process



IT professionals in the New IT Organization must still have an advanced level of technical or IT skills.

However, they also need foundational skills in processes like Agile or DevOps. In addition, they must

also possess at least a foundational level of interpersonal skills such a teamwork. Assessing technical

and non-technical skills require different assessment criteria. Interpersonal skills are assessed based on

feedback on attitudes and behaviors. IT technical skills are is generally assessed in terms of the quality

and output of the task—for example, the quality of the lines of code for software developers. Typically, IT

organizations pay little attention to teaching IT professional process or interpersonal skills as it is often

assumed technical skills are the most important. However, in the New IT Organization IT professionals

need to be able to work collaboratively in teams. For example, a software developer requires a depth of

technical skills, such as full-stack development (e.g. Java, .Net, and Node.js). However, writing code re-

quires more than just proficiency with a specific language; it requires a breadth of interpersonal skills,

such as problem solving and communications, as well as a breadth of knowledge of process skills, such as

Agile.

Define a skills assessment strategy

There are two steps in defining a skills assessment strategy; what types and when.

First, define what types of assessments are important to your organization.

1. Knowledge-based assessments

Knowledge-based assessments are a traditional part of learning design. When designing learning, we

begin by creating learning objectives that are typically based on the six levels of Bloom’s Taxonomy.

Level 1: Knowledge – Ability to recall information

Level 2: Comprehension – Ability to interpret information

Level 3: Application – Ability to apply information in a new way

Level 4: Analysis – Ability to break down information and show relationships

Level 5: Synthesis – Ability to bring together information to solve a problem

Level 6: Evaluation – Ability to make effective judgments based on information

We then create knowledge-based assessment questions on those objectives. For example,

A: Knowledge Question: Define …

B: Comprehension Question: Explain the relationship between…

C: Application Question: Could the methods … be used to …? Explain your answer.

D: Analysis Question: Compare and contrast ….

E: Synthesis Question: Create a …. that could ….

F: Evaluation Question: Evaluate … in terms of …


http://Node.js


2) Performance-based assessments

In general, a performance-based assessment measures a person’s ability to apply the skills and knowl-

edge learned. Typically, a performance-based assessment is an activity or challenge to complete.

Performance-based assessments can range in complexity level from a simple response, such as a

short answer, to a scenario where they need to provide a sample solution. The scenario requires a

higher-level of thinking because students must create or do something to achieve success. The most

effective performance-based assessments are ones that closely mirror real world situations.

Key characteristics of performance-based assessments:

1. Complex

2. Authentic

3. Process/product-oriented

4. Open ended

5. Time-bound

Example of an Assessment Activity:

Activity: Create a few storage accounts in a Cloud Subscription that will be used later in the lab.

The instructional designer creates a PowerShell script (see below) that will check if the storage

accounts were created correctly. This script is then used to assess if the student has created a

storage account correctly:

$storAccount = Get-AzureRmStorageAccount `

-ResourceGroupName ‘[email protected]’ `

‘The Storage Account has not been created.’

‘You successfully created the storage account.’

if ($storAccount -eq $null){

-Name ‘[email protected]’ `

-ErrorAction Ignore

} else {

}

$result = $false

$result = $true

$result


mailto:[email protected]

mailto:[email protected]


This is what the student will see in the lab:

• The student clicks the Score button, and the scripts will begin executing:

If the student created the storage accounts correctly, they will receive a message that says

“You successfully created the storage account.”

If the student did not create the storage accounts correctly, they will receive a message

that says “The Storage Account has not been created”.

Example of an Assessment Challenge:

Challenge Scenario: Can you Provision Data Storage for a Web App?

You will be asked to complete a large configuration task to showcase your mastery of the topic,

which could span multiple products, technologies, and platforms. You will not receive task

guidance, and an automated score will be provided to you upon completion.

In this challenge, you will provision all data components for a line of business application. This

includes Azure storage, Azure SQL database, Cosmos DB, and search.

Tasks:

1. Create and populate a storage account with blob containers.

2. Provision an Azure SQL database

3. Provision an Azure Database for MySQL server

4. Provision an Azure Cosmos DB database

For a live demo see: https://www.learnondemandsystems.com/live-demo-labs/


https://www.learnondemandsystems.com/live-demo-labs/


It is important to note that the traditional lecture/lab/test approach to training design will not help the

New IT Organization quickly adapt to rapid changes. IT training must adopt new instructional design mod-

els that include challenge-based or project-based scenarios where learners “learn by doing” and receive

near real-time feedback and assessment on their proficiency in completing the scenario.



Second, define when you want to use assessments.

The most common times for conducting assessments are prior to training (pre-assessment), during train-

ing (formative assessment) and after training (post-assessment).

Pre-Assessment: An initial assessment to gather data on current knowledge and/or skills. This is im-

portant because it provides you with baseline data that you can use to:

a) Determine if there is a gap in skills. If there is a gap, identify specific training to fit their needs.

b) Have an inventory of what skills an individual already has so you can assign them to projects or

teams.

Formative Assessment: An assessment to gather data during training to validate understanding of

the material. If they do not understand, then re-teaching using a difference approach or repeating the

lesson may be needed.

Post-Assessment: A post training assessment to validate the new knowledge and/or skills acquired

from training. This information is then available in your skills inventory.

For example, an assessment strategy to validate that someone can define the elements of the Agile

Manifesto would be to have a pre-assessment, a formative assessment and a post assessment.

KNOWLEDGE/SKILL

Define the elements of the Agile Manifesto

WHEN TO ASSESS

Pre-Assessment

Formative Assessment

Post-Assessment

WHAT TYPE OF ASSESSMENT

Activity-Based: Level 1 Multiple Choice

Activity-Based: Level 2 Matching

Performance-Based: Scenario as a project manager explaining Agile Manifesto to their team



CLOSING

IT organizations must develop organizational practices, such as skills assessments and the use of agile

teams, to keep pace with the inevitable digital transformation. In the next five years, the demand for IT

talent will significantly outstrip supply. For agile skills, the demand is expected to be four times the supply.

For big-data talent, it is projected to be 50 to 60 percent greater than anticipated supply. It will require

IT organizations to look at new organizational models and approaches to skills development. Then, use

that data to create agile teams to be able to scale up and down to respond quickly. To address skill gaps, IT

organizations are also facing the requirement to reexamine the traditional lecture/lab/test approach and

adopt new instructional design models like challenge-based or project-based scenarios where “learning

by doing” incorporates near real-time feedback with limited guidance to encourage applied learning by

experimenting. In other words, rapid changes in technology require the New IT Organization to take a

different approach on talent management. Focusing on skills assessment and “learning by doing” will be

key tools in meeting the challenges of the digital transformation.


WHY ARE SKILLS ASSESSMENTS REQUIRED IN TODAY’S IT ... · change how we develop and assess skills....

Documents

Transcript of WHY ARE SKILLS ASSESSMENTS REQUIRED IN TODAY’S IT ... · change how we develop and assess skills....