General Approach for Analyzing Data to Build a Competency Model

STEP 4 – DATA ANALYSIS AND CODING

Previous blogs have described steps 1 through 3 in Workitect’s competency modeling process. Step 3, Data Collection, was covered in these blogs:

Collecting Data to Build Competency Models
Secondary Data Collection Methods
Conducting Behavioral Event Interviews
Using Resource/Expert Panels to Build Competency Models

The general approach for analyzing the data collected in step 3 has three steps:

  1. Reviewing/modifying a set of generic competencies that will be used as the conceptual framework guiding the analysis.
  2. Separately reviewing the evidence from each data collection method to identify a potential set of competencies, drawn from the generic competency framework.
  3. Reviewing and comparing the evidence across methods to identify a set of potential competencies to include in the competency model.

Each of these steps is described below. Note that the data analysis ends with identifying a potential set of competencies to include in the competency model. Moving from that step to building the actual competency model will be described in a separate blog.

Reviewing/Modifying a Set of Generic Competencies to Include in the Competency Model

If you are using a Multiple Jobs Approach to building the competency model, you will have already identified a set of generic competencies to serve as building blocks for the various competency models you are planning. If you are using the Single Job Approach or the One Size Fits All Approach, you may have used a set of generic competencies that were rated by a Resource Panel or by persons with whom you did Job Analysis Interviews. If, by chance, you have not yet begun to use a set of generic competencies, now is the time to start, because a framework of generic competencies is useful in guiding the analysis.

You will not necessarily end up with competencies as they appear in the initial generic competency framework. You may identify and add competencies that were not initially part of the framework. You may modify some of the competencies from the initial framework. But using the framework will facilitate the analysis of the data.

Based on the experience of collecting the data, you may already have some ideas about competencies that need to be added to the framework. For example, if you are preparing a competency model for Project Managers, you may have decided to add Project Management Knowledge to the framework as a technical skill/knowledge competency. It may be clear that the wording used to describe certain competencies can be improved by substituting language that better fits the job or organization. For example, if the organization in which you are working uses the term, “Driving Innovation” instead of “Managing Change,” you may want to change the competency name accordingly

If it is clear at the start of the data analysis process that changes like these will be needed, you might as well modify the generic competency framework now, so that you can look for and track evidence for these new or modified competencies during the analysis process. It will still be possible to modify generic competencies and add new competencies later on.

Separately Reviewing the Evidence Generated from Each Data Collection Method, to Identify a Set of Potential Competencies Drawn from the Generic Competency Framework

In this step you separately review the evidence from each of the data collection methods that you used – Job Analysis Interviews, for example – and try to answer the following question: Based on this data collection method alone, which competencies should be considered for inclusion in the competency model?

In carrying out this analysis, you may encounter direct evidence for certain competencies from the generic competency framework. For example, all or most of the participants at a Resource Panel may select a competency for inclusion in the competency model. Or, when analyzing Structured Interviews with superior performers, you may find that their descriptions of their behavior during key past events include many examples of some competency.

But your analysis may also reveal indirect evidence that a competency is important. For example, if you learn that an important task for sales representatives is to deliver formal and informal presentations to clients, this task implies the need for the competency, Persuasive Communication. Or if you learn that a key performance measure for a job is Customer Satisfaction, this implies the importance of the competency, Customer Orientation.

Review and Compare the Evidence Across Methods to Identify a Set of Potential Competencies to Include in the Competency Model

Although there will be some overlap in the competencies that are suggested by each data collection method, there will probably be some differences, at least in emphasis. Job Analysis Interviews with managers of persons in the job may lead to the identification of competencies that are most important and visible to the job holders’ managers. Interviews with customers of the job holders will highlight competencies that are most visible and important to customers. Structured Event Interviews may reveal thought processes that lead to the identification of competencies such as Conceptual Thinking that are not as likely to be surfaced through other data collection methods.

One way to compare the competencies revealed by different methods is to construct a matrix of generic competencies by methods, like the one below. You can omit generic competencies that are not identified as important in the analysis of any of the data gathering methods that you used. In each cell, use H, M, or L to indicate whether the competency seems high, medium or low in importance, based on data from this method. In the Overall column, make a judgment about the importance of including this competency in the competency model. You do not need to weight each data collection method equally.

AN OVERVIEW OF METHODS FOR ANALYZING DATA GATHERED DROM INDIVIDUALS

Details of this process, and instruction on analyzing and coding the data collected in step 3, are provided in the Workitect Building Competency Models workshop and practiced by Workitect consultants in the development of job competency models.

Analyzing Data from Job Analysis Interviews

Analyzing Data from Resource Panels
Analyzing Data from Structured Event Interviews
General Principle 1. Recognize Target Data
Target Data is defined as behaviorally specific information, volunteered by the interviewee in response to non-leading questions, about what the interviewee did, said, thought, or felt in one specific past situation.
General Principle 2. Focus on Effective, Impressive Behavior
General Principle 3. Code Against the Generic Competencies and Their Behavioral Indicators
This is one approach to implement coding:
A Simple Coding Process Using a Checklist
We have developed a simple coding process that is appropriate if you are doing a small number of Structured Event Interviews with superior performers and want to use the interviews to help identify behavioral indicators used by these persons.

This coding process involves using a checklist based on the framework of generic competencies and their behavioral indicators.

Reviewing and Analyzing Other Data
Survey Data, Customer Interviews, Observational Data

Details of this process, and instruction on analyzing and coding the data collected in step 3, are provided in the Workitect Building Competency Models workshop and practiced by Workitect consultants in the development of job competency models.

Instruction on developing competency models is also contained in the Quick-Start Competency Modeling program that is included with Competency Dictionary licenses.

©2018, WORKITECT, INC., ALL RIGHTS RESERVED
No part of this work may be copied or transferred to any other expression or form without written permission or a license Workitect, Inc. – 800.870-9490 – licensing@workitect.com

Share Button

Secondary Data Collection Methods

six steps horizontal 3STEP 3 – DATA COLLECTION

In addition to the primary data collection methods that were described in the previous sections, there are several other data collection methods that may be useful in selected circumstances. These secondary data collection methods include:

  • Interviews with Customers
  • Interviews with Industry Experts
  • Observation of Job Holders
  • Surveys of Job Holders

Interviews with Customers

If the job-holders have external or internal customers, the customers can provide useful information about effective and less effective behavior among persons holding this job. External customers often have experience with staff in similar jobs in competitors’ organizations.

Customer interviews can be relatively short (15 to 30 minutes). Possible questions include:

  • What are the skills and behaviors that you have observed in the most effective people that you have dealt with, in this job?
  • In what ways do superior performers in this job differ from average performers and less effective persons in this job?
  • Think of someone in this job that was very effective. What were some of the things that this person did that set him/her apart from less effective persons with the same job.

Interviews with Industry Experts

If a competency model is desired for a new job, especially in an industry that is undergoing rapid change, there will be few people in the organization with much knowledge about the job, beyond the expected job responsibilities. In this case it can be helpful to interview an industry expert from outside of the company. The industry expert should be able to describe:

  • Market and technology trends in the industry
  • Companies that are key players and their relative positions within the industry
  • The challenges likely to be encountered in the new job in the context of the industry

If you understand the challenges likely to be encountered in the new job, you can draw logical inferences about the competencies that will be needed for superior performance.

Learn how our consulting services can help you build competency models >>>

 

Share Button

Conducting Behavioral Event Interviews

six steps horizontal 3

STEP 3 – DATA COLLECTION

A Behavioral Event Interview is a 1 to 1½ hour interview, in which the interviewee is asked to provide highly detailed accounts of how he/she approached 4-7 important accomplishments or other key events from the past year or two in the job. The interviewer uses a probing strategy to get the interviewee to walk through the sequence of what he/she did, said and thought at key points during each accomplishment or event.

The Behavioral Event Interview is usually conducted with superior performers. The assumption underlying the interview is that studying the interviewee’s actions, thoughts and words in these key situations will reveal underlying competencies responsible for superior performance.

For the analysis, each interview is tape recorded and transcribed. An analyst carefully reads each transcript and notes passages evidencing effective behaviors or thought patterns. These passages are noted on index cards or a spreadsheet template, along with the interviewee’s initials and the transcript page and line number. The analysts classify the themes by using a conceptual framework of generic competencies and behavioral indicators. The analysts then meet to review the evidence from their individual analysis and to identify competencies and behavioral indicators for the competency model. This process usually demonstrates that the superior performers used certain generic competencies and behavioral indicators from the conceptual framework used to classify the themes. But the process often reveals new behavioral indicators and competencies that were not part of the original conceptual framework.

Typical Structure of a Behavioral Event Interview

  • Introduction explaining the purpose of the project and of the format and purpose of the interview
  • Brief section on interviewee’s main responsibilities to provide orientation for the interviewer
  • “Event” questions asking the interviewee to provide detailed accounts how he/she approached key accomplishments and other work experiences
  • Follow-up probing of the interviewee’s response to each “event” question, to:
  • A closing question asking for the interviewee’s views about the personal characteristics needed for effectiveness in the job
  • Follow-up probing for examples from the interviewee’s experience

Advantages of Behavioral Event Interviews

  • Provide specific, high-quality behavioral data describing what superior performers do to achieve superior results
  • Surface non-obvious effective behaviors that job incumbents and their bosses may be unaware of or unable to articulate
  • Provide strong evidence for a competency model’s validity – evidence that is especially important if the model will be used for external selection
  • Provide excellent case material that can be adapted for use in developing training materials

Disadvantages of Behavioral Event Interviews

  • Are time consuming to conduct
  • Require extensive interview training and practice to ensure that high-quality data will be obtained
  • Are time consuming to analyze
  • Require training and practice to ensure the quality of the analysis

Structured Event Interviews

The Structured Event Interview is a simplified type of Behavioral Event Interview developed by Workitect to provide many of the benefits of Behavioral Event Interview, while significantly reducing the time and cost required to conduct and analyze the interview. This interview takes about one hour to conduct and focuses on three accomplishments, each of which is related to performance of a different main responsibility. The Structured Event Interview Protocol includes both “event” questions and specified follow-up questions that guide the interviewer through the process of probing each accomplishment. The protocol includes spaces to capture key information in response to each specified question.

This interview is taught in Workitect’s Building Competency Models workshop.

Share Button