Academic Paper

César Fernández-Llano ✉ (Department of Business, Universitat de Barcelona, Spain)
Merce Bernardo Vilamitjana (Department of Business, Universitat de Barcelona, Spain)
Yuliya Kasperskaya Riabenko (Department of Business, Universitat de Barcelona)

PDF

Introduction

Organizational leaders face constant work-place changes, challenges, and stress. In this context, executive coaching has been adopted widely within and across organizations (Rekalde, Landeta, & Albizu, 2015) as they seek to develop leadership skills, engage in succession planning, and navigate organizational change (Grant, 2014). Although coaching has become an established and popular intervention, there is limited evaluation of coaching programs by organizations and little consensus among academics as to the best mechanism for evaluation, as well as “incredibly limited use of objective outcome measures or measurement of distal and longitudinal impact of coaching” (Grover & Furnham, 2016, p.25).

If “the primary measure of success of executive coaching is its impact on the development of more effective work-place behavior” (Passmore, 2007, p.69), it is convenient to consider the context of the collaborative coaching process (Gettman, Edinger, & Wouters, 2019). This process involves the partnership of three key stakeholders: the coach, the coachee (i.e. the executive), and the coachee's sponsoring organization to link individual goals of the intervention to strategic organizational objectives (Athanasopoulou & Dopson, 2018).

This approach could help coaches to better meet expectations of coaching purchasers (e.g. HR departments) when the results of coaching are measured by positive business results and enhanced organizational-level outcomes (Athanasopoulou & Dopson, 2018). The use of business results indicators can help academics and coaches reduce the perception that executive coaching “cost is high, organizational benefit is moderate, and cost-benefit is uncertain” (Dagley, 2006, p. 43). This situation has persisted for years according to recent studies analyzing “what are the desired outcomes of coaching?” (Boyatzis et al., 2022, p. 204) and suggesting more focus on value creation for all the coachee’s stakeholders.

In addition, as organizations tighten their budgets and make decisions about resource allocation, they are increasingly concerned about the effectiveness of coaching engagements relative to their costs (Ely & Zaccaro, 2011). This lack of clear organizational impact contributes to sustaining barriers to coaching: in a recent consumer awareness survey, 15% of respondents indicated that they “do not believe that there is evidence that coaching works” (ICF, 2017, p.10).

The research objective is to address this gap; to this end, a methodology to evidence the impact of executive coaching on organizational results is proposed. A specific framework, based on the Balanced Scorecard, allows to evaluate the impact of executive coaching via a longitudinal process of Industrial Director.

Literature review

In this section, Balanced Scorecard relationships between perspectives, and their application to measurement of executive coaching efficacy are analyzed.

Executive coaching and Balanced Scorecard

Executive coaching is defined as a one-on-one relationship between a professional coach and an executive (coachee) to enhance the coachee's behavioural change through self-awareness and learning, and ultimately contribute to individual and organisational success (Bozer & Sarros, 2012). However, there is very little empirical evidence of its impact on organizational outcomes (Boysen, Cherry, Amerie, & Takagawa, 2018). In practice, even companies that invest a considerable amount in services of this type have not introduced processes to evaluate the returns on their investment, and therefore continue to base their decisions on perceptions (Walker-Fraser, 2011; Yates, 2015). However, such a situation can be counterproductive, both for the executives themselves and for coaching professionals. Yates (2015) points out that almost a third of the companies analyzed in her study are dissatisfied (a score of 6.69 out of 10) with the coaching services they receive. The most common reason offered for their dissatisfaction was the lack of robust evaluation methods and/or the difficulty in measuring the return on investment (ROI) (De Meuse, Dai & Lee, 2009; Grant, 2012b).

Grover and Furnham (2016) highlight the need to evaluate the efficacy of coaching using objective multisource measures and longitudinal approaches. But the same authors hold that the difficulty is rooted in the coaching intervention itself, because coaching is usually a customised, one-on-one intervention and coaching outcomes and goals differ from session to session and from individual to individual, for example: two salespeople may both undergo coaching, one looking to improve their relationships skills and the other their time management skills. “They both may see a considerable improvement in these areas but this may not be accurately reflected in a standardised measure of performance, which is a requirement for an academically rigorous study” (Grover and Furnham, 2016, p. 24). Another way to look at that same example is to focus not only on the people but also on their professional role. From this perspective, in the aforementioned cases, there are objective indicators of results (sales, margin, new customer acquisition) as well as efficiency of the commercial process (sales per customer; the number of visits before a sale; average order volume; cross-selling; up-selling) that allow comparisons within the same company and even the same sector (DeLoyd, Valdivieso, Vonwiller, & Viertler, 2017). These indicators can be linked to action plans between sessions to appreciate the evolution from the beginning to the end of the process. Therefore, they can be useful for the measurement approach proposed by Ely et al. (2010): formative (during the coaching process) and summative evaluation (final results achieved).

Regarding the disparity of objectives, Grant (2012) considers the objectives as internal representations of the desired states or outcomes and distinguishes a hierarchy in which the maximum level would be of higher and broader order, with more abstract goals, and would be followed by more concrete objectives that this aspiration implies and, finally, by the objectives identifying specific steps to achieve all the above. In the definition of higher-level objectives, organisational inputs can be incorporated and alignment with them, since they will be related to professional performance (Ely et al., 2010; Athanasopoulou and Dopson, 2018; Longenecker and McCartney, 2020), which opens the door to linking them with a management indicator. The lower hierarchical order objectives, which are more specific and changing, are proposed by coachees who set the agenda (Jones, Woods & Zhou, 2021) in this case, the indicators can be more personal and subjective (de Haan, 2021). Therefore, several categories of objectives and indicators can be used throughout the coaching process; the intermediate, more stable and related to organisational objectives, can incorporate performance or management indicators, and for this, it is useful to resort to the Balanced Scorecard.

The leadership literature has a similar view: Kaiser, Hogan & Craig (2008) propose conceptualizing leadership and evaluating leaders in terms of the performance of the team or organization for which they are responsible. Yukl (2012) holds that leadership effectiveness should be assessed from the perspective of multiple stakeholders, with multiple criteria and objective measures of organizational performance. Kaiser et al. (2008) state that psychologists must include Balanced Scorecard outcomes in empirical leadership research to be more relevant to real organizations, and Walker-Fraser (2011) propose using the Balanced Scorecard approach as a method for evaluating the impact of executive coaching.

The rigorous use of the Balanced Scorecard (Kaplan & Norton, 2005; Banker, Chang & Pizzini, 2011; Kaplan, Norton & Rugelsjoen, 2010) has been shown to offer several advantages, because it allows:

• establishment of relationships between variables drawn from different perspectives (financial performance, customers, internal processes, and learning and growth).
• incorporation of different types of management indicators from each of these perspectives, at different organizational levels. This allows specify the links in the executive coaching process using action plans that include objective management indicators and business outcomes.

Considering the interrelation between different perspectives of the Balanced Scorecard approach and variables that may be relevant in this case, we specifically highlight the following:

• Links between learning and growth and internal processes. Bontis & Fitz-enz (2002) validate the links between human capital and business performance, reporting evidence of a significant impact on internal processes, and productivity. Bourne, Pavlov, Franco-Santos, Lucianetti & Mura (2013) report certain HR management practices that contribute to performance, namely ‘engagement’ and ‘a communication/guiding mechanism’. De Leeuw & Van Den Berg (2011) identified three independent clusters of operator behavior that positively correlate with performance improvement and performance management practices, namely ‘understanding’, ‘motivation’ and ‘focus on improvement’. Vanichchinchai (2012) shows that employee involvement has a direct impact on a firm’s supply performance, measured in terms of cost, flexibility, relationship, and responsiveness. Taesung (2015) assimilates learning and growth with intellectual capital and shows its causal impact on processes and customers.
• Links between internal processes, customers/market, and economic-financial performance. Ou, Liu, Hung & Yen (2010) show that improvements in supply chain management not only enhance internal operational performance, but also external customer satisfaction and, consequently, a firm’s financial performance.
• Links between customers/market and economic-financial performance. Li, Ragu-Nathan, Ragu-Nathan & Subba Rao (2006) show that the strategic supplier partnership, together with the level and quality of information sharing are determinants of competitive advantage (in terms of cost, quality, product innovation, and time-to-market), which in turn improves market and financial performance.
• Relating all perspectives simultaneously. Wang & Chang (2005) show that human capital indirectly affects performance through innovation capital and process capital. Moreover, innovation capital has an impact on customer capital, which ultimately affects performance. Gómez-Cedeño, Castán-Farrero, Guitart-Tarrés & Matute-Vallejo (2015) establish that HR management indirectly influences customer satisfaction and this, in turn, influences a firm’s economic-financial performance. Also, in part, the operational results of the supply chain influence the economic-financial outcomes. Significant contributions include the findings of Malbašić, Marimon & Mas-Machuca (2016) who demonstrate that certain organizational values impact organizational effectiveness and, furthermore, that balancing attention between the four perspectives of the Balanced Scorecard moderates this impact. Llach, Bagur, Perramon & Marimon (2016) offer a complete causal map that shows the significant mediation of the ‘internal processes’ and ‘customer’ constructs, especially in industrial firms. They also demonstrate the importance of leadership for achieving high performance.

In short, these studies establish empirical relationships between all the Balanced Scorecard perspectives. Moreover, this strictly (macro) organizational approach can also be applied at the individual (micro) level (Kaufman, 2017). Taking all these findings together, we can relate the contributions of the Balanced Scorecard with the results of Mahsud, Yukl & Prussia (2011) about human capital, efficiency, and adaptation as strategic determinants of firm performance. Therefore, we can conclude that the Balanced Scorecard approach provides a method to integrate ideas from leadership, human resource management, strategic management, change management, and organizational design. Accordingly, we can use the Balanced Scorecard to measure the global leadership impact at different organizational levels.

Additionally, the Balanced Scorecard approach can support strategic learning and act as a guide to mobilize managerial action (Kasperskaya & Tayles, 2013). Also, when designing and executing action plans, this kind of map has proved useful for the construction of meaning and communication in organisations, even if causal links may contain inconsistent and equivocal relationships, since “the causal map contains the structure, the process, and the material from which agreements and conflicts are built when people coordinate action” (Kasperskaya & Tayles, 2013, p.19).

In the next section, the application of this framework to a specific case of an Industrial Director is detailed.

Methodology

The present study employed a within subject A-B-A single case design: A (baseline) – B (intervention) – A (intervention withdrawal). The data included interviews with organizational stakeholders and objective management indicators. In order to detect possible variations, observe evolution and make a longitudinal comparison, the approach of Gerring & McDermott (2007) was adopted to collect sufficient time-series data pre-intervention, during the intervention and post-intervention. In this case, values for two years were obtained. This design also conforms to the conditions established by Yin (2009) for the analysis of time series. The process and data sources are shown in Figure 1.

Figure 1: Design of the study: PRE pre-assessment; POST post-assessment; FUP follow up-assessment. Objective data source: management system of organization

Organization and coachee

The company employs more than 180 workers and its annual sales value exceeds 40 million euros. At the express request of the company, names and identifying details have been omitted to protect the privacy of individuals. The executive coaching process responded to the need to improve the performance of the firm’s industrial division, because the Industrial Director was not fulfilling the firm’s expectations. This role had a significant organisational impact because it managed 65% of the budget and a team of 76 people.

The Industrial Director had an overly analytical and technical approach. In the words of the Managing Director:

“There is a misapprehension concerning his role (which he considers to be more technical than managerial in nature) and there are doubts even about the job profile, given that the manager is highly analytical, rather than taking action and managing his teams”.

Managing Director considered that the industrial area results presented shortcomings in meeting project deadlines, product launches, and customer service. These objective data were a clear indicator of the director’s performance to whom the executive coaching was addressed.

Procedure

Having validated the conditions for initiating the executive coaching process, i.e., organizational support, client readiness, expectations (Ely et al., 2010), the first author (ICF accredited coach with over ten years of experience) delivered the coaching process in the framework of an academic collaboration. Executive coaching took place between November 2015 and April 2016, with six monthly sessions between 90 to 120 minutes. In the first session, the coachee defined his own vision: “To perform well the role of Industrial Director”, based on an ideal self, to lead to sustained and desired change (Boyatzis, Rochford & Taylor, 2015). The vision of Industrial Director had three intermediate goals: the improvement of the team’s productivity; the improvement of the coordination with other areas; and improving the quality of the customer service. Each of these goals is related –respectively– with the learning and growth, internal processes, and customers perspectives of the Balanced Scorecard framework (Kaplan & Norton, 2005) and allowed the linking of personal aspirations with the strategic goals of the General Manager. The Balanced Scorecard perspectives and indicators were a point of reference to integrate and relate the expected outcomes from the action plans and to design the following steps, specifying which indicators would serve as evidence of progress. In the learning and growth perspective, the coachee detailed aspects related to the team’s discipline, rigor and efficacy which were reflected in the accident rate, productivity and maintenance indicators. The improvement in coordination included features related to this area (the installation of new equipment, a new planification system) as well as other areas: Laboratory, to improve the stability of formulas; Marketing and Sales to shorten the launch of of new products or promotions. Improvement in all these areas was necessary to satisfy the aggregated indicators of customer service. Since all of these indicators were objective and part of the regular monitor of the Industrial Director, they were added to the tracking of the action plans during the coaching process. The actions required to reach these improvements included: varying monitoring habits and increasing the level of demand with one’s team; holding difficult conversations or showing greater assertiveness in other areas; driving changes in internal processes.

Figure 2 illustrates how the hierarchy of goals of the coachee are channeled into operational indicators. In this figure the level of more specific goals related to action steps of each session is omitted.

Figure 2: Hierarchy of goals. Source: Own elaboration based on Grant (2012a)

The iterative action plans were the basis of formative evaluation, tracking the evolution of each phase of the process. Objective information sources and stakeholder’s interviews were employed to track the evolution. Additionally, the draft of the case was reviewed by the company itself, which guarantees the reliability of the data reflected and the chain of evidence. We therefore comply with the three criteria of construct validity proposed by (Yin, 2009).

Variables

We examine the effectiveness of the executive coaching process in accordance with key concerns expressed by Athanasopoulou & Dopson (2018): “For executive coaching purchasers (e.g. HR department), executive coaching success means positive business results” (p.75). Therefore, we assess the results from the point of view of the General Manager and the HR Director, in terms of behaviors (which indicate learning) and the business results. Drawing on contributions from management systems analysis (Banker, Bardhan & Chen, 2008; Kinney & Wempe, 2002; Camacho-Miñano, Moyano-Fuentes & Sacristan-Dıáz, 2013), we track operational variables to measure the final and distal outcomes of executive coaching:

• the key customer service rate (due to its direct impact on sales).
• the stock availability rate (a measure of service and flexibility).

Results

To obtain an overview that would allow longitudinal comparisons, the data collection covers a two-year period (Gerring & McDermott, 2007), including a series of objective indicators from different management levels, as provided in the action plans of each session. These included, for example, compliance with launch dates of new products; meeting equipment installation deadlines; maintenance indicators; accident rates; key customer service rates; and stock availability rates. These plans were verified with real data from the company’s records on management systems and interviews with General Manager and HR Director.

Objectives achieved

At the end of the executive coaching process, the General Manager and the HR Director confirmed that the Industrial Director had modified his management habits, increasing the demand and systematic monitoring team. The Industrial Director had managed to implement a new planning system and improve service agreements and coordination between areas, which, in turn, generated an improvement in the results of cross-cutting projects. As a result of these advances, customer service improved and with it, sales. Owing to the previous achievements the Industrial Director obtained the support of the local board, and later the approval of the Headquarters, for new investment plans.

This testimony can be associated with improvements in critical competencies (self-awareness, achievement orientation, conflict management and influence) that indicate effective performance (Boyatzis & Jack, 2018).

Measurement of sustainable development of managerial role

Drawing on contributions from management systems analysis (Banker et al., 2008; Kinney & Wempe, 2002; Camacho-Miñano et al., 2013), we track operational variables to measure the outcomes of executive coaching and increase the external validity of the research:

• the key customer service rate (with direct impact on sales).
• the stock availability rate (quality of service and flexibility).

Figures 3 and 4 represents the evolution of two examples of service rate, differentiating the previous period from the executive coaching process, the duration of executive coaching, and the evolution after completion of executive coaching.

Figure 3 shows the service rate achieved with a key external customer, in terms of sales that constitute a particularly high volume for the company. In the words of the Industrial Director:

“Thanks to the new planning system, we’ve been able to meet this objective – which initially seemed impossible – on a regular basis, and when we have fallen short, we’ve been able to recover quickly. The financial impact of not complying with this rate is 2% of the amount not serviced. The new planning system was implemented on 01/04/2016, and you can see that, since the system start-up, we’ve operated at above 98%.”

Figure 3. Evolution of service rate, Client A. Source: Internal management data of the company (2016)

This evolution is consistent with the action plans observed, starting with initial progress in the form of a proposal for a new planning system in December and its effective implementation in April’ (see Figure 3).

Figure 4 shows the evolution in the service rate applied to a company within the same group (also a key customer in terms of sales volume), which improved in terms of both stability and compliance. In this case, the improvement was achieved because of a renegotiation of the previous agreement in January.

Figure 4. Evolution of service rate, Group Client. Source: Internal management data of the company (2016)

To confirm a significant difference between the values for the two years under comparison, we used the Wilcoxon test (Wilcoxon, 1945), which was selected for three reasons: (1) the researchers were interested in the change in variables under two related conditions (pre-and post-executive coaching), (2) to minimize the likelihood of type I error due to the small sample size of temporal measures (12 months and 11 months), and (3) the lack of confidence in a normal distribution. The results of the test allow us to reject the null hypothesis (p> 0.05). The negative sign indicates significantly lower performance in 2015, as shown in Table 2. If we compare only the six previous and subsequent months to the executive coaching process, we observe similar results. Therefore, there have been significant and positive changes in management by the Industrial Director. No staff changes were made in the composition of the directly reporting teams, nor in the composition of directors’ team, and no additional programs or systems were implemented during the executive coaching period. The plant’s activity increased by 20% in volume compared with that in the financial year 2015, indicating that the changes to the Director’s management behaviors occurred under difficult conditions, and were therefore not the result of external factors. The General Manager and HR Director confirmed these circumstances. In conclusion, from the organizational stakeholders point of view, no external factors explain the improvement to the industrial division of the company, thus supporting the idea that the positive and sustained impact of the Industrial Director’s

Table 1. Wilcoxon signed rank test for related samples

* p<.05.    ** p<.01.

Discussion

The aim of this research was to demonstrate the possibility of using the Balanced Scorecard framework to measure the impact of an executive coaching process on organizational results. In the analyzed case, the Balanced Scorecard perspectives are useful to explain the translation of leadership development into objective indicators in a longitudinal and detailed manner. The results provide initial evidence that the Industrial Director improved his management habits and enhanced his organizational outcomes by achieving significant changes in terms of internal processes and delivery. Quantitative evidence showed that the changes began during the executive coaching process and were sustained over time.

Implications for Research

This study makes several contributions at the academic level. First, it is one of very few studies that provide an evaluation of the efficacy of executive coaching in relation to organizational improvement, using objective management variables with longitudinal data, and showing that the impact is sustainable over time in a specific context. This approach answers recent calls in this direction (Kaiser et al., 2008; Ely et al., 2010; Athanasopoulou & Dopson, 2018).

Second, the evaluation methodology, using the Balanced Scorecard framework (Kaplan & Norton, 2005), is a potential replicable contribution and can be used to analyze the impact of executive coaching on business performance using empirical data. It also can allow the establishment of relationships between the issues worked on with the coachee and their subsequent impact at different levels of management (Malbašić et al., 2016; Llach et al., 2017). The findings are consistent with the previous works on employee management impact on a firm’s supply performance (De Leeuw & Van Den Berg, 2011; Vanichchinchai, 2012); improvements in supply chain management; and enhancement of internal operational performance, external customer satisfaction, and firms’ financial performance (Ou et al., 2010; Wang & Chang, 2005; Gómez-Cedeño et al., 2015). In conclusion, different types of operational indicators allow the impact of executive coaching to be tracked. Nevertheless, due to the complexity of the business environment a relationship found to apply at one time and under a given set of conditions will not necessarily be observed in the future or under another set of conditions. Therefore, the possibility of causal relationships or maps cannot be treated as continuous in its nature, but rather considered probabilistic (Kasperskaya & Tayles, 2013).

Third, the paper contributes to overcoming some of the theoretical limitations reported in the literature in relation to the difficulty of attributing an impact on performance to a specific intervention, because of the diversity of strategic measures, multiple routes to achieve them, and differing temporal frameworks (Levenson, 2009), since the Balanced Scorecard integrate all these aspects. We also contribute to refuting the frequent criticism that the use of ROI or quantification does not consider benefits that cannot be easily estimated (Grant, 2012b; Levenson, 2009), because the methodology used allows incorporation of these qualitative aspects.

Implications for practice

For academics and practitioners, the use of the Balanced Scorecard allows categorization of the impact of expected behavioral patterns and resolves: a) objections due to the difficulty in homogeneously evaluating executive coaching processes when the responsibility level and interdependency with others is high (Levenson, 2009), and b) objections that appeal to a different nature of topics to work on because the content of the executive coaching engagement varies widely (De Meuse et al., 2009). It contributes also to specifying the expected outcomes of a executive coaching process, enhancing a common framework for future comparisons (Grover & Furnham, 2016). This approach resolves, in addition, the common dichotomy between the summative evaluation (results) and the formative evaluation (process) (De Meuse et al., 2009), because it integrates both aspects (Ely et al., 2010). These contributions can serve as the basis for applying a systematized methodology to evaluate executive coaching processes.

At the practitioners’ level, the Balanced Scorecard framework helps to relate individual or aspirational objectives with strategic and organisational objectives in the beginning of the executive coaching process. At the end of the process, the use of the Balanced Scorecard framework, integrating business indicators, allows communicate the obtained coaching outcomes in language that is appealing and understandable for the managers and executives.

Limitations

The study examined a case within one industrial company, single organisation, one participant and specific context; therefore, due caution must be exercised in its application to other formats of executive coaching (e.g., team or group processes) or to other organizational contexts or sectors.

Future lines of research should aim to investigate whether cultural factors or size represent limitations to the model or require its modification; it would additionally be of interest to test the applicability of the proposed methodology in other sectors, in small- and medium-sized companies, and in other countries. It would also be interesting to examine how a similar approach may be applied to measuring the impact of executive coaching on teams.

Acknowledgements

The authors would like to acknowledge the help of Dr Richard E. Boyatzis for examining an earlier version of this paper and facilitate valuable orientations.

A previous and reduced version of this article has been presented at ICQEM18 congress.

References

About the authors

César Fernández is PhD candidate in Universitat de Barcelona Business School. His research focuses in measuring the impact of executive coaching on business results.

Merce Bernardo is Professor at the department of Business. Her research interests focus on quality management and its relationships with other managerial practices.

Yuliya Kasperskaya is an Associate Professor at the department of Business. Her research focuses on management control and financial practices in private and public organizations.