Evaluating the 4-Fold-Integration Tool

This relates to tool no.78 and its use at Pines Calyx
Data for evaluation

What Were the Research Questions?

• We wanted to know the best conceptual framework for managing all of the other tools coherently.
• We were looking for a suitable tool for monitoring successive evolutionary stages within a team's development.
• We were looking for a tool or framework that could help us to map, loosely structure and facilitate the transition from a 'me' perspective to a 'we' perspective in the design teams.
• We hoped to find one that would be less clumsy and mechanical than Cog's Ladder.
  • N.B. This should allow for the identification of individual self-identity and the team's self-identity?
  • - Its purpose is mainly to assist the organizers/facilitators in co-designing and staging the event and fostering good teamwork.
  • - Therefore only the organizers should be aware of this tool.
  • - Ideally, it should offer suitable 'levels' of involvement that match how designers might operate.
  • - This should enable them to decide when/if to introduce a more appropriate (i.e. level) tool.

How Successful / Effective Was this Tool?

• How well did this tool work in promoting synergy?

1. This tool has proved helpful for the organizers - it seemed less clear for the team players
   (Not surprising, given its intended purpose.)
2. However, it may need further elucidation to be work for unaided novices.
3. We feel that it was highly effective for promoting synergies at different levels.

• Was synergy present as a by-product, or an emergent 'trace', rather than as something actively sought?
• The four-fold integrative framework actively fosters the conditions for synergies at the level of the self, team and whole context to emerge
• Was there another reason (other than synergy) for this tool/experiment?
• The framework also acts as a planning (in the loosest sense of the word), organising, logistical device and is vital for the co-ordination of the m21 team as well as the experience of the design teams and advisors
• and so organising a time frame is one example of the use of this tool that is not just about prompting synergy (although hopefully this approach challenges a more linear way of timetabling that might not be as effective for prompting synergies at the level of the team organising the event and the participants who take part in the event)

Particular questions:

• How did the participants experience the tools? - none mentioned it afterwards
• What was the level of exchange between the participants? (On a continuum from data to wisdom)
• How well did the tool work in practical terms? - It enabled us to notice that the group needed a departure from the original schedule... i.e. it was effective.
• What was the level of exchange between the m21 design researchers? - The design researchers are also counted as participants in the workshop and used the four-fold framework as a logistical tool as well as a tool to facilitate team one and team two's development. It became apparent in the workshop that we needed time and space within the structure of the four-fold framework for the m21 team to come together and have some reflection and adaption time. (At the workshop we found ourselves at one point in the ladies toilets trying to reconvene and respond to initial changes). It might, therefore, be helpful to anticipate and better prepare for how we as a team evolve throughout the workshop. For example, if a team researcher is unhappy with the way the workshop unfolds, there needs to be a 'feedback loop' pre-prepared for them to voice this so that other researchers don't pick up on their discomfort and read it in different ways, thus, negatively affecting how 'we' as the m21 team develop.
• How well did the tool meet its particular brief/s? - surprisingly well
• How did the tool operate in terms of divergent and convergent processes? more convergent - guiding planned process
• Which cognitive styles did the tool draw out? - too broad in scope to answer this question
• How well did the tool engage all participants? - it was a flexible, inclusive tool
• What were the similarities and differences in how the tool worked between the two teams? broadly, both teams reached satisfactory completion of the tasks
  • (This may be difficult because of the level of information for Team 1)
• What were the reasons behind the similarities and differences respectively? don't know
• To what extent was the tool accessible, usable, useful in the context of the work, and applicable externally, i.e. portable? looks very broad and helpful
• To what extent did the tool embody, promote, bring out the values implied by sustainability
  (e.g. environmental and ethical soundness) - it embodies holistic, fractal properties that would support ecological communities
• What did the tool bring to meta-design? - see previous answer

Emergent themes relating to the four-fold integrative framework:

• What are the insights generated through the evaluation of this tool?
• Theme one - Potential Realisation (Ken Fairclough, 2003)
• Theme two - The emergent nature of the role of the metadesigner in team work
• Theme three - The awkward transitions between the tools and across the phases (i.e from cultural props to rhythm tool, from 'me' to 'we' to 'we we')
• Theme four - The convergent and divergent nature of the outcomes belonging to team one and team two

Research questions for the whole event

• How well did the tools work as a sequence?
• What was their relative significance and role in the workshop as a whole?
• What did the tools bring to metadesign?
• And lots more

Emerging themes:

• What other insights did the evaluation of the workshop generate?

Data to be considered

• A) Transcriptions from follow up interviews
• B) Scribes' Notes
• C) Data ports from the days (all drawing and writing that the participants produced in the respective activities)
• D) Notes from us processing the days through the Story-telling tool.
• E) Film footage (only to be used as secondary data, when we identify an ‘itchy’ patch to dig deeper into) Archive of Pines Calyx film footage

Analysis process

Step 1 - All data in one place

• Refer to: -
• follow up interviews Transcriptions
• Scribes' Notes
• Footage in digital format
• Make accessible to everybody analysing

Step 2 - Data presentation

• Pull together the data from the various sources into one document and organise in appropriate categories.
• Four Fold Integrative Framework Data Sources
• This can mean putting together everything that has been said about the particular tool.
  (In the scribes’ notes, in the interview transcripts.) **Make sure you include a reference, ideally person, page number, date.
  Some references will be straightforward whereas other are more subtle.

Step 3 - Data evaluation

• Research questions to answer
  • Overarching question:
  • How successful was the tool in terms of prompting synergy? (See suggested template below from Julia's definitions.)

Particular questions:

First impressions
How did the participants experience the tool?
How did we experience working with the tool?

Process
How well did the tool work in practical terms? (E.g. were the space and resources appropriate, the right amount of time allocated)
How well did the tool meet its particular brief/s? (I.e. what we say that it can achieve on the wiki)
How did the tool operate in terms of divergent and convergent processes?
To what extent was the tool accessible, usable, useful in the context of the work, and applicable externally, portable?

Learning and exchange
Which cognitive styles did the tool draw out?
What was the level of exchange between the participants?

Group dynamics
How well did the tool engage all participants?
What were the similarities and differences in how the tool worked between the two teams?
What were the reasons behind the similarities and differences respectively?
Here it might be useful to cross-reference everybody’s experience of the tools. See tables above.
(By external commentator, I mean somebody else who sat in… Team 2 continues with participant 3 etc.)

Synergy
Levels (vertical) and modes (horizontal) of synergy

Metadesign

• What did the tool bring to meta-design?
• To what extent did the tool embody, promote, bring out the values implied by sustainability (e.g. environmental and ethical soundness)
• We might use this model to process the data and write up the findings.

Emerging themes:

• What other insights did the evaluation of the tool generate?
• Draw out metadata – i.e. overarching themes from the findings as a whole.

Overview of whole event

• Research questions to answer:
  • How well did the workshop manage to achieve synergy and how do we know?
• One interesting point is that when the workshop has subsequently been presented at a design research conference the project has an infectious and optimistic and playful appeal to other that feels as though it comes from the way that the researchers participated in the event and also the way the event was co-designed
  • How well did the tools work as a sequence?
  • And lots more

C.f. Ring of Fire model

Step 4 – Report

• 3-4 pages per tool
• Answers to research questions
• Emerging themes
• Key findings
• Recommendations

return / go to m21 meetings ACTIONS
return / go to m21 meetings AGENDAS
return / go to Other AU Research