CollabSense: Can Real Work Become Better Feedback?
Written with Edinburgh Napier University and published open source
CollabSense was written by Ethan Campbell and Ashkan Sami at Edinburgh Napier University, together with Sachin Goyal and Nima Soroush at ClarityLoop.
This research is a collaboration between ClarityLoop and Edinburgh Napier University, and the project materials are available publicly.
Most performance development still depends on delayed memory, forms, self-assessments, and manager interpretation. By the time feedback reaches someone, the work that created it may already be weeks or months old.
ClarityLoop is exploring a different idea: real work already contains signals about how people collaborate, communicate, support others, lead, learn, and grow. If those signals can be recovered responsibly, with evidence and human review, feedback could become more continuous, more specific, and less dependent on memory alone.
CollabSense is an open research study, created with Edinburgh Napier University, testing whether ClarityLoop can identify sentiment, strengths, and growth opportunities from real-world collaboration data. The study does not position AI as an evaluator, a manager substitute, or a decision-maker. It asks a narrower and more useful question: can AI help surface evidence-supported patterns that humans can inspect, challenge, and use for development?
The early answer is encouraging but qualified. Across six experimental configurations using public GitHub conversations from pandas-dev/pandas and kubernetes/kubernetes, the ClarityLoop analysis engine produced stable sentiment estimates with a mean score of 6.63/10, surfaced a large volume of strengths, and produced more plausible growth opportunities once collaboration histories became denser through K-Core filtering and strict peer-to-peer selection.
The strongest finding is not simply that AI can analyse text. It is that context matters. Sparse, one-off interactions were not enough for meaningful coaching signals. Repeated collaboration patterns created better conditions for identifying useful strengths and growth opportunities.[1] [2] [3]
6.63/10
mean sentiment score across more than 3,300 scored interactions
1.1%
baseline Pandas interaction density, too sparse for recurring coaching patterns
35%–75%
density achieved after K-Core filtering and strict peer-to-peer selection in Kubernetes
85%
of sampled growth opportunities judged at least partially actionable in manual review
What belief is ClarityLoop testing?
ClarityLoop's core thesis is that better performance development can be built from the work people are already doing, rather than reconstructed later from memory.
That does not mean turning workplace communication into automated judgment. It means testing whether collaboration data can support better human conversations: more specific feedback, clearer examples of strengths, earlier identification of growth opportunities, and a stronger evidence base for coaching.
In CollabSense, the analysis engine looked for three kinds of signals:
- Sentiment scores that estimate the tone of the interaction.
- Strengths that identify positive behaviours and soft-skill markers.
- Growth opportunities that point to recurring, coachable patterns rather than one-off mistakes.
For ClarityLoop, the important question is not whether a model can label a comment. It is whether repeated work interactions contain enough context for useful, evidence-supported people-development signals.
Why use open-source GitHub collaboration data?
Open-source GitHub data is not the same as enterprise communication. It is public, asynchronous, technical, and often structurally sparse. That is exactly why it is useful for early research.
Public repositories create a transparent test bed. Other researchers can inspect the source material, review the assumptions, and challenge the results. The data also puts the engine under a difficult condition: software collaboration includes direct feedback, terse technical language, many one-off exchanges, and review cultures that generic sentiment tools often misread.[4] [5] [6]
That makes GitHub a useful proxy, not because it perfectly represents a company, but because it lets ClarityLoop test the analysis engine against real collaboration with visible limitations. If useful signals appear in this setting, the finding supports the thesis. If signals fail to appear, the failure helps show what kind of context the system needs.
What CollabSense tested
CollabSense uses a custom pipeline that scrapes, filters, anonymises, and restructures GitHub pull request and issue data so it can be processed by the ClarityLoop engine in the same schema it expects from enterprise workspaces. Known bots were removed, long-term contributors were prioritised, and public identities were replaced with synthetic enterprise-style user profiles.
Two open-source projects were used:
pandas-dev/pandasfor the baseline and prompt-sensitivity experiments.kubernetes/kubernetesfor the density-engineering experiments.
The study reports six experimental conditions:
| Experiment | Comments scored | Reviewers | Recipients | Strengths | Growth opportunities | GO yield |
|---|---|---|---|---|---|---|
Pandas_Prompt_B_Radical | 1226 | 32 | 83 | 41 | 192 | 15.66% |
Pandas_LongTerm | 1162 | 63 | 194 | 120 | 0 | 0.00% |
Pandas_Prompt_A_Individual | 479 | 20 | 69 | 39 | 5 | 1.04% |
K8s_Strict_Large | 328 | 79 | 36 | 52 | 4 | 1.22% |
K8s_Artificial_Dense | 87 | 48 | 17 | 12 | 4 | 4.60% |
Pandas_ShortTerm | 32 | 5 | 15 | 6 | 0 | 0.00% |
Bottom line: the study did not just compare prompts. It compared prompts and the shape of the collaboration graph. That distinction matters because CollabSense's central finding is structural: the engine's biggest limitation on raw open-source data was not basic text comprehension, but the lack of repeated relationships.
What we found
1. The engine read technical collaboration tone without defaulting to negativity
One risk in this kind of work is that an AI system might read direct engineering language as hostile or negative. CollabSense reports the opposite pattern: the score distribution clustered around constructive and positive ranges, with a mean of 6.63/10 and very little traffic in the strongly negative bands. That matters because software engineering text is a known failure case for many generic sentiment tools.[7] [8] [9]
The baseline Pandas_LongTerm run generated 120 strengths from 1,162 scored comments, including repeated themes such as collaborative problem-solving, practical solutioning, and proactive feedback-seeking. This suggests that open-source comments can carry social and behavioural information, not only code-adjacent technical content.
2. Sparse graphs blocked growth opportunities even when the text was understandable
The baseline Pandas graph had only 1.1% density. That is enough for plenty of technical interaction, but not enough for a coaching system that looks for recurring patterns between the same people over time. In that baseline condition, the engine generated zero growth opportunities.
The research then tested prompt relaxation. A more permissive “Medium/Precision” prompt generated 5 growth opportunities, while a much looser “Radical” prompt generated 192. That showed the model could find critique-like language in the data, but it also revealed the tradeoff: when the threshold dropped too far, the system started mistaking routine code-review corrections for developmental coaching signals.[10]
This is one of the clearest findings in the project. It suggests that simply telling the model to “find more coaching” is not a good answer. Quantity can be forced. Quality depends on context.
3. Repeated collaboration patterns created better conditions for coaching signals
To test whether structure was the real bottleneck, the study moved to kubernetes/kubernetes and applied K-Core decomposition to isolate a dense maintainer core. The first pass still failed because it pulled in thousands of one-off interactions with the public edge of the network, diluting the dataset to 0.2% density. Only after a strict peer-to-peer filter was applied did the graph begin to resemble the repeated interaction structure the engine was built for.[11] [12]
Once density moved into the 35% to 75% range, the output changed. The K8s_Artificial_Dense run produced 4 growth opportunities from 87 scored comments, a 4.6% yield, while K8s_Strict_Large produced 4 from 328. This suggests that the missing variable was network structure, not model comprehension.
Why this matters for ClarityLoop
CollabSense supports the ClarityLoop thesis in a specific, bounded way: useful people-development signals appear more likely when analysis is grounded in repeated collaboration, not isolated comments.
That matters because enterprise environments often have the relationship structure that raw public GitHub graphs lack. Teams tend to work together repeatedly. Managers, peers, and collaborators build shared history. Feedback, support, communication, and leadership behaviours often appear across many small moments rather than in a single dramatic event.
For ClarityLoop, the implication is practical:
- The engine appears able to read technical collaboration without defaulting to negative sentiment.
- Positive behavioural signals are easier to recover than growth signals in sparse public graphs.
- High-quality coaching signals depend on repeated relationships and data topology, not only better prompting.
- Dense historical batches can create context saturation even when the data is “good,” which means ingestion strategy matters as much as model choice.
- Enterprise environments may be a better fit than raw open-source repositories because they naturally contain denser, more repeated interaction patterns.[14]
This is early evidence, not product validation. It points toward a design principle: ClarityLoop should focus less on one-off evaluation and more on careful accumulation of evidence over time.
Results and interpretation
Human review supported plausibility, while exposing failure modes
The study includes a preliminary manual assessment of 20 sampled growth opportunities, rated independently by two reviewers across three criteria: actionability, whether the output captured a genuine improvement area, and whether it was supported by the source comment. That matters because a people-development signal is only useful if humans can understand the evidence behind it.
| Criterion | Yes | Partial | No | At least partial |
|---|---|---|---|---|
| Actionable feedback | 13 (65%) | 4 (20%) | 3 (15%) | 17/20 (85%) |
| Genuine improvement area | 6 (30%) | 11 (55%) | 3 (15%) | 17/20 (85%) |
| Supported by evidence | 12 (60%) | 6 (30%) | 2 (10%) | 18/20 (90%) |
Agreement across the 60 judgments was 87%, with an overall Cohen’s kappa of 0.76, which indicates substantial agreement. The most conservatively scored criterion was whether the item represented a genuine developmental issue rather than ordinary review workflow. That is exactly where the “nitpick versus coaching” distinction becomes hardest in practice.
Three examples accepted by both reviewers:
- Avoid premature assignment of approvers: surfaced from Kubernetes comments about assigning approvers before receiving any LGTM.
- Improve code freeze awareness: drawn from release-management comments asking whether work remained in scope close to code slush.
- Include reproducible examples in bug reports: derived from repeated maintainer requests for copy-pastable examples rather than external attachments.
The study is also candid about failure modes. Reviewer notes identified recurring issues such as misreading active debugging as a personal growth signal, attributing feedback to the wrong participant in multi-party threads, and treating superseded work as a communication gap. That candour makes the findings more useful, because it shows where the boundary still is.
Stability and fairness checks were encouraging, not conclusive
The study also reports a small but valuable battery of bias and stability checks:
- Execution stability: matched comments scored across separate runs had a mean sentiment delta of -0.05.
- Target fairness: a Pareto check suggested growth opportunities scaled broadly with interaction volume rather than clustering on a few people.
- Verbosity bias: the reported trendline between comment length and sentiment score was flat, suggesting the model was not simply rewarding longer comments.
These checks do not prove the system is free from bias. They show that the project is measuring important risks instead of assuming them away.[13]
What this does not prove
CollabSense is deliberately limited. It should not be read as proof that ClarityLoop will perform reliably in every workplace, that software should evaluate people on its own, or that open-source behaviour maps cleanly onto enterprise development.
The study does not prove:
- That software should replace managers or human judgment.
- That performance decisions should be made from communication data alone.
- That the engine is free from bias.
- That a GitHub collaboration graph is equivalent to a workplace team.
- That every generated strength or growth opportunity is correct.
- That prompt design alone can solve the coaching-signal problem.
What it does show is narrower: when collaboration data contains repeated relationships, the engine has better conditions for surfacing evidence-supported strengths and growth opportunities that humans can review.
Limits and next steps
The research is careful not to overstate the evidence, and that is the right posture.
- Open-source collaboration is only a proxy for enterprise communication, not a substitute for it.
- Prompt sensitivity materially changed output volume and type, which means evaluation should always report multiple conditions.
- Artificial densification helps recover the signal, but it may not perfectly mirror real workplace hierarchy, cadence, or context.
- A logical next step is chronological chunking: replaying history in time-bounded slices so the model can build patterns incrementally instead of reading years of dense context in a single batch.
Further validation should test the same thesis in settings closer to how ClarityLoop is meant to be used: repeated workplace relationships, human-reviewed evidence, longitudinal data, and clear separation between coaching support and performance judgment.
That final point matters. CollabSense suggests the current long-context bottleneck may be less about whether the model can ever find coaching patterns, and more about whether we are feeding the history in the same way a live product would actually see it.
Citations
[1] Bird, Pattison, D’Souza, Filkov, and Devanbu (2008). Latent Social Structure in Open Source Projects.
[2] Kalliamvakou, Gousios, Blincoe, Singer, German, and Damian (2014). The Promises and Perils of Mining GitHub.
[3] Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, and Liang (2024). Lost in the Middle: How Language Models Use Long Contexts.
[4] Jergensen, Sarma, and Wagstrom (2011). The Onion Patch: Migration in Open Source Ecosystems.
[5] Crowston and Howison (2005). The Social Structure of Free and Open Source Software Development.
[6] Tsay, Dabbish, and Herbsleb (2014). Influence of Social and Technical Factors for Evaluating Contribution in GitHub.
[7] Jongeling, Sarkar, Datta, and Serebrenik (2017). On Negative Results When Using Sentiment Analysis Tools for Software Engineering Research.
[8] Calefato, Lanubile, Maiorano, and Novielli (2018). Sentiment Polarity Detection for Software Development.
[9] Lin, Zampetti, Bavota, Di Penta, Lanza, and Oliveto (2018). Sentiment Analysis for Software Engineering: How Far Can We Go?
[10] Reynolds and McDonell (2021). Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm.
[11] Seidman (1983). Network Structure and Minimum Degree.
[12] Batagelj and Zaveršnik (2011). Fast Algorithms for Determining (Generalized) Core Groups in Social Networks.
[13] Mäntylä, Graziotin, and Kuutila (2018). The Evolution of Sentiment Analysis.
[14] Sadowski, Söderberg, Church, Sipko, and Bacchelli (2018). Modern Code Review: A Case Study at Google.