Test proposal

Breaking down the hypothesis

Does ArchLens helps developers preserve intended dependency structure during feature implementation?

• Do ArchLens minimize the number of introduced dependency violations during feature implementation?
  • are there dependency violations introduced in the layers visible in the provided views?
  • are there dependency violations introduced in the layers NOT visible in the provided views?
  • do developers make new views?
  • maybe task completion time, and possibly a measure of rework

Survey 1

• wht is your experience

Test setup

Feature implementation

• We make repo with identical code and documentation
• The repo is linked to a project where one or more feature task are described
  • The task is designed so that there are multiple ways to implement it - here amongst ways that would adhere or break the dependency violations over multiple levels/depths
• The repo will include documentation of dependency structure and violation policies and state that the developers should be aware of architectural drift
• dependency policy description is documented with screenshot of archlens graphs

---

• We will do through the ReadMe file for the developers
• .. or the developers first get the task after 20 min

Where ArchLens comes in?

• The only difference is that the repo where the developers use archlens is forked from the first repo - still identical but has a section about how to set up archlens and that it is recommended that the developers use it
• some views are predefined/provided to the devs (those that are included in the documentation)
• The repo will also have setup an action that renders the dependencies on PR and drafts (PR not approved)

Discussion Points (Rules)

• The devs should all use VS code
• Are other dependency violation tools allowed ?
  • no
• Does documentation most often lie in the repo or outside the repo?
• Should we combat CoPilot? (read up on copilot)
  • Option A: no
  • Option B: yes - we provide the documentation in the repo wiki
• What tools can the developers use ? : ALL TOOLS
  • Option A: All tools are allowed, (but we ask for a screen recording/prompts used of the session to document and compare use of LLMs and google searches)
    • pros: more realistic
    • cons: more data to go through and factor into the analysis
  • Option B: LLMs are not allowed, but google searches are
    • pros: more contained - we can isolate the use of LLMs in the analysis
    • cons: not as realistic
  • Option C: Allow all and discuss it in the limitations section
    • pros: more realistic and not as much data to factor in
    • cons: harder to draw conclusions on

Focus group

After the feature implementation a group of 5-6 developers that either all have or have not used ArchLens respectively sit down together for a focus group. Timeboxed 20 minutes.

The group that didn't use ArchLens are introduced to it.

Then they are prompted ~2 open ended questions to discuss about:

• how ArchLens assisisted them?

• What was ArchLens' limitations?

• how could a tool like ArchLens have assisisted them?

• What could a tool like ArchLens give of complications?

The focus groups will be recorded.

Survey 2

• would you recommend it to a friend