Meeting Notes
- Lots of future work at the end
- Functional vs non-functional requirements
- Early career
- AI
- Not statistically significant, but that's okay
- Combining what people say vs what they do
- Quant: Lots of data, what is most important is still a work in progress
- Task duration, no easy solution
- Maybe as soon as next task is started, previous task gets no more time added
- Discussion point: Write down issues, how to do it better next time
- Heatmap: Grey should be white
- P-value fishing? Correlation map not good, should look for specific hypothesis
- Task duration, no easy solution
- Qual:
- Thematic analysis: Quantifying the qual data somewhat (isn't that content analysis? I will check)
- Ideally: Peer review, so we do each our themes and compare them. Maybe not possible given time constraints + Babette isn't as much into the interviews
- Maslows pyramid: If you don't have basic needs (understanding the language and codebase), then no energy to focus on higher level (architecture)
- Test status
- Two left
- Setup issues: Document it in issues
- Should the people with issues still be included in the data?
- Own category, or include with footnote
- Main result should only answer research question, not too much discussion if it is not directly relevant to RQ
- Thematic analysis, read through, add codes for relevant topics, compress information
- Report: Dream is to write a real paper with Git truck for Vissoft conference
- Deadline in June
- 10 pages should be enough, no requirements for thesis, a bit more with figures is okay
- Participants not using ArchLens: Should we force them? Use first run as pilot?
- Okay to add ArchLens demo
- Self-determining skills and proficiency, too unreliable?
- Not going to change surveys
- Using years of experience instead
- Rules or not?
- There are many places with strict rules, it is not used much. ArchLens is different
- ArchLens as an onboarding tool
- Important quotes we translate, the rest is just google translate
- Vissoft conference: Paper maybe?
- Italy in September
- Continue from yesterday, test setup
- Checking compliance for control group, tone down for treatment
- Pilot study: AI
- Why does it not work for him? Debugging
- "name" in config: "RelevantPrefixes" or similar
- Exam date
- Week 25
- Next meeting: 10/4 9:00
- Test setup, repos README, how-to-contribute, tasks
- Trunk branches: Emails too long, maybe make user name instead (wont be anonymous so can be related to name)
- As many as possible, as high quality as possible
- Stricter with completion time and breaks
- No meeting next week, Easter
- Week after: Issues
- Run through the last tasks
- Add tasks as issues for participants
- Finish test setup
- Remove breaking unit tests
- Run pilot (Tuesday night?)
- Focus group questions
- Introduction - what do we say other than "read the README and how-to-contribute files"
- Recording, how and with what?
- Issues
- Why does it not work for him?
- SonarQube
- Exam date
- Survey fixes + focus group questions
- Draft of timeslots + messages
- Program clean-up, making ready for test repos and merging in
- Guide for how to deliver tasks
- Experiment tasks
- Test repos
- Finalize timeslots
- Test rabbit Simon this weekend
- How do they test without using ArchLens?
- PyPi publish
- Test repo:
- ArchLens too "easy" to follow architecture
- Try to find tasks for ArchLens, maybe there is something there
- We have Thickets (made with Claude), but lets see if we can avoid using it
- RQ: Sound good! Yay!
- Maybe split RQ2 in 2; task completion time & confidence
- Does confidence fit better in RQ3?
- If snack budget too large, let him know, can maybe reimburse
- Surveys:
- Years of experience, be specific
- Worked on a project with policy: Be specific, separate dependency policy and architecture / "where architecture was an important part of the development" / "have you been on a project with architectural diagrams"
- Confidence: Explicit, match with intro to experiments
- Control group, introducing archlens, avoid too biased. Trust what people do, not say. Maybe brainstorm/discuss in the group without talking about the tool. Painpoints, do they encounter these problems elsewhere
- Treatment: Did you define new views: Is it relevant? Is it realistic?
- Focus groups:
- Open group discussion
- Experiment: Recording screen?
- Takes a lot of space, Mircea can maybe help find
- Maybe compress/analyze locally before they share it
- Do we have time to analyze it? Do not NEED to analyze it all, good as back-up
- Maybe we walk around and take notes during experiment instead
- PyPi: Publish as soon as merged into main repo. Very fast, no worry
- Java project: Look at pom, just use those modules in one view
- Next week also online
- [RQ1:] (Primary) To what extent does access to interactive, tool-generated dependency visualisation reduce the number of dependency-policy violations introduced during feature implementation, compared with static dependency policy documentation alone?
- [RQ2:] (Secondary) How does access to such tooling affect task completion time and developer confidence in dependency policy conformance?
- [RQ3:] (Secondary) How do developers perceive and use dependency visualisation during feature implementation, and what barriers to adoption emerge?
- Not too big
- xUnit, Moq etc.
- Use Google BigQuery to search for repos with right size and language
- Single variable: Static diagram vs live diagram
- Especially since it is now faster
- Add qualitative analysis after quantitative
- Search for open source system to use and extend upon
- Too much time to implement ourselves
- Tasks need to be easy to fail
- Maybe add time pressure so the control group "forget" the architecture
- No diff, just IDE
- Add notification that a view has a change (extension)
- Mircea has access to PyPI, we need to test setup before lab
- Extra meeting Monday 15:00, focus on just the code
- We stick to feature dev
- Single variable: With/without ArchLens
- Can use Copilot? Check what it can do
- Can use LLM's? Yes
- Time is a factor here
- Dont mention
- Everyone uses VS Code
- Three weeks for testing, easter NOT included
- Three time slots to choose from per day
- Each time slot is 3 hours:
- 30 min welcome and intro
- 2 h test (perform tasks)
- 30 min focus group interview & quick end survey
- We brings snacks & drinks
- Morning: Buns/croissants & coffee
- Afternoon: Snacks/cake & soda/beer
- Evening: Light dinner (pasta salad, bread etc.) & soda/beer
- Deadline for signups: 16/3, we send out confirmations 17/3
- Signup sheet: link
- Feature implementation (combined with onboarding, see this article)
- PR review
- Unnatural to review PRs in a system you do not know?
- Defining the views vs getting the views: Understanding of the system
- Extra time for the view-definers?
- Using ArchLens vs static documentation when onboarding new devs
- PR's: Ask Mircea on diff render and dependency graph structure
- Response from Mircea:
- Focus on what is our research question, how do we design the test to support that?
- Number of participants and specific metrics can come later
- (Current: To what extent does ArchLens improve developers’ architectural awareness of a system?)
- Does the use of ArchLens by developers reduce the number of dependency violations introduced by feature implementations? (Development)
- Does the use of ArchLens by developers improve the developers ability to detect and correct dependency violations? (Code review)
- ArchLens vs static dependency documentation
- Research question
- Beware it is not too obvious the result, requiring many iterations to ensure a good question/hypothesis and good way to test
- What's the one variable we change
- Code review might be interesting
- Challenge: Introducing the system, code review on a new system is not easy
- Validity question: PR review is not done by new developers
- View definition is also a part of it, could be evaluation in itself
- Can AI help view definition
- "Collogical?" validity - look at actual older PR's, open source history
- Related work: Are developers aware of their architectural changes
- "If its meaningful there will be related work"
- PR's and backlog
- Until next time: Use feedback on RQ to shape test and metrics, iterate
- Mail to Mircea
- Diff render:
- GitUrl vs current config
- Exception when snapshot is not there, better way to handle? Is it okay to expect users to commit snapshots?
- Is same use case okay?
- Git snapshot manager save graph async
- We need to go with our gut, which is to do controlled lab, A/B testing
- We need to research:
- Good controlled labs and A/B tests
- How to setup good system to use in testing
- Which architectural violations are most common (to add to test)
- Initial survey with background info
- Final group session with feedback and experiences
Violation prevention during feature implementation: ArchLens versus baseline tooling
- We define an intended dependency policy (layering rules, forbidden edges, perhaps a small set of domain boundaries) and instrument a feature task that - can be implemented correctly in more than one way, but where some ways introduce violations.
- Participants implement the feature under two conditions: with and without ArchLens.
- The primary outcomes are the number and severity of violations introduced, plus secondary metrics such as change size and unnecessary coupling.
- End with group interviews, how was it? Any features you think are missing?
- First round: Group discussion, 5 devs with architectural knowledge
- Second round: Lab test of use case most mentioned in first round
- Need to make timeline
- Backup of second round: More group discussions
- Alternative: Parallel tests with private and open source, still have focus groups (longitudinal vs time-box)
- Think deeply about what the problem to solve is
- Testing
- Still want to talk to companies, we want to collect different professionals
- Defining views is a big part of using ArchLens
- Ask open source projects: "Hey, here is ArchLens"
- Advantage: No security issues with open source
- Longitudinal test with Diff view: Add link for feedback
- Timeline
- Until next time: We make list of who to ask, Mircea makes list, we see if we can find enough to make it work
- Idea: Adding modules that are not included in any views
- Glob-based pattern matching
- Documentation & user guides
- Communication
- Is there something better than email? How fast can we expect answers?
- Next meeting(s)
- Wednesdays?
- Project - agree on what we are doing, present our idea ala pitch
- Overall time schedule (see image)
- Controlled user studies do not bring much value IRL, but are easier
- Companies will give much value, 3 ish teams, introduce and let them use it some weeks
- Testing usability, can people understand to define views etc.
- We have time, use it to code ArchLens well, leave it in good shape
- Creating backlog & iterations (backlog planning??)
- Mircea sick, no meeting
- Research on program comprehension, onboarding, multi-views etc.
- Finish preliminary problem statement, send to Mircea
- Evaluation of RP
- Thesis content ideas
- Problem statement & RQ's
- Our work process
- User testing ArchLens
- Both onboarding (green) and during dev (red)
- Same users (devs) in both tests
- We create one or two test projects
- On the side: Small code updates, but no larger features
- Scrum-like
- Sprints/milestones with subgoals (timeline is crucial!)
- Retro with reflection and evaluation
- Continuously updating and syncing our work
