RPC Wiki

Please see github_auth48_experiment for context.

Summary:
This was the first time I've had such a long document go through the 
AUTH48 process, but from my perspective things worked much better than 
they would have using email.

There were literally almost 100 separate changes to the text during AUTH48 
(not sure if this is typical or not for this size of document). Being able 
to track each of these individually, set the appropriate status for each 
issue, distribute the work across the authoring team and Jean, bring people 
in to comment on specific issues, and finally review proposed changes with 
full diffs and ability to easily discuss the diffs - each of these was a 
significant efficiency improvement from my perspective. However, of all 
of these, the separate tracking for each issue (rather than a single 
ginormous email) was the biggest improvement IMO.

In the end I felt that we were able to resolve all ~100 issues that were 
raised as part of AUTH48 completely and improve the document's accuracy 
and readability as a result. 

Here's what specifically worked well:
1) I wrote some scripting to parse the initial auth48 mail and turn that 
into ~60 separate github issues. This let each issue easily be discussed 
independently.
2) I triaged the issues, and those that seemed pretty straightforward 
I marked as 'editor-ready' and assigned to Jean, adding any necessary 
comments to the specific issue.
3) Those that weren't straightforward were assigned to members of the 
authoring team to track down the right solution, at which point they 
were then assigned as in 2).
4) When new questions came up, the editors or Jean would file new Github issues.
5) To address the issues assigned to her, Jean would create pull 
requests with her proposed changes, tagging the issues that the pull 
request was addressing.
6) I would review the pull request and either send comments back to 
Jean for further changes (through the pull request review tool) or 
merge the PR into the document.
7) When the PR was merged, the relevant issues were then closed, 
allowing us to easily track our progress via the size of the issues list.
8) In cases where the authoring team couldn't figure out a solution, 
we pulled relevant people (e.g., Adam Roach) into discussions to figure 
out a solution.

Here's what could have worked better:
a) as noted in 1) above, I had to write some Python to parse the email and call 
the Github API to create the issues. It would be better to have a tool that would 
let the RFC Editor directly create these issues, rather than trying to serialize 
to/from email.
b) we worked on a parallel branch to the main document, meaning that the main 
branch was frozen at jsep-24, and then a separate branch 'rfced' had the 
RFC Editor's version of the document, with all proposed changes and editor 
comments. This approach, while it avoided changing the 'master' branch with 
anything that wasn't fully vetted, ended up causing confusion. Working directly 
against master would have been easier.
c) the saga of #843 is a long and complicated story, but I think this issue is 
largely separate from the GitHub experiment. Neither the JSEP nor BUNDLE editors 
fully understood the conflict until it was too late, and a short meeting back 
in 2018 would probably have avoided the issue.

You are welcome to add my feedback to the experiment page - happy to provide 
more details as needed.

Justin

• The RPC needs to control the repository.
• The subject line of an issue needs to be succinct since the GUI truncates after 56 chars
• Each issue must contain just one topic.
• Ensure that the appropriate people are “Watch”ing the repo -- ADs, chairs, etc. (Only the owner of the repo can set people as collaborators. Only individuals can set their Watch settings for a repo.)

My biggest surprise - it is not as useful at communicating issues and changes as I thought it would be. One author was unaware of progress, and the AD never received notifications even when he was @mentioned. I also thought I knew a thing or two about Git and GitHub, but creating a workflow for the rtcweb-wg repo took time.

My sneaking suspicion - even frequent GitHub users are bitten by bad assumptions about how it works.

Things I liked - Being able to refer to issues and PRs with hashtags and have GitHub build hyperlinks in comments and commit messages. Automatic closure of issues when a PR was accepted is satisfying. Tracking of open issues is easy.

Things that were meh - interacting with Git and GitHub is not quick. Like other uses of GitHub I've seen in the IETF, technical discussions erupt on individual issues (see #843, which predated the AUTH48 issues but wasn't addressed by the authors until almost all the AUTH48 issues had been closed), and participants are slow to take the issue to a broader audience/ML (see the start of the email discussion).

Things that I would change next go around - the RPC should control the repo, which would allow us to merge the changes and add the right people as contributors so that they can (maybe) receive notifications (it may be a horse-to-water problem).

A questionnaire was sent to the author, AD, and WG chairs. The following is the list of questions and the responses received:

• What were the advantages of using GitHub during AUTH48?
  • Erik Kline (AD):
    • Being able to see meaningful diffs as text evolved; being able to track issues one by one and review comments on each one separately; being able to see how a pro handle the XML.
    • The legacy model of one giant email with all the proposed changes and 5 different authors and their email agents injecting fonts, colors, HTML indentation, and so on in one long thread was so easily eclipsed by the organisation and simplicity of this approach. I don't ever wanna go back.
  • Jen Linkova (author): It was much easier for me to track all issues. When we were using emails, it was too easy to miss some questions/comments from the RFC Editors. With emails, I was trying to respond to all issues in one email (to make sure that no issues slipped through the cracks. With Github I can easily see what issues have been addressed, and work on the remaining issues one by one.

• What were the disadvantages of using GitHub during AUTH48?
  • Erik Kline (AD): Needing to setup the alerts to route to my IETF email (this will be necessary for each new repository unless I get wise and try to make some separate github account or something).
  • Jen Linkova (author): None I can think of.

• Was communication clear and easy to follow?
  • Erik Kline (AD): Yes
  • Jen Linkova (author): Yes

• Were there any aspects of communication that were challenging?
  • Erik Kline (AD): Just getting repository alerts setup to route to my IETF email (which I see is nicely linked in the README.md).
  • Jen Linkova (author): No, all was clear. All issues I had were caused by me not paying enough attention to the documentation ;)

• How may we improve the README.md?
  • Erik Kline (AD): (no response)
  • Jen Linkova (author): Maybe it would be nice to have a short “cheat-sheet” explaining how to perform basic operations like pull requests (like fork and then pull).

• Were the issues appropriately “sized”?
  • Erik Kline (AD): Yes
  • Jen Linkova (author): Yes

• Were the issue labels helpful (e.g., rfced, question, editor-ready)?
  • Erik Kline (AD): Yes
  • Jen Linkova (author):

• In particular, did you find the process more efficient than the current email-based process?
  • Erik Kline (AD): Yes
  • Jen Linkova (author): Yes

• What parts of the process would you like to see changed in a future experiment?
  • Erik Kline (AD): (no response)
  • Jen Linkova (author): (no response)

• ADs and WG Chairs: Was it easy to track the discussion and resulting updates?
  • Erik Kline (AD): Yes

• Please provide any other comments and suggestions for improvement here.
  • Erik Kline (AD): (no response)
  • Jen Linkova (author): (no response)

• May we quote your replies on a public page documenting the experiment?
  • Erik Kline (AD): Yes
  • Jen Linkova (author): Yes