Several people have expressed interest in how I go about creating the topic branches in the -rcu tree. So when I created branches earlier this week, I actually kept track.
But why bother with topic branches? In my case, reason is to allow anyone reviewing RCU patches to focus on a particular topic and to be able to keep that topic “in cache” while reviewing all of that topic’s patches. Even more important, it makes it easy for reviewers to completely ignore patches that are not of interest to them.
Preparation
If you intend to follow along by actually executing the commands (highly recommended), first run “git checkout dev.2021.11.30b” in your clone of the -rcu tree.
Before creating branches, you need some essential tools. First, you absolutely must be able to see what you are doing. For this, there is the gitk tool, or, for those forswearing GUIs, the tig tool. If you cannot see what you are doing, your picture of your repository will diverge from that of git, and git always wins. If you can tolerate GUIs at all, install gitk. It will change your life.
Once you have gitk or tig installed, use it to review the commits that you are looking to organize into branches. This allows checking for bugs and also helps identify potential topics. Pro tip: Expand the tool’s window to the full height of the screen. Another pro tip: Restrict gitk to the commits of interest, in my case by running “gitk v5.16-rc1..&”.
From here on out, I will just say “gitk”. If you are smart enough to use a non-GUI tool like tig, you are smart enough to translate. ;–)
Sorting Commits Into Topic Branches
The next step is to run this script, placing the output in a file:
#!/bin/bash
git log --pretty=format:"%h %s" $1 | \
awk '{ print NR, "pick " $0 }' | \
sort -k1n
I have this script in my bin directory under the name git-rebase-prep.sh, so I simply type:
git-rebase-prep.sh v5.16-rc1..dev.2021.11.30b > /tmp/rebase.txt
The first five lines of this file are as follows:
126 pick 12637ec9a1505 tools/memory-model: Document locking corner cases 125 pick 5665cde49ec08 tools/memory-model: Make judgelitmus.sh note timeouts 124 pick 2e8007e79af5b tools/memory-model: Make cmplitmushist.sh note timeouts 123 pick 7a31810649915 tools/memory-model: Make judgelitmus.sh identify bad macros 122 pick 4e11469f67f2e tools/memory-model: Make judgelitmus.sh detect hard deadlocks
The first column is the commit number, with lower numbers meaning newer commits (that is, numbered in “git log” order), but the commits are ordered with the older commits first (that is, in the order that “git rebase” expects them). The second column is the commit’s SHA-1 ID, and the remaining columns are the commit-log subject line.
Edit this file in your choice of editor, but in a window that is the full height of your monitor. Adjust your editor so that you have access two two different regions of this file, for example, in vim, type control-W followed by the letter “s”. The upper pane will eventually have topic-branch names each followed by the commits in that topic branch.
This time was a bit of a special case because this past merge window’s change to the CONFIG_PREEMPT_DYNAMIC Kconfig option broke all non-preemptible CONFIG_SMP=n torture-test scenarios. (The default choice of CONFIG_PREEMPT_DYNAMIC=y forces CONFIG_PREEMPTION=y, which breaks tests of Tiny RCU and Tiny SRCU.) Therefore, the first “topic branch” is a single commit that adds CONFIG_PREEMPT_DYNAMIC=n to all affected torture-test scenarios. All RCU-related topic branches are then placed on top of this commit, thus allowing each topic branch to be torture-tested separately.
This means that the first few lines of the “/tmp/rebase.txt” file are as follows:
Underneath: 26 pick 37cf3c8c1ebda rcutorture: Add CONFIG_PREEMPT_DYNAMIC=n to tiny scenarios @@@
And this line has been removed from the remainder of this file. The “@@@” is a marker separating the topic-branched commits from those still waiting to be classified.
From my scan of the commits, I tentatively created the following topic branches, so that the first few lines of the “/tmp/rebase.txt” file are now as follows:
Underneath: 26 pick 37cf3c8c1ebda rcutorture: Add CONFIG_PREEMPT_DYNAMIC=n to tiny scenarios clocksource.2021.11.30c doc.2021.11.30c exp.2021.11.30c fastnohz.2021.11.30c fixes.2021.11.30c nocb.2021.11.30c nolibc.2021.11.30c rcutorture.2021.11.30c srcu.2021.11.30c tasks.2021.11.30c torture.2021.11.30c torturescript.2021.11.30c Merge above. kcsan.2021.11.30c (merge) lkmm.2021.11.30c (merge, then merge lkmm-dev) clocksource.2021.11.30c (merge) @@@
In other words, we will cherry-pick the first commit, rebase 11 branches on top of it (clocksource.2021.11.30c through torturescript.2021.11.30c, that is, the commits subject to some form of torture testing), and then merge the last ten of these branches together. The kcsan commits will be rebased on top of v5.16-rc1, as will the lkmm commits (but with the lkmm-dev commits rebased on top of the lkmm commits). Each of the kcsan, lkmm, and lkmm-dev branches will be merged separately on top of the ten-way merge point, and finally the clocksource.2021.11.30c branch will be merged on top of all of that. There are always a few commits not ready for the next merge window or that are to be upstreamed elsewhere, and these will be rebased on top of the clocksource.2021.11.30c merge point.
Please feel free to run “gitk v5.16-rc1..dev.2021.11.30c” in order to get a firm picture of the desired state.
The next step is to go through the remaining commits and assign them to branches. This process brought home the fact that there are no kcsan commits this time around (but there were plenty last time and likely will be plenty more next time), so I removed the “kcsan.2021.11.30c (merge)” line from /tmp/rebase.txt. In addition, there were not all that many commits combined for the rcutorture.2021.11.30c and torture.2021.11.30c topic branches, so I merged them all into torture.2021.11.30c. This is why the numbers in the first column of each commit line in /tmp/rebase.txt are important: In some cases, it is necessary to rearrange the topic branches, and it is sometimes important to get the commits in the correct order.
An this point, the /tmp/rebase.txt file looks (almost) like this.
Time to start actually creating the topic branches!
Create Topic Branches
We need two gitk instances, one for the old state (“gitk v5.16-rc1..dev.2021.11.30b”) and another for the new state which we will launch shortly:
git checkout v5.16-rc1 git cherry-pick 37cf3c8c1ebda gitk v5.16-rc1..&
The first of the above commands put us at the desired v5.16-rc1 base commit for the topic branches, which just so happens to be where dev.2021.11.30b is based. Important safety tip: Don’t try creating branches while also moving the pile of commits to some other base commit at the same time. There are just too many things that can go wrong when you try that.
The second of the above commands rebases (or “cherry-picks”) the aforementioned CONFIG_PREEMPT_DYNAMIC-compensation commit. The final command launches the other gitk instance that will show us the evolving state of the topic branches.
Note that my rebased CONFIG_PREEMPT_DYNAMIC-compensation commit happened to have the SHA-1 commit ID 8c0abfd6d2f6b0221194241ac2908751a2a0385f. If you are following along, you will need to change the git rebase argument for --onto to the corresponding SHA-1 commit ID in your tree. If you instead use my ID, everything will work, except that you will be rebasing on my rebased CONFIG_PREEMPT_DYNAMIC-compensation commit instead of your own. Rebasing some on mine and some on yours will probably actually work, but I leave the decision as to whether to carry out that experiment to your better judgment.
I suggest placing the old-state gitk at the left of your screen, the rebase.txt and your command window in the middle, and the new-state gitk at the right of your screen. Being able to see everything all the time is key to successful creation of topic branches.
The next set of commands rebases the clocksource-related commits on top of the rebased CONFIG_PREEMPT_DYNAMIC-compensation commit:
git branch clocksource.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a clocksource.2021.11.30c
The -i argument to the “git rebase” command specifies interactive mode, which will place you in an editor with a very long series of commits. In vim, type “c}” to delete all of those commits and enter vim insertion mode. Then copy and paste the two lines following clocksource.2021.11.30c from the rebase.txt file. Hit the “escape” key to exit vim insertion mode and then strip the commit numbers from all of the lines, for example, by using the “:1,.s/^[0-9]* //” vim command. Write out the file and exit the editor (in vim, your choice of ZZ, 😡, :wq, and probably others as well), at which point git will commence rebasing.
Don’t forget to hit the <F5> key in the new gitk window after the completion of each rebase command.
The next several branches are constructed in a similar manner:
git branch doc.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a doc.2021.11.30c
git branch exp.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a exp.2021.11.30c
git branch fastnohz.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a fastnohz.2021.11.30c
git branch fixes.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a fixes.2021.11.30c
git branch nocb.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a nocb.2021.11.30c
git branch nolibc.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a nolibc.2021.11.30c
For each “git rebase” command, copy and paste the corresponding section of the rebase.txt file. Don’t forget to hit <F5> in the new-state gitk window after running each “git rebase” command.
Remember that “almost” I mentioned above about the rebase.txt file? We now get to one of the deviations. At this point, I belatedly realized that there was only one SRCU commit (441a467cd9793 “srcu: Prevent redundant __srcu_read_unlock() wakeup”) and that it would be better to add that single commit to the fixes.2021.11.30c topic branch:
git checkout fixes.2021.11.30c git cherry-pick 441a467cd9793
Note that I ignored commit ordering in this case. It is safe to do so because this is the only commit in the whole pile that touches kernel/rcu/srcutiny.c, it touches no other file, and there are no other types of dependencies against the other commits. Some times you get lucky!
Next, the rest of the RCU-related branches:
git branch tasks.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a tasks.2021.11.30c
git branch torture.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a torture.2021.11.30c
git branch torturescript.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto 8c0abfd6d2f6b0221194241ac2908751a2a0385f \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a torturescript.2021.11.30c
Once again, copy and paste the commits from the corresponding section of the rebase.txt file and once again do not forget to hit <F5> after each “git rebase” command.
It is now time to do a merge! Unfortunately, I messed up the copy-and-paste. Twice. And then forgot that “git reset --hard” takes the current branch with it. Again, more than once. Then I accidentally included clocksource.2021.11.30c in the merge.
Perhaps building branches late in the day was a mistake, but here are the mistaken commands:
git merge clocksource.2021.11.30c doc.2021.11.30c exp.2021.11.30c fastnohz.2021.11.30c
git reset --hard 8c0abfd6d2f6b0221194241ac2908751a2a0385f
git merge clocksource.2021.11.30c doc.2021.11.30c exp.2021.11.30c fastnohz.2021.11.30c
git reset --hard 8c0abfd6d2f6b0221194241ac2908751a2a0385f
git merge clocksource.2021.11.30c doc.2021.11.30c exp.2021.11.30c fastnohz.2021.11.30c \
fixes.2021.11.30c nocb.2021.11.30c nolibc.2021.11.30c tasks.2021.11.30c \
torture.2021.11.30c torturescript.2021.11.30c
git reset --hard 8c0abfd6d2f6b0221194241ac2908751a2a0385f
git checkout -B torturescript.2021.11.30c 90b21bcfb2846625c4f2e4a0bf5969543cef8ba7
git merge clocksource.2021.11.30c doc.2021.11.30c exp.2021.11.30c fastnohz.2021.11.30c \
fixes.2021.11.30c nocb.2021.11.30c nolibc.2021.11.30c tasks.2021.11.30c \
torture.2021.11.30c torturescript.2021.11.30c
git reset --hard 8c0abfd6d2f6b0221194241ac2908751a2a0385f
git checkout -B torturescript.2021.11.30c 90b21bcfb2846625c4f2e4a0bf5969543cef8ba7
git checkout 8c0abfd6d2f6b0221194241ac2908751a2a0385f
git merge clocksource.2021.11.30c doc.2021.11.30c exp.2021.11.30c fastnohz.2021.11.30c \
fixes.2021.11.30c nocb.2021.11.30c nolibc.2021.11.30c tasks.2021.11.30c \
torture.2021.11.30c torturescript.2021.11.30c
git reset --hard 8c0abfd6d2f6b0221194241ac2908751a2a0385f
If nothing else, this sorry story shows that you can recover from mistakes. And making sure to hit <F5> in the new-state gitk means that all the commits are right there, enabling you to piece things back together.
But I suggest that you instead execute the following single command:
git checkout 8c0abfd6d2f6b0221194241ac2908751a2a0385f
Then hit <F5> in the new-state gitk window and verify that each topic branch has its branch name in place, that none of those branch names are in bold face, and that none of the little per-commit circles are yellow, except for the v5.16-rc1 commit. If all of that is in place, it is time to do the merge:
git merge doc.2021.11.30c exp.2021.11.30c fastnohz.2021.11.30c fixes.2021.11.30c \
nocb.2021.11.30c nolibc.2021.11.30c tasks.2021.11.30c torture.2021.11.30c \
torturescript.2021.11.30c
This will put you into your editor, where you can fill out a commit log for the merge. This is what I did, documenting the topics:
Merge branches 'doc.2021.11.30c', 'exp.2021.11.30c', 'fastnohz.2021.11.30c', 'fixes.2021.11.30c', 'nocb.2021.11.30c', 'nolibc.2021.11.30c', 'tasks.2021.11.30c', 'torture.2021.11.30c' and 'torturescript.2021.11.30c' into HEAD
doc.2021.11.30c: Documentation updates.
exp.2021.11.30c: Expedited-grace-period fixes.
fastnohz.2021.11.30c: Remove CONFIG_RCU_FAST_NO_HZ.
fixes.2021.11.30c: Miscellaneous fixes.
nocb.2021.11.30c: No-CB CPU updates.
nolibc.2021.11.30c: Tiny in-kernel library updates.
tasks.2021.11.30c: RCU-tasks updates, including update-side scalability.
torture.2021.11.30c: Torture-test in-kernel module updates.
torturescript.2021.11.30c: Torture-test scripting updates.
The text starting with “Merge branches” that extends up to but not including the blank line is a single long line, just so you know.
Yet again, be sure to hit <F5> in the new-state gitk to see the results of the merge. Copy the SHA-1 commit ID of this merge point somewhere, keeping in mind that clicking on a commit in gitk places that commit’s SHA-1 ID in your clipboard. Alternatively, you can use bash variables: “mergepoint=`git rev-parse HEAD`”
The next task is to rebase the various Linux-kernel memory model (LKMM) commits, as always substituting the commits from the corresponding portions of the rebase.txt file:
git branch lkmm.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593 git rebase -i --onto v5.16-rc1 d069e38e66dbfeca541d7a2f9790ef8e8d3c911a lkmm.2021.11.30c git branch lkmm-dev.2021.11.30c d069e38e66dbfeca541d7a2f9790ef8e8d3c911a git rebase --onto c438b7d860b4c1acb4ebff6d8d946d593ca5fe1e v5.16-rc1 lkmm-dev.2021.11.30c
Don’t forget <F5>!
The next step is to check out the previous merge point (you will need to substitute the SHA-1 ID recorded above) and do the remaining merges:
git checkout 32e5555b62e6a5f7304b13470cd1881a49a38d18 git merge lkmm.2021.11.30c git merge lkmm-dev.201.11.30c git merge lkmm-dev.2021.11.30c git merge clocksource.2021.11.30c
Finally, rebase the five remaining commits from the “On top:” section of the rebase.txt file:
git branch dev.2021.11.30c a1f5859d3cf6dee7647d39ec926774f05c8d5593
git rebase -i --onto ce410b77746063e1c03f6e5bf85d68cca3e1c7ea \
d069e38e66dbfeca541d7a2f9790ef8e8d3c911a dev.2021.11.30c
Alternatively, you might instead choose to check out the new branch and then cherry-pick the five commits as follows:
git checkout -b dev.2021.11.30c git cherry-pick 2a1d7ed8553da ca507a89ffbb9 0ebafffe561ac e824a258c37d4 74f780ac279b0
This checkout-and-cherry-pick operation is completely equivalent to the prior branch-and-rebase operation.
Either way, as always, don’t forget to hit <F5> in your new-state gitk.
Check Your Work!
And finally, the most important thing is to check your work:
git diff dev.2021.11.30b
If these diffs are non-empty, there was some mistake somewhere in the process.
Possible Complications
This topic-branch exercise went quite smoothly and took about an hour to complete. Sometimes life is harder:
- There might be rebase and/or merge conflicts. I normally take this as a hint that my assignment of commits to topic branches was suboptimal. The commit numbers assigned by the git-rebase-prep.sh script are quite helpful when rejuggling commits. Yes, sometimes the resulting conflicts must be resolved manually, but that is a blog post for another day.
- One of the topic branches might depend on another. In this case, I rebase the dependent topic branch on top of the topic branch that it depends on. Looking at it another way, sometimes the topic branches are arranged in series instead of in parallel.
- A commit might depend on a number of topics, and the rest of the topics might depend on that commit. This is rare, but it does happen. The trick in this case is to create and merge the topic branches that the commit depends on, cherry-pick that commit on top of the resulting merge point, and then create and merge the remaining topic branches on top of that commit.
If you made it all the way down here, thank you for your time and attention, and I hope that this material was helpful. If you remember nothing else from this blog post, let it be gitk (or tig, as the case may be). Being able to see what you are doing allows you to learn to git much faster and allows you to safely take on much more complicated git-related tasks.
paulmckrcu 