bug-bisect.rst 7.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165
  1. .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
  2. .. [see the bottom of this file for redistribution information]
  3. ======================
  4. Bisecting a regression
  5. ======================
  6. This document describes how to use a ``git bisect`` to find the source code
  7. change that broke something -- for example when some functionality stopped
  8. working after upgrading from Linux 6.0 to 6.1.
  9. The text focuses on the gist of the process. If you are new to bisecting the
  10. kernel, better follow Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
  11. instead: it depicts everything from start to finish while covering multiple
  12. aspects even kernel developers occasionally forget. This includes detecting
  13. situations early where a bisection would be a waste of time, as nobody would
  14. care about the result -- for example, because the problem happens after the
  15. kernel marked itself as 'tainted', occurs in an abandoned version, was already
  16. fixed, or is caused by a .config change you or your Linux distributor performed.
  17. Finding the change causing a kernel issue using a bisection
  18. ===========================================================
  19. *Note: the following process assumes you prepared everything for a bisection.
  20. This includes having a Git clone with the appropriate sources, installing the
  21. software required to build and install kernels, as well as a .config file stored
  22. in a safe place (the following example assumes '~/prepared_kernel_.config') to
  23. use as pristine base at each bisection step; ideally, you have also worked out
  24. a fully reliable and straight-forward way to reproduce the regression, too.*
  25. * Preparation: start the bisection and tell Git about the points in the history
  26. you consider to be working and broken, which Git calls 'good' and 'bad'::
  27. git bisect start
  28. git bisect good v6.0
  29. git bisect bad v6.1
  30. Instead of Git tags like 'v6.0' and 'v6.1' you can specify commit-ids, too.
  31. 1. Copy your prepared .config into the build directory and adjust it to the
  32. needs of the codebase Git checked out for testing::
  33. cp ~/prepared_kernel_.config .config
  34. make olddefconfig
  35. 2. Now build, install, and boot a kernel. This might fail for unrelated reasons,
  36. for example, when a compile error happens at the current stage of the
  37. bisection a later change resolves. In such cases run ``git bisect skip`` and
  38. go back to step 1.
  39. 3. Check if the functionality that regressed works in the kernel you just built.
  40. If it works, execute::
  41. git bisect good
  42. If it is broken, run::
  43. git bisect bad
  44. Note, getting this wrong just once will send the rest of the bisection
  45. totally off course. To prevent having to start anew later you thus want to
  46. ensure what you tell Git is correct; it is thus often wise to spend a few
  47. minutes more on testing in case your reproducer is unreliable.
  48. After issuing one of these two commands, Git will usually check out another
  49. bisection point and print something like 'Bisecting: 675 revisions left to
  50. test after this (roughly 10 steps)'. In that case go back to step 1.
  51. If Git instead prints something like 'cafecaca0c0dacafecaca0c0dacafecaca0c0da
  52. is the first bad commit', then you have finished the bisection. In that case
  53. move to the next point below. Note, right after displaying that line Git will
  54. show some details about the culprit including its patch description; this can
  55. easily fill your terminal, so you might need to scroll up to see the message
  56. mentioning the culprit's commit-id.
  57. In case you missed Git's output, you can always run ``git bisect log`` to
  58. print the status: it will show how many steps remain or mention the result of
  59. the bisection.
  60. * Recommended complementary task: put the bisection log and the current .config
  61. file aside for the bug report; furthermore tell Git to reset the sources to
  62. the state before the bisection::
  63. git bisect log > ~/bisection-log
  64. cp .config ~/bisection-config-culprit
  65. git bisect reset
  66. * Recommended optional task: try reverting the culprit on top of the latest
  67. codebase and check if that fixes your bug; if that is the case, it validates
  68. the bisection and enables developers to resolve the regression through a
  69. revert.
  70. To try this, update your clone and check out latest mainline. Then tell Git
  71. to revert the change by specifying its commit-id::
  72. git revert --no-edit cafec0cacaca0
  73. Git might reject this, for example when the bisection landed on a merge
  74. commit. In that case, abandon the attempt. Do the same, if Git fails to revert
  75. the culprit on its own because later changes depend on it -- at least unless
  76. you bisected a stable or longterm kernel series, in which case you want to
  77. check out its latest codebase and try a revert there.
  78. If a revert succeeds, build and test another kernel to check if reverting
  79. resolved your regression.
  80. With that the process is complete. Now report the regression as described by
  81. Documentation/admin-guide/reporting-issues.rst.
  82. Bisecting linux-next
  83. --------------------
  84. If you face a problem only happening in linux-next, bisect between the
  85. linux-next branches 'stable' and 'master'. The following commands will start
  86. the process for a linux-next tree you added as a remote called 'next'::
  87. git bisect start
  88. git bisect good next/stable
  89. git bisect bad next/master
  90. The 'stable' branch refers to the state of linux-mainline that the current
  91. linux-next release (found in the 'master' branch) is based on -- the former
  92. thus should be free of any problems that show up in -next, but not in Linus'
  93. tree.
  94. This will bisect across a wide range of changes, some of which you might have
  95. used in earlier linux-next releases without problems. Sadly there is no simple
  96. way to avoid checking them: bisecting from one linux-next release to a later
  97. one (say between 'next-20241020' and 'next-20241021') is impossible, as they
  98. share no common history.
  99. Additional reading material
  100. ---------------------------
  101. * The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
  102. `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
  103. in the Git documentation.
  104. * `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
  105. from kernel developer Nathan Chancellor.
  106. * `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
  107. * `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.
  108. ..
  109. end-of-content
  110. ..
  111. This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
  112. you spot a typo or small mistake, feel free to let him know directly and
  113. he'll fix it. You are free to do the same in a mostly informal way if you
  114. want to contribute changes to the text -- but for copyright reasons please CC
  115. linux-doc@vger.kernel.org and 'sign-off' your contribution as
  116. Documentation/process/submitting-patches.rst explains in the section 'Sign
  117. your work - the Developer's Certificate of Origin'.
  118. ..
  119. This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
  120. of the file. If you want to distribute this text under CC-BY-4.0 only,
  121. please use 'The Linux kernel development community' for author attribution
  122. and link this as source:
  123. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/bug-bisect.rst
  124. ..
  125. Note: Only the content of this RST file as found in the Linux kernel sources
  126. is available under CC-BY-4.0, as versions of this text that were processed
  127. (for example by the kernel's build system) might contain content taken from
  128. files which use a more restrictive license.