From d236e239ceada4a8d7e14074a618c44f7640eab4 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Tue, 29 Jan 2019 11:25:31 +0100 Subject: [PATCH 01/30] Newcomers tutorial rewrite draft --- newcomers-tutorial.md | 168 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 newcomers-tutorial.md diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md new file mode 100644 index 0000000..74b858d --- /dev/null +++ b/newcomers-tutorial.md @@ -0,0 +1,168 @@ +# The Newcomer's Guide To GHC Development + +This page is intended to serve as the first stop for those people who say, "I +want to contribute to GHC, but I don't know quite where to begin." Begin here. + +If you have any questions along the way don't hesitate to reach out to the +community. There are people on the [mailing lists and +IRC](mailing-lists-and-irc) who will gladly help you (although you may need to +be patient). Don't forget that all GHC developers are still learning; your +question is never too silly to ask. + + + +## Installing Prerequisites + +For the sake of keeping things simple, we will assume a Debian-based GNU/Linux +system. GHC builds on a plethora of platforms; check [Setting up your system +for building GHC](building/preparation) for detailed instructions for your +platform of choice. + +### Essential Dependencies + +The following list of packages are essential for GHC development: + +```sh +$ sudo apt-get install git autoconf automake libtool make gcc g++ \ + libgmp-dev ncurses-dev libtinfo-dev python3 xz-utils +``` + +### GHC + +To build GHC, you need GHC. Unfortunately, the version that ships with current +Debian systems is too old to build a current GHC; we recommend installing the +latest stable release from the [GHC downloads page](https://www.haskell.org/ghc/download_ghc_8_6_3.html). + +So: + +```sh +$ wget https://downloads.haskell.org/~ghc/8.6.3/ghc-8.6.3-x86_64-deb9-linux.tar.xz +$ tar xf ghc-8.6.3-x86_64-deb9-linux.tar.xz +$ cd ghc-8.6.3 +$ ./configure +$ sudo make install +$ cd .. +$ ghc --version +The Glorious Glasgow Haskell Compilation System, version 8.6.3 +``` + +For non-Debian platforms, refer to [the GHC download page](https://www.haskell.org/ghc/download_ghc_8_6_3.html) +for an appropriate install package. + +### Cabal + +You'll need an up-to-date version of Cabal. If you have one installed already, +you can **use cabal to install cabal**: + +```sh +$ cabal install cabal-install +``` + +Otherwise, you will have to **install Cabal manually** from Hackage: + +```sh +$ wget https://hackage.haskell.org/package/cabal-install-2.4.1.0/cabal-install-2.4.1.0.tar.gz +$ tar xf cabal-install-2.4.1.0.tar.gz +$ cd cabal-install-2.4.1.0 +$ ./bootstrap.sh +$ echo 'export PATH=$HOME/.cabal/bin:$PATH' >> ~/.bashrc +$ cabal --version +cabal-install version 2.4.1.0 +compiled using version 2.4.1.0 of the Cabal library +``` + +Refer to [the Cabal page on Hackage](https://hackage.haskell.org/package/cabal-install) +for the latest version of Cabal and adjust appropriately. + +### Alex and Happy + +These two fellows are needed for generating lexers and parsers. The versions +available from Debian are too old, but they can easily be installed with Cabal: + +```sh +$ cabal install alex happy +``` + +### Dependencies for building documentation + +```sh +$ sudo apt-get install python-sphinx texlive-xetex texlive-fonts-recommended fonts-lmodern texlive-latex-recommended texlive-latex-extra +``` + +### Dependencies for various development tasks + +```sh +$ sudo apt-get install linux-tools-generic xutils-dev +``` + + +## Getting The Code + +```sh +# needed only once, URL rewrite rule is persisted in ${HOME}/.gitconfig +$ git config --global url."git://github.com/ghc/packages-".insteadOf git://github.com/ghc/packages/ + +# clone GHC's main Git repository (creates './ghc' folder in CWD) +$ git clone --recursive git://github.com/ghc/ghc +``` + +## Making your first build + +```sh +$ cd ghc/ + +# configure build +$ cp mk/build.mk.sample mk/build.mk + +## edit mk/build.mk to remove the comment marker # on the line "BuildFlavour = devel2" + +$ ./boot +$ ./configure + +# NOTE: On Windows you need to download some binary distributables before being +# able to build. This only has to be done once and can be done by adding a flag +# to the call to configure: +$ ./configure --enable-tarballs-autodownload + +# Build GHC +# The -j flag says how many parallel jobs to use. Dependin on your system, the +# best value tends to be between N/2 and N, where N is the number of cores on +# your machine. Pick lower values if you're low on RAM, or have a slow disk. +$ make -j8 +``` + +Now go make yourself some coffee while the build runs. + +## Running your freshly-built GHC + +```sh +./inplace/bin/ghc-stage2 +``` + +GHCi is launched with: + +```sh +./inplace/bin/ghc-stage2 --interactive +``` + +## Rebuilding + +While working on GHC, you will need to rebuild often; however, most of the +time, it is possible to avoid a full, slow build. Here's a few things you can +do to speed things up: + + +1. Select `BuildFlavour = devel2` in your `mk/build.mk`. + (Further reading: [Make GHC build more quickly](building/using#-howtomake-gh-cbuildquickly)) +2. In most cases, you will not make changes in more than one subdirectory at + once. So instead of saying `make` at the top level, you can `cd` into the + directory where you made your changes first, that is, `compiler`, `utils`, + `ghc`, or `libraries`. + (Further reading: [Building a single sub-component](building/using#-buildingasinglesub-component)) +3. Set `stage=2` in your `mk/build.mk` file. This will keep the Stage 1 + compiler "frozen", re-running only Stage 2 compilation. For most development + work, this is perfectly fine. + (Further reading: [Freezing the stage 1 compiler](building/using#-freezingstage1), + [Stages](building/architecture/idiom/stages)) +4. Use `make fast`. This will skip rebuilding dependencies. + (Further reading: [Skip dependency building](building/using#-skipdependencybuilding)) From 20e2f2801df6a1ba2d3e3324e3615d7e28783be8 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Tue, 29 Jan 2019 11:58:39 +0100 Subject: [PATCH 02/30] Newcomers cleanup --- newcomers-tutorial.md | 115 +++++++++++++++++++++++++++++------------- 1 file changed, 81 insertions(+), 34 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 74b858d..67ed38d 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -23,7 +23,7 @@ platform of choice. The following list of packages are essential for GHC development: ```sh -$ sudo apt-get install git autoconf automake libtool make gcc g++ \ +sudo apt-get install git autoconf automake libtool make gcc g++ \ libgmp-dev ncurses-dev libtinfo-dev python3 xz-utils ``` @@ -31,18 +31,24 @@ $ sudo apt-get install git autoconf automake libtool make gcc g++ \ To build GHC, you need GHC. Unfortunately, the version that ships with current Debian systems is too old to build a current GHC; we recommend installing the -latest stable release from the [GHC downloads page](https://www.haskell.org/ghc/download_ghc_8_6_3.html). +latest stable release from the [GHC page](https://www.haskell.org/ghc/). At the +time of writing, the latest release is GHC 8.6.3, which can be downloaded +from [here](https://www.haskell.org/ghc/download_ghc_8_6_3.html). So: ```sh -$ wget https://downloads.haskell.org/~ghc/8.6.3/ghc-8.6.3-x86_64-deb9-linux.tar.xz -$ tar xf ghc-8.6.3-x86_64-deb9-linux.tar.xz -$ cd ghc-8.6.3 -$ ./configure -$ sudo make install -$ cd .. -$ ghc --version +wget https://downloads.haskell.org/~ghc/8.6.3/ghc-8.6.3-x86_64-deb9-linux.tar.xz +tar xf ghc-8.6.3-x86_64-deb9-linux.tar.xz +cd ghc-8.6.3 +./configure +sudo make install +cd .. +``` + +Use `ghc --version` to check that you are now using the correct version: + +``` The Glorious Glasgow Haskell Compilation System, version 8.6.3 ``` @@ -55,18 +61,22 @@ You'll need an up-to-date version of Cabal. If you have one installed already, you can **use cabal to install cabal**: ```sh -$ cabal install cabal-install +cabal install cabal-install ``` Otherwise, you will have to **install Cabal manually** from Hackage: ```sh -$ wget https://hackage.haskell.org/package/cabal-install-2.4.1.0/cabal-install-2.4.1.0.tar.gz -$ tar xf cabal-install-2.4.1.0.tar.gz -$ cd cabal-install-2.4.1.0 -$ ./bootstrap.sh -$ echo 'export PATH=$HOME/.cabal/bin:$PATH' >> ~/.bashrc -$ cabal --version +wget https://hackage.haskell.org/package/cabal-install-2.4.1.0/cabal-install-2.4.1.0.tar.gz +tar xf cabal-install-2.4.1.0.tar.gz +cd cabal-install-2.4.1.0 +./bootstrap.sh +echo 'export PATH=$HOME/.cabal/bin:$PATH' >> ~/.bashrc +``` + +Use `cabal --version` to check that you are using the correct version: + +``` cabal-install version 2.4.1.0 compiled using version 2.4.1.0 of the Cabal library ``` @@ -80,19 +90,19 @@ These two fellows are needed for generating lexers and parsers. The versions available from Debian are too old, but they can easily be installed with Cabal: ```sh -$ cabal install alex happy +cabal install alex happy ``` ### Dependencies for building documentation ```sh -$ sudo apt-get install python-sphinx texlive-xetex texlive-fonts-recommended fonts-lmodern texlive-latex-recommended texlive-latex-extra +sudo apt-get install python-sphinx texlive-xetex texlive-fonts-recommended fonts-lmodern texlive-latex-recommended texlive-latex-extra ``` ### Dependencies for various development tasks ```sh -$ sudo apt-get install linux-tools-generic xutils-dev +sudo apt-get install linux-tools-generic xutils-dev ``` @@ -100,51 +110,79 @@ $ sudo apt-get install linux-tools-generic xutils-dev ```sh # needed only once, URL rewrite rule is persisted in ${HOME}/.gitconfig -$ git config --global url."git://github.com/ghc/packages-".insteadOf git://github.com/ghc/packages/ +git config --global url."git://github.com/ghc/packages-".insteadOf git://github.com/ghc/packages/ # clone GHC's main Git repository (creates './ghc' folder in CWD) -$ git clone --recursive git://github.com/ghc/ghc +git clone --recursive git://github.com/ghc/ghc ``` ## Making your first build ```sh -$ cd ghc/ +cd ghc/ # configure build -$ cp mk/build.mk.sample mk/build.mk +cp mk/build.mk.sample mk/build.mk ## edit mk/build.mk to remove the comment marker # on the line "BuildFlavour = devel2" -$ ./boot -$ ./configure +./boot +./configure # NOTE: On Windows you need to download some binary distributables before being # able to build. This only has to be done once and can be done by adding a flag # to the call to configure: -$ ./configure --enable-tarballs-autodownload +./configure --enable-tarballs-autodownload # Build GHC -# The -j flag says how many parallel jobs to use. Dependin on your system, the +# The -j flag says how many parallel jobs to use. Depending on your system, the # best value tends to be between N/2 and N, where N is the number of cores on -# your machine. Pick lower values if you're low on RAM, or have a slow disk. -$ make -j8 +# your machine. Pick lower values if you're low on RAM, or if have a slow +# disk, or if you have other important or demanding tasks running. +make -j8 ``` Now go make yourself some coffee while the build runs. ## Running your freshly-built GHC +If all went according to plan, you should now have a working compiler in +`./inplace/bin/ghc-stage2`. Under normal circumstances, you will not want to +actually install that compiler. Instead, you can invoke it directly, either as +a regular GHC: + ```sh ./inplace/bin/ghc-stage2 ``` -GHCi is launched with: +Or in interactive mode, a.k.a. `ghci`: ```sh ./inplace/bin/ghc-stage2 --interactive ``` +Let's give it a spin. Paste the following into a file named `test.hs`: + +```haskell +module Main where + +main = do + putStrLn "All is fine." +``` + +Compile it with: + +```sh +./inplace/bin/ghc-stage2 test.hs +``` + +This should produce a binary `test`, which you can run as `./test`; it should +print: + +``` +All is fine. +``` + ## Rebuilding While working on GHC, you will need to rebuild often; however, most of the @@ -153,16 +191,25 @@ do to speed things up: 1. Select `BuildFlavour = devel2` in your `mk/build.mk`. - (Further reading: [Make GHC build more quickly](building/using#-howtomake-gh-cbuildquickly)) + (Further reading: [Make GHC build more quickly](building/using#how-to-make-ghc-build-quickly)) 2. In most cases, you will not make changes in more than one subdirectory at once. So instead of saying `make` at the top level, you can `cd` into the directory where you made your changes first, that is, `compiler`, `utils`, `ghc`, or `libraries`. - (Further reading: [Building a single sub-component](building/using#-buildingasinglesub-component)) + (Further reading: [Building a single sub-component](building/using#building-a-single-sub-component)) 3. Set `stage=2` in your `mk/build.mk` file. This will keep the Stage 1 compiler "frozen", re-running only Stage 2 compilation. For most development work, this is perfectly fine. - (Further reading: [Freezing the stage 1 compiler](building/using#-freezingstage1), + (Further reading: [Freezing the stage 1 compiler](building/using#freezing-stage-1), [Stages](building/architecture/idiom/stages)) 4. Use `make fast`. This will skip rebuilding dependencies. - (Further reading: [Skip dependency building](building/using#-skipdependencybuilding)) + (Further reading: [Skip dependency building](building/using#skip-dependency-building)) + +At this point, it is probably worth mentioning the concept of "Stages". + +In a nutshell: Stage 0 is your bootstrap compiler (installed from a binary +release); Stage 1 is the new GHC codebase compiled with Stage 0; Stage 2 is the +final, release-grade build, made with Stage 2. + +[Idiom/Stages](building/architecture/idiom/stages) explains the concept in more +detail, and provides a rationale. From c8778b34b096c9f6dc2f315d2a3a868dce18bac5 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Tue, 29 Jan 2019 14:21:15 +0100 Subject: [PATCH 03/30] Newcomers: Running the testsuite --- newcomers-tutorial.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 67ed38d..81d1738 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -183,6 +183,39 @@ print: All is fine. ``` +## Running Tests + +GHC comes with an extensive test suite, located in `./testsuite`; the actual +tests are organized into a directory tree under `./testsuite/tests`, sorted by +topic or general feature area. For example, tests relating to GHC's relatively +new dependently-typed features can be found in `./testsuite/tests/dependent`. + +To run the entire testsuite, use: + +```sh +make test +``` + +This should take a while. + +During development, you will not normally want to re-run the entire testsuite +all the time, so instead, you can navigate to a subdirectory and run `make` +there. Try it: + +```sh +cd ./testsuite/tests/dependent +make +cd ../../.. +``` + +Another option is to only run selected test cases: + +```sh +TEST=T11432 make test +``` + +This will run only tests named `T11432`, anywhere in the testsuite subtree. + ## Rebuilding While working on GHC, you will need to rebuild often; however, most of the From c920584b1b2fed3d66afc649041d7d64da33ed86 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Tue, 29 Jan 2019 16:12:11 +0100 Subject: [PATCH 04/30] WIP: newcomers "writing code" section --- newcomers-tutorial.md | 94 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 81d1738..6a8626e 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -246,3 +246,97 @@ final, release-grade build, made with Stage 2. [Idiom/Stages](building/architecture/idiom/stages) explains the concept in more detail, and provides a rationale. + +## Finding a task to work on + +Since you came here, you probably have an idea what to work on already - but if +not, do check the issues tagged "newcomer" **(TODO: put the proper link to the +issue tracker here)** - these are issues we expect to be "low hanging fruit". +Of course, we can't ever be sure of how hard a task is before doing it, so +apologies if one of these is too hard. + +Apart from that, you are encouraged to ask for a starting point on IRC or the +`ghc-devs` [mailing list](mailing-lists-and-irc). There someone familiar with +the process can help you find a ticket that matches your expertise and help you +when you get stuck. + +## Working with the code + +GHC is an old codebase, with plenty of historic quirks to it, and the things it +does are intrinsically complex, so it can be quite overwhelming. Don't let that +discourage you, and don't hesitate to ask questions. + +### Coding Style + +#### General + +GHC doesn't follow a uniform coding style across all modules, and we do not +enforce any particular one. The general rule is to stick to the same coding +style as is already used in the file you're editing. If you must make stylistic +changes, commit them separately from functional changes, so that someone +looking back through the change logs can easily distinguish them. + +It's much better to write code that is transparent than to write code that is +short. + +#### Comments + +We mainly use 3 types of comments in GHC. + +First, all top-level entities should have Haddock comments that briefly +describe what they do and, if needed, why they are there. Haddock comments +are included in HTML documentation, so as usual, this is the place to document +an entity's public-ish interface. + +Second, short explanations, caveats, warnings, etc., are written inline. As a +general rule of thumb, these should be one-liners, or even fit on the same line +together with the thing they document. This is great for nitty-gritty +implementation details. + +Third, for longer explanations, background information, bigger-picture +rationales, history, etc., we have adopted a system we call "Notes". Here's +what that looks like: + +```haskell +prepareRhs :: SimplEnv -> OutExpr -> SimplM (SimplEnv, OutExpr) +-- Adds new floats to the env iff that allows us to return a good RHS +prepareRhs env (Cast rhs co) -- see Note [Float coercions] + | (ty1, _ty2) <- coercionKind co -- Do *not* do this if rhs is unlifted + , not (isUnLiftedType ty1) -- see Note [Float coercions (unlifted)] + = do { (env', rhs') <- makeTrivial env rhs + ; return (env', Cast rhs' co) } + + ...more equations for prepareRhs.... + +{- + +Note [Float coercions] +~~~~~~~~~~~~~~~~~~~~~~ +When we find the binding + x = e `cast` co +we'd like to transform it to + x' = e + x = x `cast` co -- A trivial binding +There's a chance that e will be a constructor application or function, or something +like that, so moving the coerion to the usage site may well cancel the coersions +and lead to further optimisation. + ...more stuff about coercion floating... +-} +``` + +Note that: + +- The Note itself is a long block of prose with a header in a standard format. + It can (and often will) be quite long, and include examples. +- The wiggly underline is part of the convention and should always be present. +- The Note title acts as an "anchor", and the inline reference to the Note must + replicate the word "Note" and the Note's title that follows exactly, to ease + navigation. +- The Note reference is always put in the direct vicinity of the code it + pertains to. It may be accompanied by a very short hint, as seen here. +- The Note itself however can be anywhere that makes sense. Often, you will see + a whole slab of Notes at the end of a module, or in between sections of a + module. Sometimes, though not always, there is a bit of a narrative to the + group of Notes as a whole. +- Notes can be referenced across modules; in that case, the reference is + written like this: `-- see Note [Float coercions] in SpecConstr.lhs` From e1b210a40b91bbd2b9a8ca39ab0f08c43e8de647 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Tue, 29 Jan 2019 16:19:31 +0100 Subject: [PATCH 05/30] Structure for remainder of Newcomer's Guide --- newcomers-tutorial.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 6a8626e..96cdc63 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -340,3 +340,29 @@ Note that: group of Notes as a whole. - Notes can be referenced across modules; in that case, the reference is written like this: `-- see Note [Float coercions] in SpecConstr.lhs` + +### Adding Tests + +TODO + +## Submitting Your Code + +### Committing + +TODO + +### Validating + +TODO + +### Issuing Merge Requests + +TODO + +## When things go pear-shaped + +TODO + +## Further Reading + +TODO From a4881e6e7da0cb478f83cfebfb0ad63b7620e89d Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Wed, 30 Jan 2019 11:03:03 +0100 Subject: [PATCH 06/30] Condensed "Installing Prerequisites" --- newcomers-tutorial.md | 103 ++++++++++-------------------------------- 1 file changed, 25 insertions(+), 78 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 96cdc63..054c882 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -13,98 +13,45 @@ question is never too silly to ask. ## Installing Prerequisites -For the sake of keeping things simple, we will assume a Debian-based GNU/Linux -system. GHC builds on a plethora of platforms; check [Setting up your system +GHC builds on a plethora of platforms; check [Setting up your system for building GHC](building/preparation) for detailed instructions for your platform of choice. -### Essential Dependencies +A minimal list of prerequisites: -The following list of packages are essential for GHC development: +### Essential Tools: -```sh -sudo apt-get install git autoconf automake libtool make gcc g++ \ - libgmp-dev ncurses-dev libtinfo-dev python3 xz-utils -``` - -### GHC - -To build GHC, you need GHC. Unfortunately, the version that ships with current -Debian systems is too old to build a current GHC; we recommend installing the -latest stable release from the [GHC page](https://www.haskell.org/ghc/). At the -time of writing, the latest release is GHC 8.6.3, which can be downloaded -from [here](https://www.haskell.org/ghc/download_ghc_8_6_3.html). - -So: - -```sh -wget https://downloads.haskell.org/~ghc/8.6.3/ghc-8.6.3-x86_64-deb9-linux.tar.xz -tar xf ghc-8.6.3-x86_64-deb9-linux.tar.xz -cd ghc-8.6.3 -./configure -sudo make install -cd .. -``` - -Use `ghc --version` to check that you are now using the correct version: - -``` -The Glorious Glasgow Haskell Compilation System, version 8.6.3 -``` - -For non-Debian platforms, refer to [the GHC download page](https://www.haskell.org/ghc/download_ghc_8_6_3.html) -for an appropriate install package. - -### Cabal - -You'll need an up-to-date version of Cabal. If you have one installed already, -you can **use cabal to install cabal**: - -```sh -cabal install cabal-install -``` +- **git** +- **a typical build toolchain**: autoconf, automake, libtool, make, gcc, g++ +- **python3** -Otherwise, you will have to **install Cabal manually** from Hackage: +Install these as usual. -```sh -wget https://hackage.haskell.org/package/cabal-install-2.4.1.0/cabal-install-2.4.1.0.tar.gz -tar xf cabal-install-2.4.1.0.tar.gz -cd cabal-install-2.4.1.0 -./bootstrap.sh -echo 'export PATH=$HOME/.cabal/bin:$PATH' >> ~/.bashrc -``` +### Essential Build Dependencies: -Use `cabal --version` to check that you are using the correct version: +- **libtinfo** +- **libgmp** +- **ncurses** +- **xz-utils** -``` -cabal-install version 2.4.1.0 -compiled using version 2.4.1.0 of the Cabal library -``` - -Refer to [the Cabal page on Hackage](https://hackage.haskell.org/package/cabal-install) -for the latest version of Cabal and adjust appropriately. +Install these as usual. You will need the development packages for these +(typically named `something-dev` on most Unix-like OS distributions). -### Alex and Happy +### Haskell Toolchain: -These two fellows are needed for generating lexers and parsers. The versions -available from Debian are too old, but they can easily be installed with Cabal: - -```sh -cabal install alex happy -``` +- **GHC**; you need a reasonably new version, we recommend the latest stable + release, which you can find via the [GHC homepage](https://www.haskell.org/ghc/). +- **Cabal**; either install through your distro, and then upgrade with `cabal + install cabal-install`, or follow the steps to install manually outlined on + [Cabal's Hackage Page](https://hackage.haskell.org/package/cabal-install) +- **Alex** and **Happy**; use Cabal to install these. ### Dependencies for building documentation -```sh -sudo apt-get install python-sphinx texlive-xetex texlive-fonts-recommended fonts-lmodern texlive-latex-recommended texlive-latex-extra -``` - -### Dependencies for various development tasks - -```sh -sudo apt-get install linux-tools-generic xutils-dev -``` - +- **sphinx** (probably called `python-sphinx` or similar) +- **xetex (texlive) and fonts** (e.g. `texlive-xetex`, + `texlive-fonts-recommended`, `fonts-lmodern`, `texlive-latex-recommended`, + `texlive-latex-extra` - refer to your distro for available packages) ## Getting The Code From ce0af37ec70ebd483194b3931dad5bbbd2fe96fc Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Wed, 30 Jan 2019 13:08:54 +0100 Subject: [PATCH 07/30] Rewrite build instructions to Hadrian --- newcomers-tutorial.md | 91 +++++++++++++++++++------------------------ 1 file changed, 41 insertions(+), 50 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 054c882..c2311f8 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -68,25 +68,17 @@ git clone --recursive git://github.com/ghc/ghc ```sh cd ghc/ -# configure build -cp mk/build.mk.sample mk/build.mk - -## edit mk/build.mk to remove the comment marker # on the line "BuildFlavour = devel2" - -./boot -./configure - -# NOTE: On Windows you need to download some binary distributables before being -# able to build. This only has to be done once and can be done by adding a flag -# to the call to configure: -./configure --enable-tarballs-autodownload - # Build GHC -# The -j flag says how many parallel jobs to use. Depending on your system, the -# best value tends to be between N/2 and N, where N is the number of cores on -# your machine. Pick lower values if you're low on RAM, or if have a slow -# disk, or if you have other important or demanding tasks running. -make -j8 +./hadrian/build.sh -c -j +# On Windows, use instead: +# hadrian/build.bat -c -j + +# The -j flag says to run a parallel build, guessing an appropriate number of +# parallel jobs. If you are low on system resources (particularly CPU or disk +# I/O), it may be faster to not use this feature. +# +# The -c flag says to also bootstrap GHC's Haskell dependencies and configure +# the build, if necessary. ``` Now go make yourself some coffee while the build runs. @@ -94,18 +86,18 @@ Now go make yourself some coffee while the build runs. ## Running your freshly-built GHC If all went according to plan, you should now have a working compiler in -`./inplace/bin/ghc-stage2`. Under normal circumstances, you will not want to +`./_build/stage1/bin/ghc`. Under normal circumstances, you will not want to actually install that compiler. Instead, you can invoke it directly, either as a regular GHC: ```sh -./inplace/bin/ghc-stage2 +./_build/stage1/bin/ghc ``` Or in interactive mode, a.k.a. `ghci`: ```sh -./inplace/bin/ghc-stage2 --interactive +./_build/stage1/bin/ghc --interactive ``` Let's give it a spin. Paste the following into a file named `test.hs`: @@ -120,7 +112,7 @@ main = do Compile it with: ```sh -./inplace/bin/ghc-stage2 test.hs +./_build/stage1/bin/ghc test.hs ``` This should produce a binary `test`, which you can run as `./test`; it should @@ -140,56 +132,55 @@ new dependently-typed features can be found in `./testsuite/tests/dependent`. To run the entire testsuite, use: ```sh -make test +./hadrian/build.sh test ``` This should take a while. During development, you will not normally want to re-run the entire testsuite -all the time, so instead, you can navigate to a subdirectory and run `make` -there. Try it: +all the time; instead, you probably want to focus on just a handful of tests +to watch, only to run the full testsuite when you're done, to check for +regressions. Here's how you do that: ```sh -cd ./testsuite/tests/dependent -make -cd ../../.. +./hadrian/build.sh test --only=T11432 ``` -Another option is to only run selected test cases: +This will run only tests named `T11432`, anywhere in the testsuite subtree. You +can specify multiple tests, separated by whitespace: ```sh -TEST=T11432 make test +./hadrian/build.sh test --only="T11432 T12345 T11111" ``` -This will run only tests named `T11432`, anywhere in the testsuite subtree. - ## Rebuilding While working on GHC, you will need to rebuild often; however, most of the time, it is possible to avoid a full, slow build. Here's a few things you can do to speed things up: - -1. Select `BuildFlavour = devel2` in your `mk/build.mk`. - (Further reading: [Make GHC build more quickly](building/using#how-to-make-ghc-build-quickly)) -2. In most cases, you will not make changes in more than one subdirectory at - once. So instead of saying `make` at the top level, you can `cd` into the - directory where you made your changes first, that is, `compiler`, `utils`, - `ghc`, or `libraries`. - (Further reading: [Building a single sub-component](building/using#building-a-single-sub-component)) -3. Set `stage=2` in your `mk/build.mk` file. This will keep the Stage 1 - compiler "frozen", re-running only Stage 2 compilation. For most development - work, this is perfectly fine. - (Further reading: [Freezing the stage 1 compiler](building/using#freezing-stage-1), - [Stages](building/architecture/idiom/stages)) -4. Use `make fast`. This will skip rebuilding dependencies. - (Further reading: [Skip dependency building](building/using#skip-dependency-building)) +- Rebuild only the things that you're interested in. E.g.: + `./hadrian/build.sh _build/stage1/bin/ghc` to build only the compiler. +- Freeze the stage 1 compiler: `--freeze1`. Most of the time, rebuilding the + stage 1 compiler is not necessary, but the build system is not smart enough + to figure this out on its own; freezing stage 1 tells it to *never* rebuild + stage 1. +- Picking a faster *build flavor*. For working on the compiler, the `devel2` + flavour is usually the most appropriate: `--flavour=devel2`. If you can + afford to skip dependency rebuilds, and compile without any optimizations, + then the `fastest` flavour is probably best. At this point, it is probably worth mentioning the concept of "Stages". -In a nutshell: Stage 0 is your bootstrap compiler (installed from a binary -release); Stage 1 is the new GHC codebase compiled with Stage 0; Stage 2 is the -final, release-grade build, made with Stage 2. +In a nutshell: + +- **Stage 0** is your bootstrap compiler (installed from a binary release). +- **Stage 1** is the new GHC codebase compiled with Stage 0. +- **Stage 2** is the final, release-grade build, made with Stage 1. + +Note that the subdirectories under `./_build` are named after the stage that +produced them, not the one that is stored in them. This is why +`_build/stage1/bin/ghc` is the Stage **2** compiler, built with Stage 1. [Idiom/Stages](building/architecture/idiom/stages) explains the concept in more detail, and provides a rationale. From bd8f7fb768a743b983cdf8d8b736aefdc0fba1c2 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Wed, 30 Jan 2019 13:52:37 +0100 Subject: [PATCH 08/30] Adding tests --- newcomers-tutorial.md | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index c2311f8..f1aa3fb 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -193,6 +193,11 @@ issue tracker here)** - these are issues we expect to be "low hanging fruit". Of course, we can't ever be sure of how hard a task is before doing it, so apologies if one of these is too hard. +Either way, **if you are going to work on something, make sure a ticket exists +for it**. This is essential for all sorts of things, but most importantly, +attaching things to tickets makes sure they don't get lost. So if there is no +ticket for it yet, do file one. **(TODO: put link to issue tracker here)** + Apart from that, you are encouraged to ask for a starting point on IRC or the `ghc-devs` [mailing list](mailing-lists-and-irc). There someone familiar with the process can help you find a ticket that matches your expertise and help you @@ -281,7 +286,34 @@ Note that: ### Adding Tests -TODO +Tests live in the `./testsuite/tests` subdirectory, organized by topic / area. +Usually, tests are further organized into `should_compile`, `should_run`, and +`should_fail` subdirectories, though this is merely a convention. Inside each +"leaf" directory, you will find a file named `all.T`, which defines the tests +that live in the directory, and next to it, all the files that are needed to +compile (and possibly run) those tests. + +Here are the tests for properly adding a test case: + +1. Determine and navigate to the appropriate subdirectory in the testsuite + tree. +2. Pick a name for your test case. Unless you have a good reason to deviate, + the convention is to use the corresponding ticket number, prefixed with a + `T`. If you need more than one test case for that ticket, append something + to make it unique - either a sequential lowercase letter, or some meaningful + hint. Note however that some subdirectories use a different naming + convention, which takes precedence over the `Tnnnnn` format. + Let's assume `T12345` is your chosen test name. +3. Create a file `T12345.hs`, and implement the test in it (as Haskell code). +4. Open the `all.T` file, and add your test. It should look something like + this: `test('T12345', normal, compile_and_run, [''])`. +5. Navigate back to the project root and run your test: `./hadrian/build.sh + test --only="T12345"`. + +Further reading: + +- The full guide to [adding test cases](building/running-tests/adding) explores + more features of the testsuite driver. ## Submitting Your Code From 1e37d87e4e30cf3202a86b1de0629e83a77df740 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Thu, 31 Jan 2019 17:51:05 +0100 Subject: [PATCH 09/30] Newcomers Tutorial: Committing, Validating, Submitting --- newcomers-tutorial.md | 87 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 81 insertions(+), 6 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index f1aa3fb..41d4a3d 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -53,14 +53,21 @@ Install these as usual. You will need the development packages for these `texlive-fonts-recommended`, `fonts-lmodern`, `texlive-latex-recommended`, `texlive-latex-extra` - refer to your distro for available packages) +### Required Bureaucracy + +GHC uses its own gitlab environment, found at +[gitlab.haskell.org](https://gitlab.haskell.org/). In order to submit merge +requests, file issues, and participate in code reviews, you will need an +account. + ## Getting The Code ```sh # needed only once, URL rewrite rule is persisted in ${HOME}/.gitconfig -git config --global url."git://github.com/ghc/packages-".insteadOf git://github.com/ghc/packages/ +git config --global url."git@gitlab.haskell.org:ghc/packages-".insteadOf git@gitlab.haskell.org:ghc/packages/ # clone GHC's main Git repository (creates './ghc' folder in CWD) -git clone --recursive git://github.com/ghc/ghc +git clone --recursive git@gitlab.haskell.org:ghc/ghc.git ``` ## Making your first build @@ -319,15 +326,77 @@ Further reading: ### Committing -TODO +The usual git hygiene applies: + +- Create a new branch for each merge request, based off of `master`. If you + have push permissions on the main `ghc/ghc` repo on + [gitlab.haskell.org](https://gitlab.haskell.org), you may create and push + your branch there; otherwise, fork the project and push to your fork. +- Write a one-line summary, preferably no more than 50 columns, followed by 1 + blank line, followed by an (optional, but recommended) longer description of + the commit. +- Tastefully squash commits before submitting. A merge request may contain + multiple commits, but in the interest of keeping line noise low, functionally + related commits should be squashed together. Also, as a rule of thumb, each + individual commit should validate cleanly (see below) on its own, so as not + to disturb `git bisect`. +- Rebase onto `master` before submitting. This makes merge conflicts later on + less likely. + +On top of that, a few GHC-specific conventions: + +- Prefer imperative over third-person. Good: `Add frabber squabbing to + avoid bleebles`. Less good: `Adds frabber squabbing to avoid bleebles`. +- Mention relevant ticket(s) in the commit, like so: `#14567`. ### Validating -TODO +Before submitting your code, you should "validate" it: + +```sh +./hadrian/build.sh validate +``` + +GHC uses a Continuous Integration (CI) setup that automatically builds and +tests all new merge requests. The `validate` command allows you to perform an +identical build-and-test run locally, with the same configuration, so that you +can avoid submitting patches that break the build (and would thus needlessly +clog the CI queue). + +`validate` is also a nice, convenient way of doing a comprehensive and fairly +reliable full test run, providing more certainty at the expense of taking +(much) longer than running subsets of the testsuite on fast rebuilds. ### Issuing Merge Requests -TODO +Once you are happy with the state of your feature branch, issue a merge +request: + +- Push your branch if you haven't already +- In gitlab, navigate to your fork +- Select "Merge Requests" in the navigation menu on the left +- Click the big green "New Merge Request" button +- Follow the UI - it should be fairly self-explanatory. + +### What Happens Next? + +- The CI system will pick up your merge request, build it, run it, and report + back to gitlab. If all goes according to plan, the merge request will be + flagged "green". Keep in mind that a CI build takes quite some time, and + there is usually a queue of merge requests to be built, so don't expect + immediate feedback here. +- Someone privileged and knowledgeable will review your code. Most likely, + they will not accept your merge request right away, but instead tell you to + change a few things, or ask you to clarify why you did certain things the way + you did. A very common request is "could you add a Note explaining + this-and-that in more detail please?" So don't be alarmed or discouraged; + this is normal and part of the process, and absolutely does not mean your + contributions aren't welcome. They very much are. In a similar spirit, the + review queue tends to be quite long, so please do be patient. +- Once the reviewers are sufficiently happy with your patch, it will be merged + into `master`, and make its way into an upcoming GHC release. There is + nothing you need to do for this to happen yourself, though please do holler + if it takes awfully long. ## When things go pear-shaped @@ -335,4 +404,10 @@ TODO ## Further Reading -TODO +- [Building](building) +- [Working Conventions](working-conventions) +- [The GHC Commentary](commentary) +- [Debugging](debugging) +- [The Architecture of Open Source Applications: The Glasgow Haskell Compiler](http://www.aosabook.org/en/ghc.html), + by Simon Marlow and Simon Peyton-Jones, two of the chief architects of GHC + and Haskell itself From 384e4f35681b1bdeb89fc803de609efb4fcb75fc Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 4 Feb 2019 10:04:16 +0100 Subject: [PATCH 10/30] Build instructions in newcomers guide - Always use devel2 - Use symbolic notation for building only stage2 ghc - Reworded explanation for skipping stage1 --- newcomers-tutorial.md | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 41d4a3d..b19b7b5 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -76,16 +76,24 @@ git clone --recursive git@gitlab.haskell.org:ghc/ghc.git cd ghc/ # Build GHC -./hadrian/build.sh -c -j +./hadrian/build.sh -c -j --flavour=devel2 # On Windows, use instead: -# hadrian/build.bat -c -j +# hadrian/build.bat -c -j --flavour=devel2 + +# For the remainder of this document, we will use the Linux/Unix style +# invocation; if you are on Windows, please substitute hadrian/build.bat for +# ./hadrian/build.sh as needed. # The -j flag says to run a parallel build, guessing an appropriate number of # parallel jobs. If you are low on system resources (particularly CPU or disk # I/O), it may be faster to not use this feature. -# + # The -c flag says to also bootstrap GHC's Haskell dependencies and configure # the build, if necessary. + +# --flavour=devel2 selects a build configuration more appropriate for working +# on the compiler itself; without it, a release build will be made, which takes +# significantly longer and is less convenient for development. ``` Now go make yourself some coffee while the build runs. @@ -166,12 +174,14 @@ While working on GHC, you will need to rebuild often; however, most of the time, it is possible to avoid a full, slow build. Here's a few things you can do to speed things up: -- Rebuild only the things that you're interested in. E.g.: - `./hadrian/build.sh _build/stage1/bin/ghc` to build only the compiler. +- Rebuild only the things that you're interested in. For example, to only build + GHC itself, say `./hadrian/build.sh stage2:exe:ghc-bin`. See + [the Hadrian README](https://gitlab.haskell.org/ghc/ghc/blob/master/hadrian/README.md) + for details. - Freeze the stage 1 compiler: `--freeze1`. Most of the time, rebuilding the - stage 1 compiler is not necessary, but the build system is not smart enough - to figure this out on its own; freezing stage 1 tells it to *never* rebuild - stage 1. + stage 1 compiler is not necessary, but it is incredibly difficult to + determine this in an automatic fashion, so we need to explicitly tell the + build system to not rebuild Stage 1. - Picking a faster *build flavor*. For working on the compiler, the `devel2` flavour is usually the most appropriate: `--flavour=devel2`. If you can afford to skip dependency rebuilds, and compile without any optimizations, From c0bff939c71484063c05dc772cb1bbd75b9e2723 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 4 Feb 2019 10:16:57 +0100 Subject: [PATCH 11/30] Move Windows note out of code block and into separate section --- newcomers-tutorial.md | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index b19b7b5..efd0cdd 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -79,24 +79,31 @@ cd ghc/ ./hadrian/build.sh -c -j --flavour=devel2 # On Windows, use instead: # hadrian/build.bat -c -j --flavour=devel2 +``` +- The `-j` flag says to run a parallel build, guessing an appropriate number of + parallel jobs. If you are low on system resources (particularly CPU or disk + I/O), it may be faster to not use this feature. +- The `-c` flag says to also bootstrap GHC's Haskell dependencies and configure + the build, if necessary. +- `--flavour=devel2` selects a build configuration more appropriate for working + on the compiler itself; without it, a release build will be made, which takes + significantly longer and is less convenient for development. -# For the remainder of this document, we will use the Linux/Unix style -# invocation; if you are on Windows, please substitute hadrian/build.bat for -# ./hadrian/build.sh as needed. +Now go make yourself some coffee while the build runs. -# The -j flag says to run a parallel build, guessing an appropriate number of -# parallel jobs. If you are low on system resources (particularly CPU or disk -# I/O), it may be faster to not use this feature. +### A Note To Windows Users -# The -c flag says to also bootstrap GHC's Haskell dependencies and configure -# the build, if necessary. +For the remainder of this document, we will use the Linux/Unix style +invocations; if you are on Windows, please substitute `hadrian/build.bat` for +`./hadrian/build.sh` as needed. -# --flavour=devel2 selects a build configuration more appropriate for working -# on the compiler itself; without it, a release build will be made, which takes -# significantly longer and is less convenient for development. -``` +Further, there are several ways of getting a somewhat Unix-like command line on +Windows, and each of them comes with some caveats. Depending on which of those +approaches you used, you may want to use one of the alternative build methods: -Now go make yourself some coffee while the build runs. +- `hadrian/build.stack.bat` +- `hadrian/build.cabal.bat` +- `hadrian/build.nix.bat` ## Running your freshly-built GHC From d3cd290fd669d6253418ee9a9fca7f9777d93bd2 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 4 Feb 2019 10:19:20 +0100 Subject: [PATCH 12/30] Mention Stage 3 in newcomers guide --- newcomers-tutorial.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index efd0cdd..aebd868 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -201,6 +201,8 @@ In a nutshell: - **Stage 0** is your bootstrap compiler (installed from a binary release). - **Stage 1** is the new GHC codebase compiled with Stage 0. - **Stage 2** is the final, release-grade build, made with Stage 1. +- **Stage 3** is a compiler built with the Stage 2 compiler. It is not normally + built, but can be enabled; its main use is in building cross-compilers. Note that the subdirectories under `./_build` are named after the stage that produced them, not the one that is stored in them. This is why From 7f5545e5cca64796722dc941cb76b1266fc448a4 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 11 Feb 2019 12:35:04 +0100 Subject: [PATCH 13/30] Suggest `ghc -e 1+2` as a quick test. --- newcomers-tutorial.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index aebd868..01e54fc 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -122,7 +122,17 @@ Or in interactive mode, a.k.a. `ghci`: ./_build/stage1/bin/ghc --interactive ``` -Let's give it a spin. Paste the following into a file named `test.hs`: +Let's give it a spin: + +```sh +./_build/stage1/bin/ghc -e '1+2' +``` + +This is a nice quick test with fairly good coverage. + + +If you want to take it a bit further, paste the following into a file named +`test.hs`: ```haskell module Main where From 9861c4a447fc1d51d92e48058ce82e5cf074da5d Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 11 Feb 2019 12:36:44 +0100 Subject: [PATCH 14/30] Use `validate`, not `hadrian/build validate` --- newcomers-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 01e54fc..277719d 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -383,7 +383,7 @@ On top of that, a few GHC-specific conventions: Before submitting your code, you should "validate" it: ```sh -./hadrian/build.sh validate +./validate ``` GHC uses a Continuous Integration (CI) setup that automatically builds and From 7064ebc3fed1c55730cd89a3245b632dced88986 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 11 Feb 2019 12:42:23 +0100 Subject: [PATCH 15/30] Mention the option of logging into gitlab with a github account --- newcomers-tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 277719d..20b1e2e 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -57,8 +57,8 @@ Install these as usual. You will need the development packages for these GHC uses its own gitlab environment, found at [gitlab.haskell.org](https://gitlab.haskell.org/). In order to submit merge -requests, file issues, and participate in code reviews, you will need an -account. +requests, file issues, and participate in code reviews, you will need to create +an account, or sign in with a GitHub account. ## Getting The Code From 01975fcc5ca22aaab16182e847416bd9d0fa441a Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 11 Feb 2019 12:44:54 +0100 Subject: [PATCH 16/30] Suggest reading up on code during first build --- newcomers-tutorial.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 20b1e2e..d4f7ac3 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -91,6 +91,10 @@ cd ghc/ Now go make yourself some coffee while the build runs. +This may also be a good time to orient yourself on the GHC codebase (see the +"Further Reading" section below for some starting points) and configure your +editor. (**TODO**: add link to editor configuration page) + ### A Note To Windows Users For the remainder of this document, we will use the Linux/Unix style From d86070cae21b62dc97f786a7b4adc69e0ba08134 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 11 Feb 2019 12:45:10 +0100 Subject: [PATCH 17/30] Add note on Hadrian --- newcomers-tutorial.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index d4f7ac3..3f175f2 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -109,6 +109,14 @@ approaches you used, you may want to use one of the alternative build methods: - `hadrian/build.cabal.bat` - `hadrian/build.nix.bat` +### A Note On Hadrian + +Hadrian is GHC's new build system, based on [Shake](https://shakebuild.com/). +It replaces the previous build system, which was based on `make`, and is +starting to show its age. Hadrian is fairly new, and while most of the quirks +have been ironed out, you may occasionally run into issues, for which we +apologize in advance. + ## Running your freshly-built GHC If all went according to plan, you should now have a working compiler in From 31ff080a91251aa3c244d1290327c8f0ec9952c3 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 11 Feb 2019 13:04:39 +0100 Subject: [PATCH 18/30] Added "pear-shaped" section --- newcomers-tutorial.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 3f175f2..8d5e896 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -441,7 +441,19 @@ request: ## When things go pear-shaped -TODO +- A common source of errors is not updating your submodules after switching + branches or checking out a different commit. So if you see a build failing + mysteriously, `git submodule update` is one of the first things to try. +- You may also want to re-run the "boot" and "configure" steps; just add `-b` + to your Hadrian command to do so - or use `./hadrian/build.sh boot` and + `./hadrian/build.sh configure` to run them separately. +- Sometimes, though hopefully not often, you may have to start from scratch, + throwing away all existing build artifacts and intermediates. Simply deleting + the `_build` subdirectory will do that for you. Note that this will require a + full rebuild of the stage 1 compiler. +- For a more radical approach, `git clean` can restore your working copy to a + pristine state. You will typically also want to do a `git submodule update + --init` after this. ## Further Reading From 2e4da7a87195be180fee21a29e64cfacebc9b60e Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Tue, 26 Feb 2019 15:43:18 +0100 Subject: [PATCH 19/30] Add links to issue tracker. --- newcomers-tutorial.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 8d5e896..0a8129a 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -93,7 +93,7 @@ Now go make yourself some coffee while the build runs. This may also be a good time to orient yourself on the GHC codebase (see the "Further Reading" section below for some starting points) and configure your -editor. (**TODO**: add link to editor configuration page) +editor. ### A Note To Windows Users @@ -236,15 +236,17 @@ detail, and provides a rationale. ## Finding a task to work on Since you came here, you probably have an idea what to work on already - but if -not, do check the issues tagged "newcomer" **(TODO: put the proper link to the -issue tracker here)** - these are issues we expect to be "low hanging fruit". +not, do check the [issues tagged +"newcomer"](https://gitlab.haskell.org/ghc/ghc/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=newcomer) - +these are issues we expect to be "low hanging fruit". Of course, we can't ever be sure of how hard a task is before doing it, so apologies if one of these is too hard. Either way, **if you are going to work on something, make sure a ticket exists for it**. This is essential for all sorts of things, but most importantly, attaching things to tickets makes sure they don't get lost. So if there is no -ticket for it yet, do file one. **(TODO: put link to issue tracker here)** +ticket for it yet, do file one. The issue tracker is [right +here](https://gitlab.haskell.org/ghc/ghc/issues/new). Apart from that, you are encouraged to ask for a starting point on IRC or the `ghc-devs` [mailing list](mailing-lists-and-irc). There someone familiar with From bec6a390862364f9d12417b858170b3355ebc7ec Mon Sep 17 00:00:00 2001 From: Sylvain Henry Date: Wed, 27 Feb 2019 11:20:26 +0100 Subject: [PATCH 20/30] Minor changes --- newcomers-tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 0a8129a..7622165 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -214,7 +214,7 @@ do to speed things up: - Picking a faster *build flavor*. For working on the compiler, the `devel2` flavour is usually the most appropriate: `--flavour=devel2`. If you can afford to skip dependency rebuilds, and compile without any optimizations, - then the `fastest` flavour is probably best. + then the `quickest` flavour is probably best. At this point, it is probably worth mentioning the concept of "Stages". @@ -435,7 +435,7 @@ request: this-and-that in more detail please?" So don't be alarmed or discouraged; this is normal and part of the process, and absolutely does not mean your contributions aren't welcome. They very much are. In a similar spirit, the - review queue tends to be quite long, so please do be patient. + review queue tends to be quite long, so please be patient. - Once the reviewers are sufficiently happy with your patch, it will be merged into `master`, and make its way into an upcoming GHC release. There is nothing you need to do for this to happen yourself, though please do holler From d0ab0eaca886fc9ab260d30ef787a6384bd866ef Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Wed, 27 Feb 2019 14:27:33 +0100 Subject: [PATCH 21/30] Remove submodule rewrite setup With upcoming MRs moving submodules to absolute paths, this is no longer necessary. --- newcomers-tutorial.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 7622165..7ef6491 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -62,10 +62,8 @@ an account, or sign in with a GitHub account. ## Getting The Code -```sh -# needed only once, URL rewrite rule is persisted in ${HOME}/.gitconfig -git config --global url."git@gitlab.haskell.org:ghc/packages-".insteadOf git@gitlab.haskell.org:ghc/packages/ +```sh # clone GHC's main Git repository (creates './ghc' folder in CWD) git clone --recursive git@gitlab.haskell.org:ghc/ghc.git ``` From faaa72e42f845a9524d8d8fc7345da3d4ecf7336 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Wed, 27 Feb 2019 14:28:31 +0100 Subject: [PATCH 22/30] Instruct people to use a gitlab fork --- newcomers-tutorial.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 7ef6491..238ab1a 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -62,10 +62,15 @@ an account, or sign in with a GitHub account. ## Getting The Code +Since we will be committing patches, we will need a GitLab fork, so it's best +to get that out of the way right away. So sign into GitLab, find [the GHC +project](https://gitlab.haskell.org/ghc/ghc), and fork it. + +Then clone it to your development machine: ```sh # clone GHC's main Git repository (creates './ghc' folder in CWD) -git clone --recursive git@gitlab.haskell.org:ghc/ghc.git +git clone --recursive git@gitlab.haskell.org:yourgitlabusername/ghc.git ``` ## Making your first build @@ -369,10 +374,7 @@ Further reading: The usual git hygiene applies: -- Create a new branch for each merge request, based off of `master`. If you - have push permissions on the main `ghc/ghc` repo on - [gitlab.haskell.org](https://gitlab.haskell.org), you may create and push - your branch there; otherwise, fork the project and push to your fork. +- Create a new branch for each merge request, based off of `master`. - Write a one-line summary, preferably no more than 50 columns, followed by 1 blank line, followed by an (optional, but recommended) longer description of the commit. From 6498c620ee620238a371f6f2fe7e4c79abf402df Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Wed, 27 Feb 2019 14:28:59 +0100 Subject: [PATCH 23/30] Reword `validate` section to be friendlier We want to suggest running validate before pushing, but we don't want to make it look like it's absolutely mandatory, and we definitely don't want to scare people into not sharing their contributions at all out of fear they might break CI. --- newcomers-tutorial.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 238ab1a..a435a97 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -370,6 +370,10 @@ Further reading: ## Submitting Your Code +As a general note, don't be afraid to make mistakes here: if you do, people +will kindly tell you and help you fix them. We much prefer a contribution that +isn't perfect yet over one that we never get to see at all. + ### Committing The usual git hygiene applies: @@ -394,22 +398,26 @@ On top of that, a few GHC-specific conventions: ### Validating -Before submitting your code, you should "validate" it: +GHC uses a Continuous Integration (CI) setup that automatically builds and +tests all new merge requests, and in general, we require CI to pass before a +patch gets merged into `master`. + +The `validate` command allows you to perform exactly the same +build-and-test run locally, with the same configuration, so that you +can avoid submitting patches that break the build (and would thus needlessly +clog the CI queue). Running it is as simple as this: ```sh ./validate ``` -GHC uses a Continuous Integration (CI) setup that automatically builds and -tests all new merge requests. The `validate` command allows you to perform an -identical build-and-test run locally, with the same configuration, so that you -can avoid submitting patches that break the build (and would thus needlessly -clog the CI queue). - -`validate` is also a nice, convenient way of doing a comprehensive and fairly +`validate` is also a convenient way of doing a comprehensive and fairly reliable full test run, providing more certainty at the expense of taking (much) longer than running subsets of the testsuite on fast rebuilds. +Beware, however, that a full `validate` run can take upwards of an hour, +depending on your hardware, so it may or may not be worth it. + ### Issuing Merge Requests Once you are happy with the state of your feature branch, issue a merge From 9f792a3b66c7baa0ad1c3e4ea665f2896dce7804 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Wed, 27 Feb 2019 14:42:37 +0100 Subject: [PATCH 24/30] Shorten comments section --- newcomers-tutorial.md | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index a435a97..f017162 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -277,21 +277,12 @@ short. #### Comments -We mainly use 3 types of comments in GHC. +Most of the comments in our codebase are fairly straightforward. One convention +that deserves some explanation however is a system we call "Notes", which we +believe strikes a good balance between minimal disruption of the code itself, +and sufficient amounts of explanations, background information, etc. -First, all top-level entities should have Haddock comments that briefly -describe what they do and, if needed, why they are there. Haddock comments -are included in HTML documentation, so as usual, this is the place to document -an entity's public-ish interface. - -Second, short explanations, caveats, warnings, etc., are written inline. As a -general rule of thumb, these should be one-liners, or even fit on the same line -together with the thing they document. This is great for nitty-gritty -implementation details. - -Third, for longer explanations, background information, bigger-picture -rationales, history, etc., we have adopted a system we call "Notes". Here's -what that looks like: +Here's what that looks like: ```haskell prepareRhs :: SimplEnv -> OutExpr -> SimplM (SimplEnv, OutExpr) From b68466aaffb07b75b21dd2d660e1e1db388d314a Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Wed, 27 Feb 2019 14:52:40 +0100 Subject: [PATCH 25/30] Link to hadrian documentation --- newcomers-tutorial.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index f017162..7422e4b 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -120,6 +120,10 @@ starting to show its age. Hadrian is fairly new, and while most of the quirks have been ironed out, you may occasionally run into issues, for which we apologize in advance. +Further information on building GHC with Hadrian can be found +[here](https://gitlab.staging.haskell.org/ghc/ghc/wikis/building/hadrian/quick-start), +and on [hadrian's own repository](https://github.com/snowleopard/hadrian). + ## Running your freshly-built GHC If all went according to plan, you should now have a working compiler in @@ -459,6 +463,8 @@ request: ## Further Reading - [Building](building) +- [Building GHC With Hadrian](https://gitlab.staging.haskell.org/ghc/ghc/wikis/building/hadrian/quick-start) +- [Hadrian](https://github.com/snowleopard/hadrian) - [Working Conventions](working-conventions) - [The GHC Commentary](commentary) - [Debugging](debugging) From 0bb62f9f215319703e3ebc063d5ae84f7345bfa4 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 4 Mar 2019 16:29:58 +0100 Subject: [PATCH 26/30] newcomers guide: Flatten dependencies list --- newcomers-tutorial.md | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 7422e4b..542f2a6 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -19,25 +19,18 @@ platform of choice. A minimal list of prerequisites: -### Essential Tools: - - **git** - **a typical build toolchain**: autoconf, automake, libtool, make, gcc, g++ - **python3** - -Install these as usual. - -### Essential Build Dependencies: - - **libtinfo** - **libgmp** - **ncurses** - **xz-utils** -Install these as usual. You will need the development packages for these -(typically named `something-dev` on most Unix-like OS distributions). +Install these as usual. You will need the development packages for any system +libraries (typically named `something-dev` on a Unix-like OS distribution). -### Haskell Toolchain: +And a Haskell toolchain: - **GHC**; you need a reasonably new version, we recommend the latest stable release, which you can find via the [GHC homepage](https://www.haskell.org/ghc/). @@ -46,7 +39,8 @@ Install these as usual. You will need the development packages for these [Cabal's Hackage Page](https://hackage.haskell.org/package/cabal-install) - **Alex** and **Happy**; use Cabal to install these. -### Dependencies for building documentation +For building documentation, you will also need the following, though you may +opt to skip this part for the time being. - **sphinx** (probably called `python-sphinx` or similar) - **xetex (texlive) and fonts** (e.g. `texlive-xetex`, From 04373da12b8204b04ffaf11c8f2790bdec7616fe Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 4 Mar 2019 16:30:26 +0100 Subject: [PATCH 27/30] newcomers guide: highlight --recursive --- newcomers-tutorial.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 542f2a6..520d5e6 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -67,6 +67,10 @@ Then clone it to your development machine: git clone --recursive git@gitlab.haskell.org:yourgitlabusername/ghc.git ``` +Note the `--recursive`; this is needed to pull in all the required git +submodules. Should you forget this, you can say +`git submodule update --init` after cloning. + ## Making your first build ```sh From cacfae6266caabbdd29a80a164587729b7d55248 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 4 Mar 2019 16:30:46 +0100 Subject: [PATCH 28/30] Formatting --- newcomers-tutorial.md | 1 + 1 file changed, 1 insertion(+) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 520d5e6..04d7fbb 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -81,6 +81,7 @@ cd ghc/ # On Windows, use instead: # hadrian/build.bat -c -j --flavour=devel2 ``` + - The `-j` flag says to run a parallel build, guessing an appropriate number of parallel jobs. If you are low on system resources (particularly CPU or disk I/O), it may be faster to not use this feature. From f98cba0beaabd76abb7beac7218350331418dc2d Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Mon, 4 Mar 2019 16:30:55 +0100 Subject: [PATCH 29/30] newcomers guide: hadrian reworded --- newcomers-tutorial.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 04d7fbb..684dc78 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -114,10 +114,8 @@ approaches you used, you may want to use one of the alternative build methods: ### A Note On Hadrian Hadrian is GHC's new build system, based on [Shake](https://shakebuild.com/). -It replaces the previous build system, which was based on `make`, and is -starting to show its age. Hadrian is fairly new, and while most of the quirks -have been ironed out, you may occasionally run into issues, for which we -apologize in advance. +It is pretty stable already and should work just fine, but if you run into +issues regardless, it may not be your fault. Further information on building GHC with Hadrian can be found [here](https://gitlab.staging.haskell.org/ghc/ghc/wikis/building/hadrian/quick-start), @@ -456,13 +454,14 @@ request: the `_build` subdirectory will do that for you. Note that this will require a full rebuild of the stage 1 compiler. - For a more radical approach, `git clean` can restore your working copy to a - pristine state. You will typically also want to do a `git submodule update - --init` after this. + pristine state. Due to the submodule setup we use, you will have to issue a + command like this one: + `git clean -xdf && git submodule foreach 'git clean -xdf' && git submodule update --init`. ## Further Reading - [Building](building) -- [Building GHC With Hadrian](https://gitlab.staging.haskell.org/ghc/ghc/wikis/building/hadrian/quick-start) +- [Building GHC With Hadrian](building/hadrian/quick-start) - [Hadrian](https://github.com/snowleopard/hadrian) - [Working Conventions](working-conventions) - [The GHC Commentary](commentary) From 5f88f0c7bc40a3fba0e04e6acf8308601c1f0f62 Mon Sep 17 00:00:00 2001 From: Tobias Dammers Date: Tue, 5 Mar 2019 12:41:14 +0100 Subject: [PATCH 30/30] newcomers-guide: restructure, add overview --- newcomers-tutorial.md | 75 +++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 34 deletions(-) diff --git a/newcomers-tutorial.md b/newcomers-tutorial.md index 684dc78..5d54ac1 100644 --- a/newcomers-tutorial.md +++ b/newcomers-tutorial.md @@ -9,7 +9,14 @@ IRC](mailing-lists-and-irc) who will gladly help you (although you may need to be patient). Don't forget that all GHC developers are still learning; your question is never too silly to ask. +The path to a successful merge request looks like this: +- Installing Prerequisites +- Getting The Code +- Building GHC +- Finding a task to work on +- Working with the code +- Submitting Your Code ## Installing Prerequisites @@ -71,7 +78,7 @@ Note the `--recursive`; this is needed to pull in all the required git submodules. Should you forget this, you can say `git submodule update --init` after cloning. -## Making your first build +## Building GHC ```sh cd ghc/ @@ -121,7 +128,7 @@ Further information on building GHC with Hadrian can be found [here](https://gitlab.staging.haskell.org/ghc/ghc/wikis/building/hadrian/quick-start), and on [hadrian's own repository](https://github.com/snowleopard/hadrian). -## Running your freshly-built GHC +### Running your freshly-built GHC If all went according to plan, you should now have a working compiler in `./_build/stage1/bin/ghc`. Under normal circumstances, you will not want to @@ -170,38 +177,7 @@ print: All is fine. ``` -## Running Tests - -GHC comes with an extensive test suite, located in `./testsuite`; the actual -tests are organized into a directory tree under `./testsuite/tests`, sorted by -topic or general feature area. For example, tests relating to GHC's relatively -new dependently-typed features can be found in `./testsuite/tests/dependent`. - -To run the entire testsuite, use: - -```sh -./hadrian/build.sh test -``` - -This should take a while. - -During development, you will not normally want to re-run the entire testsuite -all the time; instead, you probably want to focus on just a handful of tests -to watch, only to run the full testsuite when you're done, to check for -regressions. Here's how you do that: - -```sh -./hadrian/build.sh test --only=T11432 -``` - -This will run only tests named `T11432`, anywhere in the testsuite subtree. You -can specify multiple tests, separated by whitespace: - -```sh -./hadrian/build.sh test --only="T11432 T12345 T11111" -``` - -## Rebuilding +### Rebuilding While working on GHC, you will need to rebuild often; however, most of the time, it is possible to avoid a full, slow build. Here's a few things you can @@ -237,6 +213,37 @@ produced them, not the one that is stored in them. This is why [Idiom/Stages](building/architecture/idiom/stages) explains the concept in more detail, and provides a rationale. +### Running Tests + +GHC comes with an extensive test suite, located in `./testsuite`; the actual +tests are organized into a directory tree under `./testsuite/tests`, sorted by +topic or general feature area. For example, tests relating to GHC's relatively +new dependently-typed features can be found in `./testsuite/tests/dependent`. + +To run the entire testsuite, use: + +```sh +./hadrian/build.sh test +``` + +This should take a while. + +During development, you will not normally want to re-run the entire testsuite +all the time; instead, you probably want to focus on just a handful of tests +to watch, only to run the full testsuite when you're done, to check for +regressions. Here's how you do that: + +```sh +./hadrian/build.sh test --only=T11432 +``` + +This will run only tests named `T11432`, anywhere in the testsuite subtree. You +can specify multiple tests, separated by whitespace: + +```sh +./hadrian/build.sh test --only="T11432 T12345 T11111" +``` + ## Finding a task to work on Since you came here, you probably have an idea what to work on already - but if