User Tools

Site Tools


build:fallbacks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
build:fallbacks [2018/03/01 17:05]
Yann Pouillon [Frequently Asked Querstions]
build:fallbacks [2020/08/19 12:55] (current)
Jean-Michel Beuken [Building the fallbacks with Abinit]
Line 1: Line 1:
 ====== Fallbacks ====== ====== Fallbacks ======
 +
 +===== In brief =====
 +
 +
 +Abinit Fallbacks is a package builder for the external dependencies of Abinit,
 +in development environments lacking these components. They do not provide full
 +support for the advanced features of Abinit nor HPC-grade calculation
 +capabilities. They are designed to let developers quickly test new versions of
 +these external dependencies in various situations before proposing upgrades, as
 +well as to allow end-users to run calculations on their favorite PCs.
  
 ===== Objectives ===== ===== Objectives =====
  
-In case some dependencies are missing on your computers, Abinit provides fallback libraries that you can build and install from their sources before compiling Abinit itself. ​Although ​they do not offer the same level of reliability and performance as packages that a skilled administrator would build, test, and install, ​they will let you preview some advanced features and decide which ones are the most important to you and would be worth installing fully on your system.+In case some dependencies are missing on your computers, Abinit provides fallback libraries that you can build and install from their sources before compiling Abinit itself. ​Note however that they do not offer the same level of reliability and performance as packages that a skilled administrator would build, test, and install, ​
  
 Fallbacks are also useful for developers who want to test a new version of an external dependency before setting it as a default for future releases of Abinit and have it deployed on the Abinit Test Farm. Except when this new version fixes a bug affecting significantly the accuracy and/or stability of production calculations,​ such an operation is always performed on a __**development version**__ of Abinit and tested by several developers before being made available to the users. Fallbacks are also useful for developers who want to test a new version of an external dependency before setting it as a default for future releases of Abinit and have it deployed on the Abinit Test Farm. Except when this new version fixes a bug affecting significantly the accuracy and/or stability of production calculations,​ such an operation is always performed on a __**development version**__ of Abinit and tested by several developers before being made available to the users.
Line 14: Line 24:
  
 Here are the most relevant links to get useful information about the project and its status: Here are the most relevant links to get useful information about the project and its status:
-  * https://​gitlab.abinit.org/​buildbot/​abinit-fallbacks +  * [[https://​gitlab.abinit.org/​buildbot/​abinit-fallbacks|sources]] 
-  * https://​gitlab.abinit.org/​buildbot/​abinit-fallbacks/​issues +  * [[https://​gitlab.abinit.org/​buildbot/​abinit-fallbacks/​issues|issues]] 
-  * https://​gitlab.abinit.org/​buildbot/​abinit-fallbacks/​milestones +  * [[https://​gitlab.abinit.org/​buildbot/​abinit-fallbacks/​milestones|milestones]]
- +
-For historical reasons, a secondary repository is still kept on Launchpad, where the contents of the home repository are periodically mirrored: +
-  * https://​launchpad.net/​abinit-fallbacks +
 ===== Minimum requirements ===== ===== Minimum requirements =====
  
Line 27: Line 33:
   * the [[http://​www.gnu.org/​software/​wget/​|Wget]] and/or [[http://​curl.haxx.se/​|cURL]] utilities.   * the [[http://​www.gnu.org/​software/​wget/​|Wget]] and/or [[http://​curl.haxx.se/​|cURL]] utilities.
 If your network configuration does not allow for the downloading of remote packages, you will have to place the source tarballs in the //​~/​.abinit/​tarballs%%/​%%//​ directory manually. If your network configuration does not allow for the downloading of remote packages, you will have to place the source tarballs in the //​~/​.abinit/​tarballs%%/​%%//​ directory manually.
- 
-Since the recommended way of getting the Abinit Fallbacks is by cloning its repository from the Abinit Website, we highly recommend you to install the GNU Autotools on your system first: 
-  * [[http://​www.gnu.org/​software/​autoconf/​|Autoconf]];​ 
-  * [[http://​savannah.gnu.org/​projects/​autoconf-archive/​|Autoconf Archive]]; 
-  * [[http://​www.gnu.org/​software/​automake/​|Automake]];​ 
-  * [[http://​www.gnu.org/​software/​libtool/​|Libtool]]. 
  
 To build the fallbacks, you will also need a working development environment,​ including in particular: To build the fallbacks, you will also need a working development environment,​ including in particular:
Line 47: Line 47:
 Since the fallbacks evolve at a very different pace from Abinit and have very different objectives, they are available as a separate package. Typically, in one year, there will be 3 minor releases of Abinit, while 2 or 3 of the fallbacks at most will be upgraded asynchronously,​ mainly during active development phases. In parallel, the build system that wraps their installation will be refactored, upgraded, and tested. Since the fallbacks evolve at a very different pace from Abinit and have very different objectives, they are available as a separate package. Typically, in one year, there will be 3 minor releases of Abinit, while 2 or 3 of the fallbacks at most will be upgraded asynchronously,​ mainly during active development phases. In parallel, the build system that wraps their installation will be refactored, upgraded, and tested.
  
-The independent fallbacks can be downloaded either from the Abinit Website ​or Launchpad, at your preference. You can either download the source tarballs released with Abinit or clone one of the Git repositories directly.+The independent fallbacks can be downloaded either from the [[https://​www.abinit.org/​fallbacks|Abinit Website]]. You can either download the source tarballs released with Abinit or clone one of the Git repositories directly.
  
 +==== Requirements ====
 +
 +Since the recommended way of getting the Abinit Fallbacks is by cloning its repository from the Abinit Website, we highly recommend you to install the GNU Autotools on your system first:
 +  * [[http://​www.gnu.org/​software/​autoconf/​|Autoconf]];​
 +  * [[http://​savannah.gnu.org/​projects/​autoconf-archive/​|Autoconf Archive]];
 +  * [[http://​www.gnu.org/​software/​automake/​|Automake]];​
 +  * [[http://​www.gnu.org/​software/​libtool/​|Libtool]].
 ==== Getting the source code from the Abinit Website ==== ==== Getting the source code from the Abinit Website ====
  
-The official Abinit Fallbacks project page is there: ​[[https://​gitlab.abinit.org/​buildbot/​abinit-fallbacks]]+The official Abinit Fallbacks project page is [[https://​gitlab.abinit.org/​buildbot/​abinit-fallbacks|here]]
  
 If you are an active Abinit Developer, we highly recommend you to [[developers:​git:​access_config|configure your access]] before anything else. We will suppose this is the case from now on. If you are an active Abinit Developer, we highly recommend you to [[developers:​git:​access_config|configure your access]] before anything else. We will suppose this is the case from now on.
  
 To get the latest version, just type the following: To get the latest version, just type the following:
-<code bash>git clone abinit-forge:​buildbot/​abinit-fallbacks</​code>​+<code bash>git clone abinit-forge:​buildbot/​abinit-fallbacks.git</​code>​
  
-==== Getting the source code from Launchpad ==== 
- 
-The Launchpad mirror project page is there: [[https://​launchpad.net/​abinit-fallbacks]] 
- 
-It is provided for historical reasons and convenience,​ but all our activities related to the fallbacks take place on the Abinit Website. 
- 
-The first time you download the fallbacks from Launchpad, we highly recommend you to set your environment as described in the [[developers:​git:​access_config|Access configuration]] page. In the following, we will suppose you did so. 
- 
-To get the latest version of the fallbacks, type: 
-<code bash>git clone lp:​abinit-fallbacks</​code>​ 
  
 ==== Downloading a source tarball ==== ==== Downloading a source tarball ====
Line 73: Line 70:
 When a new major or minor version of Abinit is about to be released, we usually package a snapshot of the fallbacks and distribute it as a source tarball. This file is called //​abinit-fallbacks-X.Y.Z.tar.gz//,​ where X and Y correspond to the highest X.Y Abinit version for which compatibility has been tested, and Z is a patch level starting from 0. Please note that Z is used exclusively by the fallbacks and has nothing to do with Abinit. For instance, the tarball //​abinit-falbacks-8.4.2.tar.gz//​ contains fallbacks compatible with all 8.4.* versions of Abinit. When a new major or minor version of Abinit is about to be released, we usually package a snapshot of the fallbacks and distribute it as a source tarball. This file is called //​abinit-fallbacks-X.Y.Z.tar.gz//,​ where X and Y correspond to the highest X.Y Abinit version for which compatibility has been tested, and Z is a patch level starting from 0. Please note that Z is used exclusively by the fallbacks and has nothing to do with Abinit. For instance, the tarball //​abinit-falbacks-8.4.2.tar.gz//​ contains fallbacks compatible with all 8.4.* versions of Abinit.
  
-Source tarballs generated so far can be downloaded from Launchpad (https://​launchpad.net/​abinit-fallbacks/​+download),​ but this location might change in the future.+/* Source tarballs generated so far can be downloaded from Launchpad (https://​launchpad.net/​abinit-fallbacks/​+download),​ but this location might change in the future. 
 +*/
  
 <WRAP info>​Since Abinit dependencies evolve slowly, a fallbacks bundle with version X.Y.Z is often compatible with the X.(Y-1).* and X.(Y+1).* versions of Abinit.</​WRAP>​ <WRAP info>​Since Abinit dependencies evolve slowly, a fallbacks bundle with version X.Y.Z is often compatible with the X.(Y-1).* and X.(Y+1).* versions of Abinit.</​WRAP>​
Line 103: Line 101:
 ===== Building the fallbacks with Abinit ===== ===== Building the fallbacks with Abinit =====
  
-For convenience,​ a release of the fallbacks comes with the source tarball of Abinit. It is possible to build it along with Abinit, although much more efficient to build it and install it separately, since you do not have to rebuild the fallbacks each time you wish to make a new build of Abinit. Please note that the version of the fallbacks you may find within the Abinit source tree will always ​be outdated.+For convenience,​ a release of the fallbacks comes with the source tarball of Abinit.\\ 
 +It is possible to build it along with Abinit, although much more efficient to build it and install it separately, since you do not have to rebuild the fallbacks each time you wish to make a new build of Abinit.\\  
 +/*Please note that the version of the fallbacks you may find within the Abinit source tree will perhaps ​be outdated.*/
  
-Building the fallbacks within Abinit is mainly of interest to developers who want to experiment with compile flags and explore issues related to compilers and portability. It can also help - to a reduced extent only - users who are stuck within very rigid configurations and restricted build environments. Here is the way to proceed:+Building the fallbacks within Abinit is mainly of interest to developers who want to experiment with compile flags and explore issues related to compilers and portability. It can also help - to a reduced extent only - users who are stuck within very rigid configurations and restricted build environments.\\ 
 +Here is the way to proceed:
 <code bash> <code bash>
 tar xvzf abinit-x.y.z.tar.gz tar xvzf abinit-x.y.z.tar.gz
Line 111: Line 112:
 mkdir tmp-build mkdir tmp-build
 cd tmp-build cd tmp-build
-../​configure ​+ 
-  ​--prefix="/​abinit/​install/​path" ​... options ... \ +# Create a "​config.ac9"​ file with all needed options 
-Abinit ​drops a build script into the transient/ subdirectory and configure +../​configure --with-config-file="config.ac9" 
-# fails because ​the fallbacks ​are missing ​and cannot be tested+</​code>​ 
-cd transient + 
-# Edit the build-abinit-fallbacks.sh script and set its install prefix +What happens in this case is that we first configure ​Abinit, so that it generates all the necessary information to build the fallbacks for missing external dependencies. The configure ​script of Abinit always gives priority to properly-installed external dependencies and will only select ​the minimal necessary ​fallbacks, in order to save time, trigger less bugs, and aim at maximum reliability.\\ 
-# as instructed therein+At this stage, ​the configure script has to fail, because it cannot anticipate the successful ​build of the fallbacks. 
 + 
 +<code bash> 
 +cd fallbacks
 ./​build-abinit-fallbacks.sh ./​build-abinit-fallbacks.sh
-make + 
-make install+# At the end of the process, the script provides the options with prefixes to fallbacks. 
 +# e.g. 
 +#   ​with_hdf5=/​WORKSPACE_PREFIX/​abinit-x.y.z/​tmp-build/​fallbacks/​install_fb/​vendor/​version/​hdf5/​1.10.6 
 +
 +# You can link these fallbacks with Abinit by copying the options to your ac9 file. 
 cd .. cd ..
-../​configure ​--prefix=/​abinit/​install/​path \ +../​configure --with-config-file="config.ac9
-  ​--with-fallbacks-prefix="/​path/​to/​fallbacks/​prefix" ​\ +
-  ... options ... +
 make -j8   # For example, if you have at least 8 cores available make -j8   # For example, if you have at least 8 cores available
 make check make check
 make install make install
 </​code>​ </​code>​
 +/* ../​configure --prefix=/​abinit/​install/​path \
 +*  --with-fallbacks-prefix="/​path/​to/​fallbacks/​prefix"​ \
 +*  ... options ... 
 +*/
  
-What happens in this case is that we first configure Abinit, so that it generates all the necessary ​information to build the fallbacks for missing external dependencies. The configure script of Abinit always gives priority to properly-installed external dependencies and will only select ​the minimal necessary ​fallbacks, ​in order to save time, trigger less bugs, and aim at maximum reliabilityAt this stage, ​the configure script has to failbecause it cannot anticipate the successful build of the fallbacks.+Once the information ​is available ​to the configure script of the fallbacks, ​we can build them within the Abinit source tree and install them outside.\\ 
 +NextAbinit uses them during its own build and we can install it normally after checking that everything is fine.\\ 
 +In further builds with the same compiler, the same fallbacks ​can be reused, i.e. no need to rebuild them.
  
-Once the information is available to the configure script of the fallbacks, we can build them within ​the Abinit source tree and install them outsideNext, Abinit uses them during its own build and we can install it normally after checking ​that everything is fine. In further builds with the same compiler, the same fallbacks can be reused, i.e. no need to rebuild them.+By default, ​the fallbacks are installed in "/​WORKSPACE_PREFIX/​abinit-x.y.z/​tmp-build/​fallbacks/​install_fb"​.\\ 
 +The variable //​fallbacks_prefix//​ can be changed in the //fallbacks/build-abinit-fallbacks.sh//​ script if you want to change ​the installation folder.\\ 
 +Please note that this path must be absolute, i.e. start with a slash\\
  
-One line must be changed in the //​transient/​build-abinit-fallbacks.sh//​ script for it to work. This script can indeed not guess where you want to install the fallbacks. This is why you have to replace the **PREFIX** keyword by the full installation path of your choice. Please note that this path must be absolute, i.e. start with a slash. The line to change is located at the top of the script: 
 <code bash> <code bash>
 +#!/bin/bash
 +
 +# Init
 +fallbacks_prefix="​@abinit_builddir@/​fallbacks/​install_fb/​@abi_fc_vendor@/​@abi_fc_version@"​
 +
 +...
 +</​code>​
 +
 +/* -------------------------------------------------------------------------------------------------
 +One line must be changed in the //​transient/​build-abinit-fallbacks.sh//​ script for it to work. This script can indeed not guess where you want to install the fallbacks. This is why you have to replace the **PREFIX** keyword by the full installation path of your choice. Please note that this path must be absolute, i.e. start with a slash. The line to change is located at the top of the script:
 +
 #!/bin/sh #!/bin/sh
  
Line 151: Line 177:
  
 ... ...
-</code>+*/
  
 ===== Using installed fallbacks ===== ===== Using installed fallbacks =====
Line 185: Line 211:
 .    .   
 </​code>​ </​code>​
-where files are ordered by compiler vendor, then compiler version, and finally build variant. Each variant corresponds to a set of build parameters and is summarized by a keyword:+where files are ordered by compiler vendor ​(intel, gnu, nag,...), then compiler version, and finally build variant. Each variant corresponds to a set of build parameters and is summarized by a keyword:
   * //gpu// for a GPU-aware configuration;​   * //gpu// for a GPU-aware configuration;​
-  * //mpi// for a MPI-aware configuration;​+  * //openmpi// or //mpich// for a MPI-aware configuration;​
   * //omp// for an OpenMP-aware configuration;​   * //omp// for an OpenMP-aware configuration;​
   * //sdebug// for a serial configuration with enhanced debug flags;   * //sdebug// for a serial configuration with enhanced debug flags;
Line 206: Line 232:
 ==== The configuration of a package fails ==== ==== The configuration of a package fails ====
  
-Errors may appear at configure time when you build the fallbacks. In order to help you diagnose the problems, most packages provide a //​config.log//​ file. You will find it in //​~fallbacks/​sources/​package-version/​config.log//,​ where you replace //package// by the name of the package and //version// by the corresponding version. Open this file and look for the last occurence of the "​failed"​ keyword, this is usually the place where you will get the most useful information. Looking around the corresponding line will give you useful hints about what made the configure script abort. With a little bit of practice, you will understand and fix most errors relatively quickly.+Errors may appear at configure time when you build the fallbacks. In order to help you diagnose the problems, most packages provide a //​config.log//​ file. You will find it in //​~fallbacks/​sources/​package-version/tmp-build/​config.log//,​ where you replace //package// by the name of the package and //version// by the corresponding version. Open this file and look for the last occurence of the "​failed"​ keyword, this is usually the place where you will get the most useful information. Looking around the corresponding line will give you useful hints about what made the configure script abort. With a little bit of practice, you will understand and fix most errors relatively quickly.
  
 ==== "​make"​ fails ==== ==== "​make"​ fails ====
build/fallbacks.1519920354.txt.gz · Last modified: 2018/03/01 17:05 by Yann Pouillon