Developer Setup 

Introduction 

This section explains how to build a workspace for maliput development. You’ll learn how to work with maliput and with delphyne repositories in a multi-repository workspace.

Workspaces 

Supported platforms 

Docker containerized workspaces (recommended): A docker image is provided in order to show the steps needed to set up the environment in a containerized workspace. When setting up docker, do not add yourself to the “docker” group since that represents a security risk (it is equivalent to password-less sudo). As a workaround, the instructions below use sudo for building the image and running the container.
Non-containerized workspaces: Ubuntu Focal Fossa 20.04 LTS only.

Prerequisites 

To get all the necessary tools, clone maliput_infrastructure locally.

git clone git@github.com:maliput/maliput_infrastructure.git

To pull private repositories, the current user default SSH keys will be used (and thus assumed as both necessary and sufficient for the purpose).
Containerized workspaces require having docker engine installed in host machine. Also, you can use nvidia-docker2. Follow their instructions if you want to install it.

The following assumes that you want to create a workspace containing all of Maliput’s repositories, with a focus on Maliput and Malidrive development. As such, it uses a workspace directory named maliput_ws and pulls sources from foxy/maliput.repos.

The instructions also cover how to optionally add delphyne repositories into your workspace. These repositories include a simulator and visualizer useful when working with Maliput road networks.

The instructions below first cover how to create a non-containerized workspace, followed by a containerized workspace. Note that using a containerized workspace is recommended.

Option 1: Create a non-containerized workspace 

The following creates a non-containerized workspace. Follow it if you are willing to install Maliput and its dependencies directly in your base operating system. Bear in mind that using a non-containerized workspace makes reproducing and troubleshooting issues harder for others. If in doubt, install your workspace within a container by following the instructions in the subsequent “Option 2” section.

Create the workspace folder 

mkdir maliput_ws

Note

These instructions assume maliput_ws is at the same level as maliput_infrastructure and repos_index.

Copy .repos file 

copy maliput_infrastructure/repos_index/foxy/maliput.repos into maliput_ws. It will be used to add the Maliput-related repositories to the workspace.

cp maliput_infrastructure/repos_index/foxy/maliput.repos maliput_ws/

Note

You can optionally add Delphyne-related repositories to your workspace:

cp maliput_infrastructure/repos_index/foxy/delphyne.repos maliput_ws/

Install dependencies 

sudo ./maliput_infrastructure/tools/install_dependencies.sh

Update all the repositories in your workspace 

Bring all the repositories listed in maliput.repos file:

cd maliput_ws
mkdir src
vcs import src < maliput.repos  # clone and/or checkout
# Optionally, run:
# vcs import src < delphyne.repos
vcs pull src  # fetch and merge (usually fast-forward)

This will clone repositories and/or checkout branches, tags or commits as necessary, followed by fetching and (likely) fast-forward merging to get branches up to date with their upstream counterpart. No merging takes place when a repository is at a given tag or commit. Note that you can continue to bring other repositories into your workspace by repeating the import and pull operation using additional .repos files.

Install all packages’ dependencies 

First update the ROS_DISTRO environment variable with your ros2 version, e.g.:

export ROS_DISTRO=foxy

Install dependencies via `rosdep`

rosdep update --include-eol-distros
rosdep install -i -y --rosdistro $ROS_DISTRO --skip-keys "pybind11" --from-paths src

Warning

Package dependencies are installed system wide. rosdep does not provide any support to remove the dependencies it brings. In this regard, disposable containerized workspaces help keep development environments clean (as system wide installations within a container are limited to that container).

Install drake 

Installing drake is only necessary when working with delphyne.repos, otherwise it will fail because drake_vendor is not in the workspace.

sudo ./src/drake_vendor/drake_installer

Source ROS environment 

source /opt/ros/$ROS_DISTRO/setup.bash

Option 2: Create a containerized workspace 

The following creates a containerized workspace. Maliput and its dependencies remain in the container and do not impact your operating system. Likewise, packages installed on your operating system do not impact the container. The uniformity of the container’s environment makes it easier for other developers to reproduce and resolve problems you may encounter.

Configuring a containerized workspace is similar to that of a non-containerized workspace. When the steps are identical, links to the non-containerized setup instructions are used. Machinery is provided to build and run a docker image and container for Maliput workspace development:

Build the docker image 

./maliput_infrastructure/docker/build.sh

If you are using nvidia-docker2 add the --nvidia option.

./maliput_infrastructure/docker/build.sh --nvidia

Note

build.sh --help for more options:

-i --image_name Name of the image to be built (default maliput_ws_ubuntu_focal).

-w --workspace_name Name of the workspace folder (default maliput_ws).

Create the workspace folder 

Copy .repos file 

Run the container 

./maliput_infrastructure/docker/run.sh

If you are using nvidia-docker2 add the --nvidia option.

./maliput_infrastructure/docker/run.sh --nvidia

Note

run.sh --help for more options:

-i --image_name Name of the image to be run (default maliput_ws_ubuntu_focal).

-c --container_name Name of the container (default maliput_ws_focal).

-w --workspace Relative or absolute path to the workspace you want to bind (default to location of maliput_infrastructure folder).

Install dependencies 

During docker build stage a script is copied into the container at /home/$USER/.

sudo ./../install_dependencies.sh

Update all the repositories in your workspace 

Install all packages’ dependencies 

Install drake 

Source ROS environment 

Staging changes in your container 

Once you finish your setup and tried the workspace, you might want to stage it. You can achieve that by exit-ing the container and accepting to commit the changes.

user@a3b6a70d7b7d:~/maliput_ws$ exit
exit
access control enabled, only authorized clients can connect
Do you want to overwrite the image called 'maliput_ws_ubuntu' with the current changes? [y/n]: y
Overwriting docker image...
[sudo] password for user:
sha256:9fdf391051f702f6b3fcd9c7ab258e5e014361bf18918b86155db3acda355147

Check your workspace 

Workspace state as a whole encompasses both current local repositories’ state plus the state of the filesystem that hosts it. However, if a workspace is containerized and not customized, repositories alone carry the source code and state the list of system dependencies necessary to build and execute. And we can easily inspect repositories.

To check repositories’ status, run:

vcs status src

To see changes in the repositories’ working tree, run:

vcs diff src

To see if (most of) our versioned packages’ dependencies have been met, run:

rosdep check --rosdistro $ROS_DISTRO --skip-keys "pybind11" --from-paths src

Note: not all workspace prerequisites are handled using rosdep meaning rosdep check may fall short. For example, pure binary dependencies like drake‘s binary tarball is not handled by rosdep. Another example is apt source lists.

In any given case, one can always resort to the specific tool used for repository versioning (e.g. git) if vcs isn’t enough or to the specific package managers (e.g. apt or pip) if rosdep isn’t enough.

Build your workspace 

Build 

Change the directory to maliput_ws:

cd ~/maliput_ws

To build all packages:

colcon build

To build some packages, use --packages-up-to. For example, to build maliput and maliput_malidrive:

colcon build --packages-up-to maliput maliput_malidrive

To build some packages and only those packages (i.e. without their dependencies), use --packages-select:

colcon build --packages-select maliput maliput_malidrive

Note that if dependencies cannot be met, regardless of whether it’s because they are not installed or not built, the build will fail. Thus, this flag is usually helpful only to quickly rebuild a package after building it along with its dependencies.

Note

If you are building drake from source as well, make sure --cmake-args -DWITH_PYTHON_VERSION=3 is passed to colcon. Otherwise, python packages and scripts in delphyne and delphyne_gui packages won’t find pydrake.

Note

To build with debug symbols, and given that we use CMake packages only, just make sure that CMAKE_BUILD_TYPE=Debug. You can force it by passing --cmake-args -DCMAKE_BUILD_TYPE=Debug to colcon.

Note

If you want to build with clang-8, run the following:

LDFLAGS="-fuse-ld=lld-8" CC=clang-8 CXX=clang++-8 colcon build --packages-up-to maliput maliput_malidrive

Source the workspace 

source install/setup.bash

Note

If delphyne is available, run delphyne-gazoo and delphyne-mali to see if everything is working.

Note

See colcon build documentation for further reference on build support.

Test your workspace 

In a built workspace, run:

colcon test --event-handlers=console_direct+ --return-code-on-test-failure --packages-skip pybind11

Note

See colcon test documentation for further reference on test support.

Build your workspace using Static Analyzer 

To verify your code, run the Clang Static Analyzer. A useful script called run_scan_build is located in .github in every repository.

The script will forward arguments to colcon build so you can use Colcon’s CLI machinery to choose which packages to evaluate.

To run scan-build on all packages in the workspace:

./src/maliput/.github/run_scan_build

To run scan-build up to maliput_malidrive:

./src/maliput/.github/run_scan_build --packages-up-to maliput_malidrive

Build doxygen documentation 

Build the workspace. In particular, we are interested in compiling dsim_docs_bundler.

cd ~/maliput_ws
colcon build --packages-up-to dsim_docs_bundler

Open the documentation with your favorite browser. If Google Chrome is available, you can run:

google-chrome install/dsim-docs-bundler/share/dsim-docs-bundler/doc/dsim-docs/html/index.html

Delete your workspace 

Containerized workspace could be deleted simply deleting the docker image:

docker rmi maliput_ws_ubuntu_focal

Consider replacing maliput_ws_ubuntu_focal by your image name when using a custom one.

Contributing 

Usual workflow 

Ours is similar to ROS2’s development workflow, and thus many of their tools and practices apply equally.

Workspaces are managed via vcs , a tool that helps in dealing with sources distributed across multiple repositories, not necessarily versioned with the same tool (support for git, hg, svn and bazaar is readily available). vcs uses .repos files for a listing of version pinned sources.

Dependency management is taken care of by rosdep, a tool that can crawl package.xml files and resolve dependencies into a call to the appropriate package manager for the current platform by means of a public database known as rosdistro.

To build and test packages, colcon abstracts away the details of the specific build system and testing tools in use and arbitrates these operations to take place in topological order. Operations will be run in parallel by default.

Note

In all three cases above, the tools delegate the actual work to the right tool for each package and focus instead on bridging the gap between them. Thus, for instance, colcon builds interdependent CMake packages by running cmake and make in the right order and setting up the environment for the artifacts to be available. Same applies for vcs and rosdep.

Note

These tools do not strive to act like a proxy for every configuration setting or command line option that underlying tools they delegate work to may have. Thus, it may be necessary to configure the underlying tool in addition to the configuration for these tools to attain a desired behavior. For instance, limiting colcon parallelism with the --parallel-workers switch has no impact on make parallelization settings if this tool is being used.

Using binary underlays 

In ROS 2 workspace parlance, an overlay workspace is a workspace that builds on top of another, previously built workspace i.e. the underlay workspace. A binary underlay is thus the install space of a pre-built workspace, that packages in downstream workspaces can use to meet their dependencies. As a result, the amount of code that needs to be compiled when building downstream workspaces gets reduced, enabling faster builds. You may refer to colcon documentation and tutorials for further details.

Several binary underlays are available for download and installation:

dsim-desktop-YYYYMMDD-focal-tar.gz

Built nightly, targeting Ubuntu Focal 20.04 LTS. Contains all known packages in all our repositories as of the specified date (DD/MM/YYYY). To be found at s3://driving-sim/projects/maliput/packages/nightlies/.
dsim-desktop-latest-focal.tar.gz

Built nightly, targeting Ubuntu Focal 20.04 LTS. Contains the most recent versions of all packages known in all our repositories. To be found at s3://driving-sim/projects/maliput/packages/nightlies/.

In the following, it is assumed that you want to use a full dsim-desktop underlay for working on a downstream package of your own. As such, it suggests the installation of a dsim-desktop binary underlay, that brings all known packages in all our repositories. You should choose an underlay that is appropriate for your intended purpose.

Download the binary underlay tarball of choice from dsim’s S3 bucket 

aws s3 cp s3://driving-sim/projects/maliput/packages/nightlies/dsim-desktop-latest-focal.tar.gz \
    /path/to/workspace/dsim-desktop-latest-focal.tar.gz

It is assumed that you have the right AWS credentials configured in your system. See AWS CLI user guide to configuration for further reference.

Extract binary underlay tarball 

sudo mkdir -p /opt/dsim-desktop
sudo tar -zxvf dsim-desktop-latest-focal.tar.gz -C /opt/dsim-desktop --strip 1

Install prerequisites 

echo "deb http://packages.ros.org/ros2/ubuntu $(lsb_release -cs) main" | \
    sudo tee --append /etc/apt/sources.list.d/ros2-latest.list

sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

sudo apt update
sudo apt install -y python3-rosdep
sudo rosdep init

Install all underlay packages’ dependencies 

For foxy

export ROS_DISTRO=foxy
rosdep update --include-eol-distros
rosdep install -i -y --rosdistro $ROS_DISTRO --skip-keys "pybind11" --from-paths /opt/dsim-desktop/*

Install drake 

cd /opt/dsim-desktop
./drake_vendor/bin/drake_installer -f drake_vendor/share/VERSION.TXT

From then on, before building the workspace, you must source the underlay as follows:

source /opt/dsim-desktop/setup.bash

Note

Having an underlay around does not make it a requirement for all workspace builds, but only for those that rely on that underlay to get their dependencies met.

How to use CI 

CI jobs build and test relevant packages for each repository on every PR. Being a multi-repository project, patches that are not limited to a single repository must be separately PR’d but built and tested together. To that end, make sure that all PR’d branches that are part of the same patch have the same name e.g. my_github_user/my_patch_name.

Warning

Fork based development is currently not supported. All PRs must come from origin and not a fork.

Troubleshooting 

Issue Forensics 

When reproducing issues, either related to the codebase or to the infrastructure that supports it, recreating the environment in which these issues arose is crucial.