YARP  2.3.70.2
Yet Another Robot Platform
YARP's own build structure
Author
Paul Fitzpatrick

For the reference of YARP developers, and the curious, here is a sketch of how YARP builds, including plugins.

The YARP project is profoundly grateful to the creators of CMake, for finally giving us a sane way to build a cross-platform library.

Suggested ways to use this document:

  • Look at src/CMakeLists.txt to pick out the general area you are interested in (e.g. devices/carriers).
  • Or find a specific file or directory of interest using the index.

Index

The big picture

Before getting into the details, here are a few general notes on YARP's source code and build:

  • YARP is often used without installation. It can be used either directly from its build directory ("in-tree use"), or via an install.
  • Although not required, YARP is designed with out-of-source builds in mind. The description from here on assumes you are doing that.
  • Each YARP library has its own set of header files, kept separate in the source but laid out so that they can be folded together after installation. By convention, YARP header files are accessed from C++ as "#include <yarp/os/...>", "#include <yarp/sig/...>" etc. The header file locations are chosen to allow the difference between in-tree and installed operation to be hidden. For in-tree use, we have an include path of src/libYARP_OS/include, src/libYARP_sig/include, ... For installed use, the include path collapses to just one directory, [prefix]/include. This is the reason for the apparent redundancy in header file paths, e.g. src/libYARP_OS/include/yarp/os/Port.h - it seems strange to have to include "yarp" and "os" in when compiling "libYARP_OS", but in the big picture it makes sense.
  • Header files in "impl" directories are not intended for use outside of YARP. They are accessible during in-tree use, but not after installation. If there's a good reason to install them, let us know though!
  • YARP uses ACE in its implementation, but does not expose that use in any public header files (anything not in an "impl" directory). Doing so causes all sorts of arcane problems. YARP developers should take care to maintain this constraint.
  • YARP uses no standard C++ libraries in its core libraries. STL is not used, nor is fstream/iostream/... Nor are exceptions used. YARP developers should take care to maintain this constraint. (ACE in turn can be compiled under the same constraints).
  • Users of YARP are of course free to use the STL, exceptions, and all the goodies. In fact, they should. That's a separate issue from what a library that aims at being portable (especially in binary form) should do. YARP executables are also free to use the STL etc.
  • Documentation is produced using doxygen. Extended tutorials like this one are to be found in src/[filename].dox, where the filename should closely match the name before .html you see in your browsers.

Where it all starts, the top-level CMakeLists.txt

The main CMakeLists.txt file for YARP is simple. It does the following:

  • Sets the CMAKE_MODULE_PATH to pick up scripts in the "cmake" directory.
  • Initializes some empty lists: YARP_TREE_INCLUDE_DIRS, YARP_LIBS, YARP_DEFS. These will be used to keep track of paths to header files, names of library targets, and extra compiler definitions needed.
  • Includes the cmake/YarpOptions.cmake file, to define general configuration options.
  • Includes the cmake/YarpVersion.cmake file, to define the current YARP version.
  • Includes the cmake/YarpSystemCheck.cmake file, to get properties of the system YARP is being compiled on.
  • Enables the CMake "test" target.
  • Enters the cmake directory for some preliminary code generation.
  • Enters the src directory to set up compilation targets.
  • Includes the cmake/YarpDescribe.cmake file, to store information about how YARP is configured. This is important to make YARP easier to use from external projects.
  • Includes the cmake/YarpPackage.cmake file, which sets up targets for packaging YARP as a tarball, zip, etc.
  • Includes the cmake/YarpDoc.cmake file, which sets up a target for generating YARP's documentation.

And we're done!

cmake/YarpOptions.cmake

First we set up:

  • The CMake LIBRARY_OUTPUT_PATH variable (set to the "lib" directory).
  • The CMake EXECUTABLE_OUTPUT_PATH variable (set to the "bin" directory).
  • The CMake CMAKE_BUILD_TYPE variable (debug, release, etc).

[ADVANCED] Then we check if the "static_libs" directory exists. If so, any static libraries within it are unconditionally linked. This is handy for building stand-alone binaries that link everything they need. Done carefully, such binaries can be used on different machines in the same general OS family.

Now we prepare some options for the user. Some may be hidden as "advanced" options.

  • The CREATE_SHARED_LIBRARY option - if set to true, shared libraries (.so/.dll/.dylib/...) will be built, instead of static.
  • [ADVANCED] The USE_STL_STRING option - internally, YARP uses an ACE string class. This can be toggled to std::string if desired.
  • The ENABLE_DASHBOARD_SUBMIT flag - if enabled, builds and tests can be submitted to the YARP online dashboard.

cmake/YarpVersion.cmake

This is a very simple file, defining a version number for YARP. YARP versioning is as follows:

  • YARP_VERSION_MAJOR - has only changed once or twice, ever.
  • YARP_VERSION_MINOR - changes perhaps once every year or two.
  • YARP_VERSION_PATCH - changes with every release.
  • YARP_VERSION_TWEAK - Used for bugfix releases, and in development branches to pinpoint new features.
  • YARP_VERSION_SHORT - The short version number (MAJOR.MINOR.PATCH.TWEAK)
  • YARP_VERSION - The full version number, including git commit and dirty state when yarp is built from a git clone, otherwise same as YARP_VERSION_SHORT.

cmake/YarpSystemCheck.cmake

Here we probe for some system properties needed to compile YARP correctly. We set up compiler flags reflecting those properties.

  • We set YARP_BIG_ENDIAN/YARP_LITTLE_ENDIAN depending on the order of bytes in integers. For example, if the number 42, expressed as a four byte integer is "42 0 0 0", then YARP_LITTLE_ENDIAN is set. If instead it is "0 0 0 42", then YARP_BIG_ENDIAN is set. This will affect the implementation of the yarp::os::NetInt32 type. On little endian systems, NetInt32 maps straight onto native integers. On big endian systems, NetInt32 is a class that simulates little endian integers.
  • We figure out an appropriate native type for 16-bit integers, 32-bit integers, and 64-bit floating point numbers. The types are stored in YARP_INT16, YARP_INT32, and YARP_FLOAT64 respectively.
  • If the YARP_ADMIN environment flag is set, we turn on some more aggressive warnings and errors in the compiler. YARP developers should generally have this flag set if possible.
  • We define YARP_PRESENT for compilation. This definition is unused.
  • We define _REENTRANT, for thread-safe C library calls.
  • Compiler flags that may also be needed by users are recorded in a global property "YARP_DEFS" for later export.
  • If on Windows, compiling natively, we make sure WIN32 and _WINDOWS are set (ACE relies on them).
  • If on Windows, compiling under Cygwin, we define CYGWIN (ACE needs this).
  • If using MINGW and/or MSYS, we set appropriate flags.
  • We attempt to locate the ACE library using the cmake/FindACE.cmake script.
  • We check whether the type ACE_String_Base_Const::size_type exists. Older ACE versions didn't have this type, newer versions do. If it is present, it must be used, but there's no obvious way to check for it in code, so we need to explicitly test for it.
  • Under MSVC, we turn off a set of warnings that ACE triggers. The warnings are for deprecated functions that ACE provides wrappers for.
  • Under MSVC, we set CMAKE_DEBUG_POSTFIX to "d", to add "d" to library names in debug mode. This is just a convention.

cmake/CMakeLists.txt

Here we generate some header files to hold configuration options.

  • A subdirectory "generated_include" of the build directory is created. We add that directory to "YARP_TREE_INCLUDE_DIRS", which will become a list of the paths to all header files needed by YARP.
  • We instantate src/libYARP_conf/include/yarp/conf/options.h.in to hold general compile options.
  • We instantate src/libYARP_conf/include/yarp/conf/version.h.in. This holds a representation of the YARP version number.
  • We instantate src/libYARP_conf/include/yarp/conf/system.h.in. This holds some facts about the system upon which YARP is being built.
  • We copy many macros from the "cmake" directory in the source to a "cmake" directory in the build, since they may be useful for users (library find scripts etc).

src/CMakeLists.txt

This file is a simple list of subdirectories, for all the parts of YARP. At the time of writing, these are the following - libraries first:

  • libYARP_OS - this contains the basic YARP implementation of ports, threads, etc.
  • libYARP_sig - signal processing: images, sound, vectors, and the like. Intended mostly to be easy to connect with other libraries (e.g. OpenCV).
  • libYARP_math - math library.
  • libYARP_dev - device management.
  • devices - optional plugin devices (some under non-LGPL licenses).
  • carriers - optional plugin network protocols.
  • libYARP_init - a tiny wrapper that initializes all optional devices and carriers.
  • libYARP_name - a library to help in construction of name servers.
  • libyarpc - an experimental C API to YARP.
  • libyarpcxx - an experimental C++ interface to YARP that is a simple wrapper around libyarpc. These two experimental libraries together make it easier to build very portable binary libraries for YARP.

And then executables:

  • yarp - the main command line interface to YARP.
  • yarpserver - the yarpserver program, a YARP name server based on Sqlite.
  • yarprun - a tool for starting/stopping programs across the network.
  • yarpview - a viewer for YARP image streams.
  • yarphear - a way to listen to YARP audio streams.
  • yarpdev - the main command line interface for YARP devices.

And finally tests:

  • tests - this directory pulls in regression tests from throughout the build.

cmake/YarpDescribe.cmake

Our job here is to capture everything a project that uses YARP will need to know, and export that. We need to deal with two important cases: where YARP is used directly from its build directory (without an install), or where YARP is installed in system directories.

First we set up some variables:

  • We set YARP_INCLUDE_DIRS from the global property YARP_TREE_INCLUDE_DIRS. All libraries configured to build will have appended the internal paths they need here. These paths are important for using YARP without an install.
  • We set YARP_LIBRARIES from the global property YARP_LIBS. All libraries configured to build will have appended their target name here.
  • We set YARP_ALL_DEFINES from the global property YARP_DEFS. This has any special compiler flags needed by the configured build.
  • We set YARP_HAS_MATH_LIB to true if the math library was configured to build.

We now generate three files in the build directory:

  • YARPConfig.cmake - this is an important file for using YARP directly from its build directory without an install. If the YARP_DIR environment variable is set to the build directory, A CMake call to "FIND_PACKAGE(YARP)" will find this file and pull in everything it needs to know about YARP. The template for this file is cmake/template/YARPConfig.cmake.in. It stores the variables we set up above, tries to read YARPDependencies.cmake (see next file), and does some backwards-compatibility gymnastics.
  • YARPDependencies.cmake - this contains information about the dependencies of all YARP targets. It is generated by the CMake "EXPORT" command.
  • YARPConfigForInstall.cmake - this is an alternate version of YARPConfig.cmake, appropriate after an install. After installation, it would be placed into [prefix]/lib/YARP/YARPConfig.cmake. An equivalent of YARPDependencies.cmake would also be generated upon an installation, in [prefix]/lib/YARP/YARP.cmake. This is generated using the CMake "INSTALL(EXPORT ...)" command Again, these are locations that "FIND_PACKAGE(YARP)" can pick up automatically.

cmake/YarpPackage.cmake

CMake comes with a companion utility CPack for generating tarballs, zip files, Windows installers, etc. This file sets some simple variables (project name, version number, license, etc) and puts CPack to work. The line "INCLUDE(CPack)" creates new targets: package and package_source. They are ludicrously easy to use, just "make package" or "make package_source". It is a good idea to work from a clean build based on a clean export from svn before doing this.

cmake/YarpDoc.cmake

Here we check if the "doxygen" program is available, and if so we set up a documentation-generating target called "dox". This is used to build YARP's documentation.

src/libYARP_OS/CMakeLists.txt

This sets up the YARP_OS target, which is the core YARP library. Source code is in the src subdirectory, header files are in the include subdirectory.

Our include directory is appended to the YARP_TREE_INCLUDE_DIRS global property for reference by later targets.

We place everything in YARP_TREE_INCLUDE_DIRS in our include path. At this stage, that should be just our own include directory, and the directory in which generated header files lie (see cmake/CMakeLists.txt). We also add ACE's include path.

We go ahead and create the YARP_OS target, linking it against all ACE libraries (ACE itself and system dependencies - pthread, rt, etc, or whatever the FindACE script decided was needed).

We set up installation, and add the target name to the YARP_LIBS global property for later reference. We also set a property "INCLUDE_DIRS" of the YARP_OS target, to hold its include path for later reference.

src/libYARP_sig/CMakeLists.txt

This sets up the YARP_sig target, for signal processing. Source code is in the src subdirectory, header files are in the include subdirectory. Our include directory is appended to the YARP_TREE_INCLUDE_DIRS global property for reference by later targets.

We place everything in YARP_TREE_INCLUDE_DIRS in our include path. We also add ACE's include path, since we use ACE in our implementation.

We create the YARP_sig target, linking it against YARP_OS.

We set up installation, and add the target name to the YARP_LIBS global property for later reference. We also set a property "INCLUDE_DIRS" of the YARP_sig target, to hold its include path for later reference.

src/libYARP_math/CMakeLists.txt

We follow the pattern of src/libYARP_sig/CMakeLists.txt to create the YARP_math library, for mathematical operations. There are a few wrinkles:

  • YARP_math is optional. We create an option CREATE_LIB_MATH which defaults to OFF. If OFF, this target is skipped.
  • YARP_math depends on Eigen3.

src/libYARP_dev/CMakeLists.txt

We follow exactly the pattern of src/libYARP_sig/CMakeLists.txt to create the YARP_dev library, for device management. YARP_dev links against YARP_sig and YARP_OS.

src/libYARP_init/CMakeLists.txt

This is a tiny library. Its role is to make sure that all the optional parts of YARP get initialized in user code. To avoid circular library dependencies, which can cause mayhem in some configurations, it is important the YARP is initialized directly by the user. YARP_init solves this problem by defining yarp::os::Network::init and yarp::os::Network::fini at the last possible minute (and definitely not in libYARP_OS). All the other libraries in YARP use yarp::os::NetworkBase rather than yarp::os::Network - NetworkBase is Network without init and fini.

src/libYARP_name/CMakeLists.txt

We follow the pattern of src/libYARP_sig/CMakeLists.txt to create the YARP_name library.

src/devices/CMakeLists.txt

This is one of a pair of directories in YARP that contain optional plugins (the other is src/carriers). This directory is devoted to devices, while the carriers directory is devoted to network protocols. Plugins in YARP are handled using a set of macros defined in cmake/YarpPlugin.cmake. Thes macros allow us to create a library (in this case "yarpmod") containing a lot of optional components using code like this:

YARP_BEGIN_PLUGIN_LIBRARY(yarpmod)
  add_subdirectory(cuda)
  add_subdirectory(fakebot)
  ...
YARP_END_PLUGIN_LIBRARY(yarpmod)

The key thing to know is that the "yarpmod" library is a regular CMake library target, containing some automatically generated code for initialization, and it has been linked to any and all libraries created in the listed subdirectories. We are given a list of the linked libraries as ${yarpmod_LIBRARIES}. The generated code for this library and the plugins within it will be placed in a "generated_code" subdirectory of the build.

We add the yarpmod library and the ${yarpmod_LIBRARIES} list to the global property YARP_LIBS. This is a variable keeping track of all the libraries that are configurated to be compiled. It is needed by cmake/YarpDescribe.cmake.

We tell CMake that, upon installing, the yarpmod library should go to [prefix]/lib. We cannot tell it to install the ${yarpmod_LIBRARIES}, since CMake requires that installation should be set up within the CMake script that creates the target. YarpPlugin.cmake currently sets up the install (this may change soon).

There is some moderately complicated code in this file that looks for extra devices listed in an "ExternalModules.cmake" file. This is an old mechanism that continues to be supported but for which there is not much need any more.

src/carriers/CMakeLists.txt

This directory is for network protocol plugins. It works just like the src/devices directory, except that it makes the "yarpcar" library target rather than the "yarpmod" library target.

The yarpcar target links against YARP_OS, YARP_sig, YARP_dev, yarpmod, and their dependencies. At first glance this may seem excessive, and that YARP_OS would suffice. However, there are some important cases to consider:

  • We may want network protocols specialized for media. For example the optional "mjpeg_carrier" is intended to transport an image stream. So it is good to have YARP_sig available.
  • We may want access to special devices in order to implement new protocols. This hasn't happened yet, but YARP is ready!

cmake/YarpPlugin.cmake

Here we define macros for creating a "plugin" library that links to many optional components.

Here are the "user-facing" macros:

  • YARP_BEGIN_PLUGIN_LIBRARY(lib_name): this sets a flag called X_YARP_PLUGIN_MODE to true, which activates a change of behavior of calls to ADD_LIBRARY and FIND_PACKAGE (see below).
  • YARP_END_PLUGIN_LIBRARY(lib_name): this turns off X_YARP_PLUGIN_MODE, and sets up a library target called lib_name that links to all libraries created after the corresponding call to YARP_BEGIN_PLUGIN_LIBRARY. This target contains a small piece of generated code to activate all plugins configured to compile.
  • ADD_PLUGIN_LIBRARY_EXECUTABLE(exec_name lib_name): this creates a simple test program called exec_name for the plugin library lib_name.
  • YARP_PREPARE_DEVICE(dev_name TYPE DeviceClass INCLUDE DeviceClass.h WRAPPER wrapper_name): this generates a CMake flag to select compilation of a device called "dev_name". If that CMake flag is turned on, then this macro generates some code to initialize the named device. That code will be included in the next library created (see the ADD_LIBRARY override below).
  • YARP_PREPARE_CARRIER(carrier_name TYPE CarrierClass INCLUDE CarrierClass.h): this works exactly like YARP_PREPARE_DEVICE, except the template for the generated code is different.

For implementation we also override the following CMake macros:

  • FIND_PACKAGE - we intercept calls to FIND_PACKAGE(YARP), since these won't work before YARP is compiled. Such calls are bypassed, and always succeed. Otherwise, FIND_PACKAGE's behavior is entirely unchanged.
  • ADD_LIBRARY - we intercept all ADD_LIBRARY calls. If they are not import operations (which continue unchanged), then we add the name of the target to a global list, and call the regular ADD_LIBRARY implementation with any extra generated code appended to the list of source code. Generated code will be present if calls to YARP_PREPARE_DEVICE/YARP_PREPARE_CARRIER have been made. If compiling YARP, we set up installation of the library target.

Here is a list of small code templates used by YarpPlugin.cmake:

Within YARP, the devices and carriers libraries are initialized by libYARP_init.

The plugin macros can also be used outside of YARP to create a library of plugins. The behavior of those macros changes in the following respects:

  • Installation of targets - when used outside of YARP, it is up to the programmer to set up installation of targets. Which is good, because YARP probably wouldn't do it the way you'd like.
  • Calling FIND_PACKAGE(YARP) - within YARP, lines like this are simulated, because YARP does not exist yet. Outside of YARP, such calls to FIND_PACKAGE behave normally.
  • Library initialization - when used outside of YARP, it is up to the programmer to set up initialization of the plugin library.

To help with the last point, two macros are declared in yarp/os/Network.h:

// Make plugins in a library available for use
#define YARP_DECLARE_PLUGINS(name) extern "C" void add_ ## name ## _devices();
#define YARP_REGISTER_PLUGINS(name) add_ ## name ## _devices();

The programmer should use YARP_DECLARE_PLUGINS(<library_name>) to declare their library's initialization function, and then execute YARP_REGISTER_PLUGINS(<library_name>) to actually call that function. See yarp_build_structure_template_libh. Note that the name of the library initialization function may change in future versions of YARP, so it is best to use these macros.

src/libYARP_conf/include/yarp/conf/options.h.in

This is the template for a generated header file built during configuration. It is placed in <CMAKE_BINARY_DIR>/src/conf/libYARP_conf/include/yarp/conf/options.h, and should be accessed as "#include <yarp/conf/options.h>". It is intended to store options about what parts of YARP have been compiled. Right now, it just optionally defines YARP_HAS_MATH_LIB, since that is all anyone has needed so far.

See also
cmake/CMakeLists.txt
cmake/YarpOptions.cmake

src/libYARP_conf/include/yarp/conf/version.h.in

This is the template for a generated header file built during configuration. It is placed in <CMAKE_BINARY_DIR>/src/conf/libYARP_conf/include/yarp/conf/version.h, and should be accessed as "#include <yarp/conf/version.h>". It is intended to store the version number of YARP.

See also
cmake/CMakeLists.txt
cmake/YarpVersion.cmake

src/libYARP_conf/include/yarp/conf/system.h.in

This is the template for a generated header file built during configuration. It is placed in <CMAKE_BINARY_DIR>/src/conf/libYARP_conf/include/yarp/conf/system.h, and should be accessed as "#include <yarp/conf/system.h>". It is intended to store some facts about the system upon which YARP is built: the types YARP_INT32, YARP_INT16, YARP_FLOAT64, whether the system is big- or little-endian, and whether to use ACE strings or STL strings

See also
cmake/CMakeLists.txt
cmake/YarpSystemCheck.cmake

src/libYARP_conf/include/yarp/conf/api.h.in

This is the template for a generated header file built during configuration. It is placed in <CMAKE_BINARY_DIR>/src/conf/libYARP_conf/include/yarp/conf/api.h, and should be accessed as "#include <yarp/conf/api.h>". It holds macros for defining the public API to YARP, if YARP has been configured build as a shared library and the YARP_CLEAN_API flag has been turned on.

See also
cmake/CMakeLists.txt
cmake/YarpOptions.cmake

cmake/template/YARPConfig.cmake.in

This is a template for YARPConfig.cmake, an important file for users of YARP. It lets CMake automatically pull in the properties of YARP needed to build user code. YARPConfig.cmake is placed in the build directory. If YARP_DIR is pointed at this directory during configuration of a user project, then "FIND_PACKAGE(YARP)" will succeed, and the following variables are defined:

  • YARP_LIBRARIES
  • YARP_INCLUDE_DIRS
  • YARP_DEFINES
  • YARP_MODULE_PATH Moreover, it includes the CMake modules required to use YARP commands in your cmake files.
See also
cmake/YarpDescribe.cmake

cmake/template/yarp_plugin_carrier.cpp.in

A snippet of code to add a carrier plugin. We make an instance of the carrier class, and pass it to a global yarp::os::Carriers object. The code is contained in a function called add_FOO() where FOO is the name of the carrier.

It is important that instances of carriers don't use special resources before they are put to use, since YARP expects to be able to instantiate carriers freely for administrative purposes.

See also
cmake/template/yarp_plugin_device.cpp.in
cmake/YarpPlugin.cmake

cmake/template/yarp_plugin_device.cpp.in

A snippet of code to add a device plugin. We create a "factory object" for the the device and hand it to yarp::dev::Drivers::factory(). The code is contained in a function called add_FOO() where FOO is the name of the device.

See also
cmake/template/yarp_plugin_carrier.cpp.in
cmake/YarpPlugin.cmake

cmake/template/yarp_plugin_portmonitor.cpp.in

A snippet of code to add a portmonitor plugin.

cmake/template/yarp_plugin_yarpdev_main.cpp.in

A snippet of code to define the main entry point for a plugin library (the "yarpdev" in the name is an anachronism). It defines a function added add_FOO_devices (where FOO is the name of the plugin library). The function expands to a series of calls to add_NAME1(), addNAME2(), ... where NAME1, NAME2, ... are the names of configured plugins (devices or carriers).

See also
cmake/template/yarp_plugin_device.cpp.in
cmake/template/yarp_plugin_carrier.cpp.in
cmake/YarpPlugin.cmake

cmake/template/yarpdev_lib_main.cpp.in

This is a template for a small test program which wraps a plugin library with the equivalent of the "yarpdev" utility. It is used by ADD_PLUGIN_LIBRARY_EXECUTABLE.

See also
cmake/YarpPlugin.cmake

src/yarp/CMakeLists.txt

Here we set up the main command line interface to YARP (contained in yarp.cpp). We link against YARP_init, which in turn links against all the YARP libraries.

We also set up the yarpserver executable, the standard name server for YARP. Its source code is in yarpserver.cpp.

src/yarpserver/CMakeLists.txt

This is the home of yarpserver, the YARP name server. It uses an sqlite amalgamation.