Always use virtualenv
for any kind of deployment (testing, building,
production, … everything). This ensures there will be no dependency issues:
deployment collisions, missing dependencies in releases, etc. See the
virtualenv site for details.
Build environment
Important
We primarily support Ubuntu operating system. The following instruction steps were tested on Ubuntu 20.04 version.
There are only a few system-wide packages needed:
sudo apt update && \
sudo apt install -y python3-pip python3-dev virtualenv redis-server libssl-dev
Git submodules can be used to clone and manage all repositories at once:
git clone https://github.com/occopus/master.git github-occo --recursive
cd github-occo
git submodule foreach git checkout devel
Most scripts included in these components rely on this exact directory structure (especially testing and documentation dependencies).
There is a Vagrantfile to bootstrap the Occopus environment. After checkout just
simply execute vagrant up
and the virtual machine (created by VirtualBox) should be
correctly set up on your machine.
One should work on an Occopus component in a virtualenv. The following shows how
to setup the api
repo. By doing this the occopus-
commands will appear
and work correctly.
cd github-occopus
cd api
./reset-env.sh
source env/occopus/bin/activate
To try the occopus-
commands, go to the Tutorial section of the Users’ Guide
and follow the instructions. There you will find examples prepared for different
cloud backends and you can have proper configuration very fast. Users’
Guide can be found at the Occopus Website. Alternatively, you can go to the
docs
reporisory and find examples under the tutorial
directory.
Virtualenvs should be placed in the env/
directory, so they don’t linger in
the working tree. git
will ignore the contents of the env/
directory so
virtualenvs will not be commited accidentally.
Packaging and deployment
Occopus is split into several Python packages. The packages can be made available on the LPDS internal PyPI server (or package index) as Python wheels.
The internal PyPI server at the time of writing is on
192.168.155.11
. It is accessible through an Apache proxy using the
pip3.lpds.sztaki.hu
hostname.
Pip can use the following switches to use this package index:
pip --trusted-host pip3.lpds.sztaki.hu --find-links http://pip3.lpds.sztaki.hu/packages --no-index
The packages must be versioned according to the Semantic Versioning standard.
Development should be done using locally checked out Occopus packages instead of
using package dependencies. The requirements_test.txt
files rely on local
dependencies (pip install -e ...
) to encourage this. This is to avoid
uploading too many useless package versions to the package index.
In each repository there is a package.sh
which generates the wheels to be
published. upload.sh
will upload the output package to the pip3.lpds.sztaki.hu
,
provided the uploader has root access to it.
Managing the internal PyPI server
All dependencies can be found in this index. Future dependencies can be added to the index thus:
ssh ubuntu@192.168.155.11
cd /opt/packages/
pip download pymongo==2.8 # For example
This will download the new dependency from the community servers and installs (caches) it on the internal PyPI server. Locally mirroring and maintaining all used packages in an organization is a common practive anyway.
Dependency Manifests
There are three dependency manifests to be maintained in each package.
setup.py
Used by
pip
, this module contains package information, including dependencies.The dependencies declared here are abstract (versionless) dependencies, declaring only the relations among packages.
requirements.txt
Used for deployment, this text contains the real dependencies of the package, including version constraints.
This file will be used by the users of Occopus, so it must contain package names as references and no source information (cf.
requirements_test.txt
).This file should contain strict kinds of version specifications (
==
or possibly~>
), specifying the dependencies against which the package has been tested and verified.
requirements_test.txt
This file specifies the packages needed to test the package. This includes nosetests, and the current package itself (as a modifiable reference:
-e .
).Unlike
requirements.txt
, this file references other Occopus packages as local, modifiable repositories (e.g.-e ../util
). This helps the coding-testing cycle as modifications to other packages will be immediately “visible”, without reinstallation.This file contains the source of the packages (LPDS internal PyPI server) hard-coded.
This file must contain
==
type version specifications so the testing results are deterministic and reliable.
Creating Packages
The packages can be generated with the package.sh
script in each package’s
directory. This script creates and prepares an empty virtualenv and uses pip
wheel
to generate wheels. While building the new wheel, it gathers all its
dependencies too, so the resulting wheelhouse
directory will be a
self-contained set of packages that can be vendored. This script relies on
the internal PyPI server to gather the dependencies.
Vendoring Packages
The generated wheel packages can be uploaded to the internal PyPI server using
the upload.sh
script in each package’s directory. It uploads everything
found in the wheelhouse
directory generated by package.sh
. This is
redundant, as the dependencies already exist on the server, but this makes the
upload script dead simple.
When a package is uploaded, its version should be bumped unless it is otherwise justified.
Packages (in a topological order)
This is one possible topological ordering of the packages; i.e., they can be built/tested/deployed in this order.
Only interdependencies are annotated here, dependencies on external packages are omitted.
Depends |
– |
Repository |
|
Description |
Generic utility functions, configuration, communication,
etc. See:
occo.util . |
Testing |
The virtualenv must be bootstrapped by executing
occo_test/bootstrap_tests.sh . |
Depends |
OCCO-Util |
Repository |
|
Description |
Compiler module for OCCO. See:
occo.compiler . |
Depends |
OCCO-Util |
Repository |
|
Description |
Information broker for the OCCO system.
See:
occo.infobroker . |
Depends |
OCCO-Util, OCCO-Compiler, OCCO-InfoBroker |
Repository |
|
Description |
Active component of the OCCO infrastructure maintenance
system. See:
occo.enactor . |
Depends |
OCCO-Util, OCCO-InfoBroker |
Repository |
|
Description |
Central processor and synchronizer of the OCCO system. See:
occo.infraprocessor . |
Depends |
OCCO-Util, OCCO-InfoBroker |
Repository |
|
Description |
Backend component of the OCCO system, responsible
for handling specific kinds of resources. See
occo.resourcehandler . |
Depends |
OCCO-Util, OCCO-InfoBroker |
Repository |
|
Description |
Responsible for provisioning, setting up, configuring, etc.
the nodes instantiated by the resource handler.
|
Depends |
all OCCO packages |
Repository |
|
Description |
This package combines the primitives provided by other occo
packages into higher level services and features. This
package is intended to be the top-level package of the Occopus
system upon which use-cases, user interfaces can be built.
|