Tarantella is an open-source, distributed Deep Learning framework built on top of TensorFlow 2, providing scalable Deep Neural Network training on CPU and GPU compute clusters.

Tarantella is easy-to-use, allows to re-use existing TensorFlow 2/Keras models, and does not require any knowledge of parallel computing.

Table of contents

Overview

• Why Tarantella?
  • Goals
  • Performance Results
• Distributed Data Parallel Training
  • The general idea
  • Distribution of mini-batches
  • Overlapping communication with computation
  • Tensor Fusion
  • Model initialization and loading
• Distributed Datasets
• Points to Consider
  • Global versus local batch size
  • Batch normalization layers
  • Managing individual devices

Getting started

• Installation
  • Installing dependencies
  • SSH key-based authentication
  • Building Tarantella from source
  • [Optional] Building Tarantella with Infiniband support
  • [Optional] Building and running tests
  • [Optional] Building documentation
• Quick Start
  • Code example: LeNet-5 on MNIST
  • Executing your model with tarantella
  • Save and load Tarantella models
  • Using distributed datasets
  • Callbacks
  • Important points
• Tutorials
  • Prerequisites
  • ResNet-50
  • Transformers
• Advanced Topics
  • GASPI ranks
  • Using local batch sizes
  • Setting Tensor Fusion threshold
  • Reproducibility
• Frequently Asked Questions (FAQ)

Community

• Bug Reports
• Feature Requests
• Contributing
• Contact
• License