Using python tools as middleware, we demonstrate a loosely coupled service-orientated reference architecture for distributed computing workflows, implemented as a series of Docker contained services coupled together by a Python RESTful API. These containers in the reference architecture use MongoDB, RabbitMQ, Django RESTful and Celery to build a loosely coupled and scale-out tool stack. This reference stack can be used to manage data, register computational worker nodes with defined tasks and store process metadata on tasks that are run by the infrastructure. Computations can be scaled across a series of worker nodes on bare-metal or virtualized environments. This architecture stack serves as the reference environment used by The University of Oklahoma Libraries Informatics team to automate workflows for digital objects. This distribute processing of scientific research codes, both simple and complex, to command-line, web and mobile interfaces.
- Restful API
- Database - Data Catalog
- Asynchronous task queue/job queue based on distributed message passing. Celery
- Web and Visualizations
- API http://www.django-rest-framework.org/
- Tutorial 1-7
- Git Software carpentry
- Cybercommons
- Python Pandas
- HTML and Javascript
- Visualization Integration
- Work and familierize with Visualization and Back End
- D3 training and example projects
- Work with ETAG data and determine 3 visuals
- Work and familierize with Web and Back End
- U.S. Congressional Hearings Search Engine
- Electronic Transponder Analysis Gateway (ETAG Data Portal)