Machine Design

Recommended: Read From Source to Success first!

Introduction

Hark (the language) compiles into bytecode that runs on a virtual machine designed for concurrency and portability.

Concurrency: When you do y = async f(x), f(x) is started in a new thread (Lambda instance, on AWS). And then when you do await y, the current thread halts, and automatically continues when y is finished being computed.

Portability: Two implementations of this VM exist so far—local and AWS Lambda, but there’s no reason Hark couldn’t run on top of (for example) Kubernetes (See Issue #8).

Abstractions

At its core, the VM has two abstractions:

  • Storage: what does it mean to store/retrieve a value? For example, this is done with DynamoDB in AWS.

  • Invocation: what does it mean to “invoke” a function? In AWS, this is currently done with Lambda.

These are both run-time abstractions, but which could be guided by compile-time source-code annotations. For example, if some functions have very high memory requirements, the VM can happily invoke them on an appropriate instance, while invoking other functions in a smaller instance.

Implement both of these, and you can run Hark.

Here’s the current implementation landscape.

StorageInvocationPlatformNotes
In-memorythreadingLocalPython threads are not actually concurrent!
DynamoDBthreadingLocalOnly useful for testing the DynamoDB storage.
DynamoDBmultiprocessingLocalConcurrent. Use DynamoDB Local.
DynamoDBLambdaAWSConcurrent. Limitations on DynamoDB item size!