Patch Note 2018-12-27

New features

• Superpowered metadata views
• The metadata tab in the Execution view got a couple new controls to allow you to adjust the scales of the charted data. However, that's not all...

• You can now also compare the metadata of multiple executions within a task with the new Metadata tab in the Task view!
  

• OpenStack Swift support

• OpenStack Swift data stores are now supported, just like S3, Azure Blob Storage, and Google Cloud Storage are!

• Since there are many ways in which your OpenStack administrators may have configured your installation, if you have problems configuring your Swift store, please don't hesitate to get in touch.

• URI resolution support for Azure, Google, and Swift

• You can now use "native" URIs to refer to files within configured private Azure, Google and Swift storages. Support for S3 was pre-existing, but for clarity's sake, the URI syntaxes are all listed below.

  • Azure: azure://{account}/{container}/{key}
  • Google Storage: gs://{bucket}/{key}
  • Swift: swift://{project}/{container}/{key}
  • S3: s3://{bucket}/{key}

• Wildcard URI support for S3, Azure, Google, and Swift

• Not only can you use the above URI syntax to refer to single files, but you can finally also use wildcards within execution/task input expressions!
  The wildcard syntax supported is familiar from UNIX shells:
  • * matches any string except / (e.g. s3://my-bucket/dataset/images/*.jpg)
  • ** matches any string, including /, so it traverses subdirectories
  • ? matches any single character except /
  • [a-z] matches any character within the given range.
• Matching is limited to at most 10,000 files, and wildcard expressions are expanded during execution creation time.

• If you need additional syntax we haven't thought of yet, we'd love to hear about it!

• Scaling fixes

• Faster scaling

• More aggressive downscaling in private clouds

• Minor polish

• Very long column names are now shown a little nicer in tables
• All parameter or metadata columns in the execution table can now be toggled on or off with a single click

• The event API (and log view) was given a nice speed boost.

• Our task runner learned some new tricks!

• The amount of available disk space on the machine your job is running on is now printed out.
• Uploads to Swift are now supported.
• Sensitive parameters are stripped out of input URLs.
• Subsequent runs using the same image start a little faster.

Bugfixes

• Git SSH URLs with custom port numbers now work as documented. Oops!
• Non-secret environment variables couldn't be turned secret. Now they can!
• Input downloads are retried up to five times.
• Out-of-memory problems within the task runner are a little less likely due to better housekeeping.