In the process of creating images, some of Joyent's internal-use SSH public keys were inadvertently left in certain published images. This led to the risk of potential unauthorized access to instances using the affected images.
Joyent acknowledges the assistance of an Open Source user in discovering this issue.
Joyent creates and publishes images to our Triton public cloud. These images are of various operating systems, to be used by customers in creating instances that run on the cloud.
We also make these images available to customers running Triton in their own datacenters, and to stand-alone SmartOS users.
An Open Source user reported to Joyent that certain images contained some unexpected public key data. Joyent investigated and confirmed that -- due to a new bug in the image-building system -- the following images were not properly stripped of SSH public keys used during the development process and still contain these keys in the /home/ubuntu/.ssh/authorized_keys file:
- ubuntu-certified-16.04-20190122 81c0ef69-e9d7-4e93-a15b-efd7ea9c9ee8
- ubuntu-certified-18.04-20190122 c9db249c-93ba-4507-9fa4-b4d0f81265fc
As part of the instance creation process, these keys would be propagated to the running instances that were using these images. These keys would make it possible for the Joyent developers (and Joyent development systems with the corresponding private key) to access these customer systems.
To the best of our knowledge, no such access was ever attempted in any systems using these images.
Actions Taken by Joyent
- The images listed above were disabled in our Triton public cloud, so that no further instances could be created from them.
- An audit was conducted to inventory all Triton public cloud instances created from the affected images, and all affected public cloud customers will be notified within hours of this advisory's first posting.
- As a good-faith measure, the applicable SSH keys of Joyent employees/systems were all rotated out.
Actions You Need to Take
Triton Enterprise Software Users
and Triton Public Cloud Users
The keys left in the image have the following fingerprints:
If you have any non-deleted instance(s) created from the images listed in the "Description" part of the Overvew above (running in the Triton public cloud, or in your on-premises datacenter), proceed as follows:
- Check your instances to ensure that any user's .ssh/authorized_keys file contains all of the keys for which you control the matching private key, and only those keys. If an instance's file contains any other keys with fingerprints matching the above list, it may have been affected.
- Affected instances should be destroyed and recreated using unaffected images, to ensure that unauthorized Joyent SSH keys are not present. Note: Before destroying an instance, back up any data you wish to retain.
Open Source Triton Users
Follow the instructions shown for software and public cloud users above.
Direct any further questions to The SmartOS Community Mailing Lists and IRC.
If you are a Joyent customer and have any further questions or concerns after reading the information provided above, please contact Joyent Support.
As noted above, if you are a Open Source Triton user, please direct any further questions to the SmartOS Community Mailing Lists and IRC.