docker2aci-0.12.0/000077500000000000000000000000001273674375500136235ustar00rootroot00000000000000docker2aci-0.12.0/.gitignore000066400000000000000000000001451273674375500156130ustar00rootroot00000000000000*.swp tests/distribution *.aci bin gopath tests/rendered-test-* tests/rkt-uuid-test-* tests/*.docker docker2aci-0.12.0/CHANGELOG.md000066400000000000000000000056721273674375500154460ustar00rootroot00000000000000## v0.12.0 v0.12.0 introduces support for the Docker v2.2 image format and OCI image format. It also fixes a bug that prevented pulling by digest to work. - backend/repository2: don't ignore when there's an image digest ([#171](https://github.com/appc/docker2aci/pull/171)). - lib/repository2: add support for docker v2.2 and OCI ([#176](https://github.com/appc/docker2aci/pull/176)). ## v0.11.1 v0.11.1 is a bugfix release. - Fix parallel pull synchronisation ([#167](https://github.com/appc/docker2aci/pull/167), [#168](https://github.com/appc/docker2aci/pull/168)). ## v0.11.0 This release splits the `--insecure` flag in two, `--insecure-skip-verify` to skip TLS verification, and `--insecure-allow-http` to allow unencrypted connections when fetching images. It also includes a couple of bugfixes. - Add missing message to channel on successful layer download ([#161](https://github.com/appc/docker2aci/pull/161)). - Fix a panic when a layer being fetched encounters an error ([#162](https://github.com/appc/docker2aci/pull/162)). - Split `--insecure` flag in two ([#163](https://github.com/appc/docker2aci/pull/163)). ## v0.10.0 This release includes two major performance optimizations: parallel layer pull and parallel ACI compression. - Pull layers in parallel ([#158](https://github.com/appc/docker2aci/pull/158)). - Use a parallel compression library ([#157](https://github.com/appc/docker2aci/pull/157)). - Fix auth token parsing to handle services with spaces in their names ([#150](https://github.com/appc/docker2aci/pull/150)). ## v0.9.3 v0.9.3 is a minor bug fix release. - Use the default transport when doing HTTP requests ([#147](https://github.com/appc/docker2aci/pull/147)). We were using an empty transport which didn't pass on the proxy configuration. ## v0.9.2 v0.9.2 is a minor release with a bug fix and a cleanup over the previous one. - Use upstream docker functions to parse docker URLs and parse digest ([#140](https://github.com/appc/docker2aci/pull/140)). - Change docker entrypoint/cmd annotations to json ([#142](https://github.com/appc/docker2aci/pull/142)). ## v0.9.1 v0.9.1 is mainly a bugfix and cleanup release. - Remove redundant dependency fetching, we're vendoring them now ([#134](https://github.com/appc/docker2aci/pull/134)). - Export ParseDockerURL which is used by rkt ([#135](https://github.com/appc/docker2aci/pull/135)). - Export annotations so people can use them outside docker2aci ([#135](https://github.com/appc/docker2aci/pull/135)). - Refactor the library so internal functions are in the "internal" package ([#135](https://github.com/appc/docker2aci/pull/135)). - Document release process and add a bump-version script ([#137](https://github.com/appc/docker2aci/pull/137)). ## v0.9.0 v0.9.0 is the initial release of docker2aci. docker2aci converts to ACI Docker images from a remote repository or from a local file generated with "docker save". It supports v1 and v2 Docker registries, compression, and layer squashing. docker2aci-0.12.0/Documentation/000077500000000000000000000000001273674375500164345ustar00rootroot00000000000000docker2aci-0.12.0/Documentation/devel/000077500000000000000000000000001273674375500175335ustar00rootroot00000000000000docker2aci-0.12.0/Documentation/devel/release.md000066400000000000000000000063101273674375500214750ustar00rootroot00000000000000# docker2aci release guide How to perform a release of docker2aci. This guide is probably unnecessarily verbose, so improvements welcomed. Only parts of the procedure are automated; this is somewhat intentional (manual steps for sanity checking) but it can probably be further scripted, please help. The following example assumes we're going from version 0.9.0 (`v0.9.0`) to 0.9.1 (`v0.9.1`). Let's get started: - Start at the relevant milestone on GitHub (e.g. https://github.com/appc/docker2aci/milestones/v0.9.1): ensure all referenced issues are closed (or moved elsewhere, if they're not done). Close the milestone. - Branch from the latest master, make sure your git status is clean - Ensure the build is clean! - `git clean -ffdx && ./build.sh && ./tests/test.sh` should work - Integration tests on CI should be green - Update the [release notes](https://github.com/appc/docker2aci/blob/master/CHANGELOG.md). Try to capture most of the salient changes since the last release, but don't go into unnecessary detail (better to link/reference the documentation wherever possible). The docker2aci version is [hardcoded in the repository](https://github.com/appc/docker2aci/blob/master/lib/version.go#L19), so the first thing to do is bump it: - Run `scripts/bump-release v0.9.1`. This should generate two commits: a bump to the actual release (e.g. v0.9.1), and then a bump to the release+git (e.g. v0.9.1+git). The actual release version should only exist in a single commit! - Sanity check what the script did with `git diff HEAD^^` or similar. - If the script didn't work, yell at the author and/or fix it. It can almost certainly be improved. - File a PR and get a review from another [MAINTAINER](https://github.com/appc/docker2aci/blob/master/MAINTAINERS). This is useful to a) sanity check the diff, and b) be very explicit/public that a release is happening - Ensure the CI on the release PR is green! After merging and going back to master branch, we check out the release version and tag it: - `git checkout HEAD^` should work; sanity check lib/version.go (the `Version` variable) after doing this - Add a signed tag: `git tag -s v0.9.1`. - Build docker2aci - `sudo git clean -ffdx && ./build.sh` - Sanity check `bin/docker2aci -version` - Push the tag to GitHub: `git push --tags` Now we switch to the GitHub web UI to conduct the release: - https://github.com/appc/docker2aci/releases/new - For now, check "This is a pre-release" - Tag "v0.9.1", release title "v0.9.1" - Copy-paste the release notes you added earlier in [CHANGELOG.md](https://github.com/appc/docker2aci/blob/master/CHANGELOG.md) - You can also add a little more detail and polish to the release notes here if you wish, as it is more targeted towards users (vs the changelog being more for developers); use your best judgement and see previous releases on GH for examples. - Attach the release. This is a simple tarball: ``` export NAME="docker2aci-v0.9.1" mkdir $NAME cp bin/docker2aci $NAME/ sudo chown -R root:root $NAME/ tar czvf $NAME.tar.gz --numeric-owner $NAME/ ``` - Attach the release signature; your personal GPG is okay for now: ``` gpg --detach-sign $NAME.tar.gz ``` - Publish the release! - Clean your git tree: `sudo git clean -ffdx`. docker2aci-0.12.0/Godeps/000077500000000000000000000000001273674375500150445ustar00rootroot00000000000000docker2aci-0.12.0/Godeps/Godeps.json000066400000000000000000000064111273674375500171620ustar00rootroot00000000000000{ "ImportPath": "github.com/appc/docker2aci", "GoVersion": "go1.6", "GodepVersion": "v72", "Deps": [ { "ImportPath": "github.com/appc/spec/aci", "Comment": "v0.8.2", "Rev": "05d50e649f47c4f4bfad9bbf9a95bbde1b26c783" }, { "ImportPath": "github.com/appc/spec/pkg/acirenderer", "Comment": "v0.8.2", "Rev": "05d50e649f47c4f4bfad9bbf9a95bbde1b26c783" }, { "ImportPath": "github.com/appc/spec/pkg/device", "Comment": "v0.8.2", "Rev": "05d50e649f47c4f4bfad9bbf9a95bbde1b26c783" }, { "ImportPath": "github.com/appc/spec/pkg/tarheader", "Comment": "v0.8.2", "Rev": "05d50e649f47c4f4bfad9bbf9a95bbde1b26c783" }, { "ImportPath": "github.com/appc/spec/schema", "Comment": "v0.8.2", "Rev": "05d50e649f47c4f4bfad9bbf9a95bbde1b26c783" }, { "ImportPath": "github.com/appc/spec/schema/common", "Comment": "v0.8.2", "Rev": "05d50e649f47c4f4bfad9bbf9a95bbde1b26c783" }, { "ImportPath": "github.com/appc/spec/schema/types", "Comment": "v0.8.2", "Rev": "05d50e649f47c4f4bfad9bbf9a95bbde1b26c783" }, { "ImportPath": "github.com/coreos/go-semver/semver", "Rev": "6fe83ccda8fb9b7549c9ab4ba47f47858bc950aa" }, { "ImportPath": "github.com/coreos/ioprogress", "Rev": "e7fc03058804de5488baed8df5b89f3924b9ec9a" }, { "ImportPath": "github.com/coreos/pkg/progressutil", "Comment": "v2", "Rev": "7f080b6c11ac2d2347c3cd7521e810207ea1a041" }, { "ImportPath": "github.com/docker/distribution/digest", "Comment": "v2.3.0", "Rev": "099622876197e3b24d627a72053d1bcc8968076a" }, { "ImportPath": "github.com/docker/distribution/reference", "Comment": "v2.3.0", "Rev": "099622876197e3b24d627a72053d1bcc8968076a" }, { "ImportPath": "github.com/gogo/protobuf/proto", "Comment": "v0.2", "Rev": "4168943e65a2802828518e95310aeeed6d84c4e5" }, { "ImportPath": "github.com/klauspost/compress/flate", "Comment": "v1.0", "Rev": "006acde2c5d283d2f8b8aa03d8f0cd2891c680cf" }, { "ImportPath": "github.com/klauspost/cpuid", "Comment": "v1.0", "Rev": "09cded8978dc9e80714c4d85b0322337b0a1e5e0" }, { "ImportPath": "github.com/klauspost/crc32", "Comment": "v1.0", "Rev": "19b0b332c9e4516a6370a0456e6182c3b5036720" }, { "ImportPath": "github.com/klauspost/pgzip", "Comment": "v1.0", "Rev": "95e8170c5d4da28db9c64dfc9ec3138ea4466fd4" }, { "ImportPath": "github.com/spf13/pflag", "Rev": "08b1a584251b5b62f458943640fc8ebd4d50aaa5" }, { "ImportPath": "go4.org/errorutil", "Rev": "03efcb870d84809319ea509714dd6d19a1498483" }, { "ImportPath": "golang.org/x/crypto/ssh/terminal", "Rev": "a7ead6ddf06233883deca151dffaef2effbf498f" }, { "ImportPath": "gopkg.in/inf.v0", "Comment": "v0.9.0", "Rev": "3887ee99ecf07df5b447e9b00d9c0b2adaa9f3e4" }, { "ImportPath": "k8s.io/kubernetes/pkg/api/resource", "Comment": "v1.3.0-alpha.4-205-g9625926", "Rev": "9625926852e215caa35e278c2cd7926744532291" }, { "ImportPath": "k8s.io/kubernetes/pkg/conversion", "Comment": "v1.3.0-alpha.4-205-g9625926", "Rev": "9625926852e215caa35e278c2cd7926744532291" }, { "ImportPath": "k8s.io/kubernetes/third_party/forked/reflect", "Comment": "v1.3.0-alpha.4-205-g9625926", "Rev": "9625926852e215caa35e278c2cd7926744532291" } ] } docker2aci-0.12.0/Godeps/Readme000066400000000000000000000002101273674375500161550ustar00rootroot00000000000000This directory tree is generated automatically by godep. Please do not edit. See https://github.com/tools/godep for more information. docker2aci-0.12.0/Godeps/_workspace/000077500000000000000000000000001273674375500172015ustar00rootroot00000000000000docker2aci-0.12.0/Godeps/_workspace/.gitignore000066400000000000000000000000121273674375500211620ustar00rootroot00000000000000/pkg /bin docker2aci-0.12.0/LICENSE000066400000000000000000000260751273674375500146420ustar00rootroot00000000000000Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright {yyyy} {name of copyright owner} Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. docker2aci-0.12.0/MAINTAINERS000066400000000000000000000002141273674375500153150ustar00rootroot00000000000000Alban Crequy (@alban) Iago López Galeiras (@iaguis) Krzesimir Nowak (@krnowak) docker2aci-0.12.0/README.md000066400000000000000000000120231273674375500151000ustar00rootroot00000000000000# docker2aci - Convert docker images to ACI [![Build Status](https://semaphoreci.com/api/v1/projects/4472761c-2b88-41f2-b2de-bf0447a8a290/610597/badge.svg)](https://semaphoreci.com/appc/docker2aci) docker2aci is a small library and CLI binary that converts Docker images to [ACI][aci]. It takes as input either a file generated by "docker save" or a Docker registry URL. It gets all the layers of a Docker image and squashes them into an ACI image. Optionally, it can generate one ACI for each layer, setting the correct dependencies. All ACIs generated are compressed with gzip by default. Compression can be disabled by specifying `--compression=none`. ## Build git clone git://github.com/appc/docker2aci cd docker2aci ./build.sh ## Volumes Docker Volumes get converted to mountPoints in the [Image Manifest Schema][imageschema]. Since mountPoints need a name and Docker Volumes don't, docker2aci generates a name by appending the path to `volume-` replacing non-alphanumeric characters with dashes. That is, if a Volume has `/var/tmp` as path, the resulting mountPoint name will be `volume-var-tmp`. When the docker2aci CLI binary converts a Docker Volume to a mountPoint it will print its name, path and whether it is read-only or not. ## Ports Docker Ports get converted to ports in the [Image Manifest Schema][imageschema]. The resulting port name will be the port number and the protocol separated by a dash. For example: `6379-tcp`. ## CLI examples ``` $ docker2aci docker://busybox Downloading sha256:55dc925c23d: [==============================] 674 KB/674 KB Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Generated ACI(s): library-busybox-latest.aci $ actool --debug validate library-busybox-latest.aci library-busybox-latest.aci: valid app container image ``` ``` $ /docker2aci --nosquash docker://quay.io/coreos/etcd:latest Downloading sha256:f05e5379dcb: [==============================] 3.98 MB/3.98 MB Downloading sha256:af1897d2d32: [==============================] 3.5 MB/3.5 MB Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Converted ports: name: "2379-tcp", protocol: "tcp", port: 2379, count: 1, socketActivated: false name: "2380-tcp", protocol: "tcp", port: 2380, count: 1, socketActivated: false name: "4001-tcp", protocol: "tcp", port: 4001, count: 1, socketActivated: false name: "7001-tcp", protocol: "tcp", port: 7001, count: 1, socketActivated: false Generated ACI(s): coreos-etcd-d21dd9a5886270b7c2c379c02fc548e0696b139c43bb12fdb2d9b63409717485-latest-linux-amd64-3.aci coreos-etcd-620329641f386e62c7b0e0fa60a9acef100e71058124ddc7f1969557c72b2458-latest-linux-amd64-2.aci coreos-etcd-9cd3f08f7ccfaad24c73757a5b4f79601f2790726d6ccdd556a82e5c9c5ddbfa-latest-linux-amd64-1.aci coreos-etcd-9cd3f08f7ccfaad24c73757a5b4f79601f2790726d6ccdd556a82e5c9c5ddbfa-latest-linux-amd64-0.aci ``` ``` $ docker save -o ubuntu.docker ubuntu $ docker2aci ubuntu.docker Extracting 706766fe1019 Extracting a62a42e77c9c Extracting 2c014f14d3d9 Extracting b7cf8f0d9e82 Generated ACI(s): ubuntu-latest.aci $ actool --debug validate ubuntu-latest.aci ubuntu-latest.aci: valid app container image ``` ``` $ docker2aci docker://redis Downloading sha256:c666c10c893: [==============================] 37.2 MB/37.2 MB Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:d6f52360d0a: [==============================] 1.69 KB/1.69 KB Downloading sha256:8c3a687fd4c: [==============================] 5.93 MB/5.93 MB Downloading sha256:15554e0e598: [==============================] 109 KB/109 KB Downloading sha256:3286d490a29: [==============================] 611 KB/611 KB Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:a3d89b95a63: [==============================] 3.04 MB/3.04 MB Downloading sha256:1c4db557158: [==============================] 98 B/98 B Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:a1a961e320b: [==============================] 196 B/196 B Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B Converted volumes: name: "volume-data", path: "/data", readOnly: false Converted ports: name: "6379-tcp", protocol: "tcp", port: 6379, count: 1, socketActivated: false Generated ACI(s): library-redis-latest.aci $ actool --debug validate library-redis-latest.aci library-redis-latest.aci: valid app container image ``` [aci]: https://github.com/appc/spec/blob/master/SPEC.md#app-container-image [imageschema]: https://github.com/appc/spec/blob/master/spec/aci.md#image-manifest-schema docker2aci-0.12.0/build.sh000077500000000000000000000010011273674375500152510ustar00rootroot00000000000000#!/usr/bin/env bash set -e ORG_PATH="github.com/appc" REPO_PATH="${ORG_PATH}/docker2aci" VERSION=$(git describe --dirty) GLDFLAGS="-X github.com/appc/docker2aci/lib.Version=${VERSION}" if [ ! -h gopath/src/${REPO_PATH} ]; then mkdir -p gopath/src/${ORG_PATH} ln -s ../../../.. gopath/src/${REPO_PATH} || exit 255 fi export GOBIN=${PWD}/bin export GOPATH=${PWD}/gopath:${PWD}/Godeps/_workspace eval $(go env) echo "Building docker2aci..." go build -o $GOBIN/docker2aci -ldflags "${GLDFLAGS}" ${REPO_PATH}/ docker2aci-0.12.0/lib/000077500000000000000000000000001273674375500143715ustar00rootroot00000000000000docker2aci-0.12.0/lib/common/000077500000000000000000000000001273674375500156615ustar00rootroot00000000000000docker2aci-0.12.0/lib/common/common.go000066400000000000000000000033661273674375500175100ustar00rootroot00000000000000// Copyright 2016 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // Package common provides misc types and variables. package common import ( "github.com/appc/docker2aci/lib/internal/docker" "github.com/appc/docker2aci/lib/internal/types" ) type Compression int const ( NoCompression = iota GzipCompression ) type ParsedDockerURL types.ParsedDockerURL const ( AppcDockerRegistryURL = "appc.io/docker/registryurl" AppcDockerRepository = "appc.io/docker/repository" AppcDockerTag = "appc.io/docker/tag" AppcDockerImageID = "appc.io/docker/imageid" AppcDockerParentImageID = "appc.io/docker/parentimageid" AppcDockerEntrypoint = "appc.io/docker/entrypoint" AppcDockerCmd = "appc.io/docker/cmd" ) type ErrSeveralImages struct { Msg string Images []string } // InsecureConfig represents the different insecure options available type InsecureConfig struct { SkipVerify bool AllowHTTP bool } func (e *ErrSeveralImages) Error() string { return e.Msg } // ParseDockerURL takes a Docker URL and returns a ParsedDockerURL with its // index URL, image name, and tag. func ParseDockerURL(arg string) (*ParsedDockerURL, error) { p, err := docker.ParseDockerURL(arg) return (*ParsedDockerURL)(p), err } docker2aci-0.12.0/lib/conversion_store.go000066400000000000000000000060511273674375500203230ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package docker2aci import ( "crypto/sha512" "fmt" "hash" "io" "io/ioutil" "os" "github.com/appc/spec/aci" "github.com/appc/spec/schema" "github.com/appc/spec/schema/types" ) const ( hashPrefix = "sha512-" ) type aciInfo struct { path string key string ImageManifest *schema.ImageManifest } // conversionStore is an simple implementation of the acirenderer.ACIRegistry // interface. It stores the Docker layers converted to ACI so we can take // advantage of acirenderer to generate a squashed ACI Image. type conversionStore struct { acis map[string]*aciInfo } func newConversionStore() *conversionStore { return &conversionStore{acis: make(map[string]*aciInfo)} } func (ms *conversionStore) WriteACI(path string) (string, error) { f, err := os.Open(path) if err != nil { return "", err } defer f.Close() cr, err := aci.NewCompressedReader(f) if err != nil { return "", err } defer cr.Close() h := sha512.New() r := io.TeeReader(cr, h) // read the file so we can get the hash if _, err := io.Copy(ioutil.Discard, r); err != nil { return "", fmt.Errorf("error reading ACI: %v", err) } im, err := aci.ManifestFromImage(f) if err != nil { return "", err } key := ms.HashToKey(h) ms.acis[key] = &aciInfo{path: path, key: key, ImageManifest: im} return key, nil } func (ms *conversionStore) GetImageManifest(key string) (*schema.ImageManifest, error) { aci, ok := ms.acis[key] if !ok { return nil, fmt.Errorf("aci with key: %s not found", key) } return aci.ImageManifest, nil } func (ms *conversionStore) GetACI(name types.ACIdentifier, labels types.Labels) (string, error) { for _, aci := range ms.acis { // we implement this function to comply with the interface so don't // bother implementing a proper label check if aci.ImageManifest.Name.String() == name.String() { return aci.key, nil } } return "", fmt.Errorf("aci not found") } func (ms *conversionStore) ReadStream(key string) (io.ReadCloser, error) { img, ok := ms.acis[key] if !ok { return nil, fmt.Errorf("stream for key: %s not found", key) } f, err := os.Open(img.path) if err != nil { return nil, fmt.Errorf("error opening aci: %s", img.path) } tr, err := aci.NewCompressedReader(f) if err != nil { return nil, err } return tr, nil } func (ms *conversionStore) ResolveKey(key string) (string, error) { return key, nil } func (ms *conversionStore) HashToKey(h hash.Hash) string { s := h.Sum(nil) return fmt.Sprintf("%s%x", hashPrefix, s) } docker2aci-0.12.0/lib/docker2aci.go000066400000000000000000000326401273674375500167330ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // Package docker2aci implements a simple library for converting docker images to // App Container Images (ACIs). package docker2aci import ( "archive/tar" "fmt" "io" "io/ioutil" "os" "path/filepath" "strings" "github.com/appc/docker2aci/lib/common" "github.com/appc/docker2aci/lib/internal" "github.com/appc/docker2aci/lib/internal/backend/file" "github.com/appc/docker2aci/lib/internal/backend/repository" "github.com/appc/docker2aci/lib/internal/docker" "github.com/appc/docker2aci/lib/internal/tarball" "github.com/appc/docker2aci/lib/internal/types" "github.com/appc/docker2aci/lib/internal/util" "github.com/appc/docker2aci/pkg/log" "github.com/appc/spec/pkg/acirenderer" "github.com/appc/spec/schema" appctypes "github.com/appc/spec/schema/types" gzip "github.com/klauspost/pgzip" ) // CommonConfig represents the shared configuration options for converting // Docker images. type CommonConfig struct { Squash bool // squash the layers in one file OutputDir string // where to put the resulting ACI TmpDir string // directory to use for temporary files Compression common.Compression // which compression to use for the resulting file(s) } // RemoteConfig represents the remote repository specific configuration for // converting Docker images. type RemoteConfig struct { CommonConfig Username string // username to use if the image to convert needs authentication Password string // password to use if the image to convert needs authentication Insecure common.InsecureConfig // Insecure options } // FileConfig represents the saved file specific configuration for converting // Docker images. type FileConfig struct { CommonConfig DockerURL string // select an image if there are several images/tags in the file, Syntax: "{docker registry URL}/{image name}:{tag}" } // ConvertRemoteRepo generates ACI images from docker registry URLs. It takes // as input a dockerURL of the form: // // {registry URL}/{repository}:{reference[tag|digest]} // // It then gets all the layers of the requested image and converts each of // them to ACI. // It returns the list of generated ACI paths. func ConvertRemoteRepo(dockerURL string, config RemoteConfig) ([]string, error) { repositoryBackend := repository.NewRepositoryBackend(config.Username, config.Password, config.Insecure) return convertReal(repositoryBackend, dockerURL, config.Squash, config.OutputDir, config.TmpDir, config.Compression) } // ConvertSavedFile generates ACI images from a file generated with "docker // save". If there are several images/tags in the file, a particular image can // be chosen via FileConfig.DockerURL. // // It returns the list of generated ACI paths. func ConvertSavedFile(dockerSavedFile string, config FileConfig) ([]string, error) { f, err := os.Open(dockerSavedFile) if err != nil { return nil, fmt.Errorf("error opening file: %v", err) } defer f.Close() fileBackend := file.NewFileBackend(f) return convertReal(fileBackend, config.DockerURL, config.Squash, config.OutputDir, config.TmpDir, config.Compression) } // GetIndexName returns the docker index server from a docker URL. func GetIndexName(dockerURL string) string { index, _ := docker.SplitReposName(dockerURL) return index } // GetDockercfgAuth reads a ~/.dockercfg file and returns the username and password // of the given docker index server. func GetDockercfgAuth(indexServer string) (string, string, error) { return docker.GetAuthInfo(indexServer) } func convertReal(backend internal.Docker2ACIBackend, dockerURL string, squash bool, outputDir string, tmpDir string, compression common.Compression) ([]string, error) { log.Debug("Getting image info...") ancestry, parsedDockerURL, err := backend.GetImageInfo(dockerURL) if err != nil { return nil, err } layersOutputDir := outputDir if squash { layersOutputDir, err = ioutil.TempDir(tmpDir, "docker2aci-") if err != nil { return nil, fmt.Errorf("error creating dir: %v", err) } defer os.RemoveAll(layersOutputDir) } conversionStore := newConversionStore() // only compress individual layers if we're not squashing layerCompression := compression if squash { layerCompression = common.NoCompression } aciLayerPaths, aciManifests, err := backend.BuildACI(ancestry, parsedDockerURL, layersOutputDir, tmpDir, layerCompression) if err != nil { return nil, err } var images acirenderer.Images for i, aciLayerPath := range aciLayerPaths { key, err := conversionStore.WriteACI(aciLayerPath) if err != nil { return nil, fmt.Errorf("error inserting in the conversion store: %v", err) } images = append(images, acirenderer.Image{Im: aciManifests[i], Key: key, Level: uint16(len(aciLayerPaths) - 1 - i)}) } // acirenderer expects images in order from upper to base layer images = util.ReverseImages(images) if squash { squashedImagePath, err := squashLayers(images, conversionStore, *parsedDockerURL, outputDir, compression) if err != nil { return nil, fmt.Errorf("error squashing image: %v", err) } aciLayerPaths = []string{squashedImagePath} } return aciLayerPaths, nil } // squashLayers receives a list of ACI layer file names ordered from base image // to application image and squashes them into one ACI func squashLayers(images []acirenderer.Image, aciRegistry acirenderer.ACIRegistry, parsedDockerURL types.ParsedDockerURL, outputDir string, compression common.Compression) (path string, err error) { log.Debug("Squashing layers...") log.Debug("Rendering ACI...") renderedACI, err := acirenderer.GetRenderedACIFromList(images, aciRegistry) if err != nil { return "", fmt.Errorf("error rendering squashed image: %v", err) } manifests, err := getManifests(renderedACI, aciRegistry) if err != nil { return "", fmt.Errorf("error getting manifests: %v", err) } squashedFilename := getSquashedFilename(parsedDockerURL) squashedImagePath := filepath.Join(outputDir, squashedFilename) squashedTempFile, err := ioutil.TempFile(outputDir, "docker2aci-squashedFile-") if err != nil { return "", err } defer func() { if err == nil { err = squashedTempFile.Close() } else { // remove temp file on error // we ignore its error to not mask the real error os.Remove(squashedTempFile.Name()) } }() log.Debug("Writing squashed ACI...") if err := writeSquashedImage(squashedTempFile, renderedACI, aciRegistry, manifests, compression); err != nil { return "", fmt.Errorf("error writing squashed image: %v", err) } log.Debug("Validating squashed ACI...") if err := internal.ValidateACI(squashedTempFile.Name()); err != nil { return "", fmt.Errorf("error validating image: %v", err) } if err := os.Rename(squashedTempFile.Name(), squashedImagePath); err != nil { return "", err } log.Debug("ACI squashed!") return squashedImagePath, nil } func getSquashedFilename(parsedDockerURL types.ParsedDockerURL) string { squashedFilename := strings.Replace(parsedDockerURL.ImageName, "/", "-", -1) if parsedDockerURL.Tag != "" { squashedFilename += "-" + parsedDockerURL.Tag } squashedFilename += ".aci" return squashedFilename } func getManifests(renderedACI acirenderer.RenderedACI, aciRegistry acirenderer.ACIRegistry) ([]schema.ImageManifest, error) { var manifests []schema.ImageManifest for _, aci := range renderedACI { im, err := aciRegistry.GetImageManifest(aci.Key) if err != nil { return nil, err } manifests = append(manifests, *im) } return manifests, nil } func writeSquashedImage(outputFile *os.File, renderedACI acirenderer.RenderedACI, aciProvider acirenderer.ACIProvider, manifests []schema.ImageManifest, compression common.Compression) error { var tarWriterTarget io.WriteCloser = outputFile switch compression { case common.NoCompression: case common.GzipCompression: tarWriterTarget = gzip.NewWriter(outputFile) defer tarWriterTarget.Close() default: return fmt.Errorf("unexpected compression enum value: %d", compression) } outputWriter := tar.NewWriter(tarWriterTarget) defer outputWriter.Close() finalManifest := mergeManifests(manifests) if err := internal.WriteManifest(outputWriter, finalManifest); err != nil { return err } if err := internal.WriteRootfsDir(outputWriter); err != nil { return err } type hardLinkEntry struct { firstLinkCleanName string firstLinkHeader tar.Header keepOriginal bool walked bool } // map aciFileKey -> cleanTarget -> hardLinkEntry hardLinks := make(map[string]map[string]hardLinkEntry) // first pass: read all the entries and build the hardLinks map in memory // but don't write on disk for _, aciFile := range renderedACI { rs, err := aciProvider.ReadStream(aciFile.Key) if err != nil { return err } defer rs.Close() hardLinks[aciFile.Key] = map[string]hardLinkEntry{} squashWalker := func(t *tarball.TarFile) error { cleanName := filepath.Clean(t.Name()) // the rootfs and the squashed manifest are added separately if cleanName == "manifest" || cleanName == "rootfs" { return nil } _, keep := aciFile.FileMap[cleanName] if keep && t.Header.Typeflag == tar.TypeLink { cleanTarget := filepath.Clean(t.Linkname()) if _, ok := hardLinks[aciFile.Key][cleanTarget]; !ok { _, keepOriginal := aciFile.FileMap[cleanTarget] hardLinks[aciFile.Key][cleanTarget] = hardLinkEntry{cleanName, *t.Header, keepOriginal, false} } } return nil } tr := tar.NewReader(rs) if err := tarball.Walk(*tr, squashWalker); err != nil { return err } } // second pass: write on disk for _, aciFile := range renderedACI { rs, err := aciProvider.ReadStream(aciFile.Key) if err != nil { return err } defer rs.Close() squashWalker := func(t *tarball.TarFile) error { cleanName := filepath.Clean(t.Name()) // the rootfs and the squashed manifest are added separately if cleanName == "manifest" || cleanName == "rootfs" { return nil } _, keep := aciFile.FileMap[cleanName] if link, ok := hardLinks[aciFile.Key][cleanName]; ok { if keep != link.keepOriginal { return fmt.Errorf("logic error: should we keep file %q?", cleanName) } if keep { if err := outputWriter.WriteHeader(t.Header); err != nil { return fmt.Errorf("error writing header: %v", err) } if _, err := io.Copy(outputWriter, t.TarStream); err != nil { return fmt.Errorf("error copying file into the tar out: %v", err) } } else { // The current file does not remain but there is a hard link pointing to // it. Write the current file but with the filename of the first hard link // pointing to it. That first hard link will not be written later, see // variable "alreadyWritten". link.firstLinkHeader.Size = t.Header.Size link.firstLinkHeader.Typeflag = t.Header.Typeflag link.firstLinkHeader.Linkname = "" if err := outputWriter.WriteHeader(&link.firstLinkHeader); err != nil { return fmt.Errorf("error writing header: %v", err) } if _, err := io.Copy(outputWriter, t.TarStream); err != nil { return fmt.Errorf("error copying file into the tar out: %v", err) } } } else if keep { alreadyWritten := false if t.Header.Typeflag == tar.TypeLink { cleanTarget := filepath.Clean(t.Linkname()) if link, ok := hardLinks[aciFile.Key][cleanTarget]; ok { if !link.keepOriginal { if link.walked { t.Header.Linkname = link.firstLinkCleanName } else { alreadyWritten = true } } link.walked = true hardLinks[aciFile.Key][cleanTarget] = link } } if !alreadyWritten { if err := outputWriter.WriteHeader(t.Header); err != nil { return fmt.Errorf("error writing header: %v", err) } if _, err := io.Copy(outputWriter, t.TarStream); err != nil { return fmt.Errorf("error copying file into the tar out: %v", err) } } } return nil } tr := tar.NewReader(rs) if err := tarball.Walk(*tr, squashWalker); err != nil { return err } } return nil } func mergeManifests(manifests []schema.ImageManifest) schema.ImageManifest { // FIXME(iaguis) we take app layer's manifest as the final manifest for now manifest := manifests[0] manifest.Dependencies = nil layerIndex := -1 for i, l := range manifest.Labels { if l.Name.String() == "layer" { layerIndex = i } } if layerIndex != -1 { manifest.Labels = append(manifest.Labels[:layerIndex], manifest.Labels[layerIndex+1:]...) } nameWithoutLayerID := appctypes.MustACIdentifier(stripLayerID(manifest.Name.String())) manifest.Name = *nameWithoutLayerID // once the image is squashed, we don't need a pathWhitelist manifest.PathWhitelist = nil return manifest } // striplayerID strips the layer ID from an app name: // // myregistry.com/organization/app-name-85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281 // myregistry.com/organization/app-name func stripLayerID(layerName string) string { n := strings.LastIndex(layerName, "-") return layerName[:n] } docker2aci-0.12.0/lib/internal/000077500000000000000000000000001273674375500162055ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/backend/000077500000000000000000000000001273674375500175745ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/backend/file/000077500000000000000000000000001273674375500205135ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/backend/file/file.go000066400000000000000000000177571273674375500220020ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // Package file is an implementation of Docker2ACIBackend for files saved via // "docker save". // // Note: this package is an implementation detail and shouldn't be used outside // of docker2aci. package file import ( "archive/tar" "encoding/json" "fmt" "io" "io/ioutil" "os" "path" "path/filepath" "github.com/appc/docker2aci/lib/common" "github.com/appc/docker2aci/lib/internal" "github.com/appc/docker2aci/lib/internal/docker" "github.com/appc/docker2aci/lib/internal/tarball" "github.com/appc/docker2aci/lib/internal/types" "github.com/appc/docker2aci/pkg/log" "github.com/appc/spec/schema" ) type FileBackend struct { file *os.File } func NewFileBackend(file *os.File) *FileBackend { return &FileBackend{ file: file, } } func (lb *FileBackend) GetImageInfo(dockerURL string) ([]string, *types.ParsedDockerURL, error) { parsedDockerURL, err := docker.ParseDockerURL(dockerURL) if err != nil { // a missing Docker URL could mean that the file only contains one // image, so we ignore the error here, we'll handle it in getImageID } appImageID, parsedDockerURL, err := getImageID(lb.file, parsedDockerURL) if err != nil { return nil, nil, err } ancestry, err := getAncestry(lb.file, appImageID) if err != nil { return nil, nil, fmt.Errorf("error getting ancestry: %v", err) } return ancestry, parsedDockerURL, nil } func (lb *FileBackend) BuildACI(layerIDs []string, dockerURL *types.ParsedDockerURL, outputDir string, tmpBaseDir string, compression common.Compression) ([]string, []*schema.ImageManifest, error) { var aciLayerPaths []string var aciManifests []*schema.ImageManifest var curPwl []string for i := len(layerIDs) - 1; i >= 0; i-- { tmpDir, err := ioutil.TempDir(tmpBaseDir, "docker2aci-") if err != nil { return nil, nil, fmt.Errorf("error creating dir: %v", err) } defer os.RemoveAll(tmpDir) j, err := getJson(lb.file, layerIDs[i]) if err != nil { return nil, nil, fmt.Errorf("error getting image json: %v", err) } layerData := types.DockerImageData{} if err := json.Unmarshal(j, &layerData); err != nil { return nil, nil, fmt.Errorf("error unmarshaling layer data: %v", err) } tmpLayerPath := path.Join(tmpDir, layerIDs[i]) tmpLayerPath += ".tar" layerFile, err := extractEmbeddedLayer(lb.file, layerIDs[i], tmpLayerPath) if err != nil { return nil, nil, fmt.Errorf("error getting layer from file: %v", err) } defer layerFile.Close() log.Debug("Generating layer ACI...") aciPath, manifest, err := internal.GenerateACI(i, layerData, dockerURL, outputDir, layerFile, curPwl, compression) if err != nil { return nil, nil, fmt.Errorf("error generating ACI: %v", err) } aciLayerPaths = append(aciLayerPaths, aciPath) aciManifests = append(aciManifests, manifest) curPwl = manifest.PathWhitelist } return aciLayerPaths, aciManifests, nil } func getImageID(file *os.File, dockerURL *types.ParsedDockerURL) (string, *types.ParsedDockerURL, error) { type tags map[string]string type apps map[string]tags _, err := file.Seek(0, 0) if err != nil { return "", nil, fmt.Errorf("error seeking file: %v", err) } var imageID string var appName string reposWalker := func(t *tarball.TarFile) error { if filepath.Clean(t.Name()) == "repositories" { repob, err := ioutil.ReadAll(t.TarStream) if err != nil { return fmt.Errorf("error reading repositories file: %v", err) } var repositories apps if err := json.Unmarshal(repob, &repositories); err != nil { return fmt.Errorf("error unmarshaling repositories file") } if dockerURL == nil { n := len(repositories) switch { case n == 1: for key, _ := range repositories { appName = key } case n > 1: var appNames []string for key, _ := range repositories { appNames = append(appNames, key) } return &common.ErrSeveralImages{ Msg: "several images found", Images: appNames, } default: return fmt.Errorf("no images found") } } else { appName = dockerURL.ImageName } tag := "latest" if dockerURL != nil { tag = dockerURL.Tag } app, ok := repositories[appName] if !ok { return fmt.Errorf("app %q not found", appName) } _, ok = app[tag] if !ok { if len(app) == 1 { for key, _ := range app { tag = key } } else { return fmt.Errorf("tag %q not found", tag) } } if dockerURL == nil { dockerURL = &types.ParsedDockerURL{ IndexURL: "", Tag: tag, ImageName: appName, } } imageID = string(app[tag]) } return nil } tr := tar.NewReader(file) if err := tarball.Walk(*tr, reposWalker); err != nil { return "", nil, err } if imageID == "" { return "", nil, fmt.Errorf("repositories file not found") } return imageID, dockerURL, nil } func getJson(file *os.File, layerID string) ([]byte, error) { jsonPath := path.Join(layerID, "json") return getTarFileBytes(file, jsonPath) } func getTarFileBytes(file *os.File, path string) ([]byte, error) { _, err := file.Seek(0, 0) if err != nil { fmt.Errorf("error seeking file: %v", err) } var fileBytes []byte fileWalker := func(t *tarball.TarFile) error { if filepath.Clean(t.Name()) == path { fileBytes, err = ioutil.ReadAll(t.TarStream) if err != nil { return err } } return nil } tr := tar.NewReader(file) if err := tarball.Walk(*tr, fileWalker); err != nil { return nil, err } if fileBytes == nil { return nil, fmt.Errorf("file %q not found", path) } return fileBytes, nil } func extractEmbeddedLayer(file *os.File, layerID string, outputPath string) (*os.File, error) { log.Info("Extracting ", layerID[:12], "\n") _, err := file.Seek(0, 0) if err != nil { fmt.Errorf("error seeking file: %v", err) } layerTarPath := path.Join(layerID, "layer.tar") var layerFile *os.File fileWalker := func(t *tarball.TarFile) error { if filepath.Clean(t.Name()) == layerTarPath { layerFile, err = os.Create(outputPath) if err != nil { return fmt.Errorf("error creating layer: %v", err) } _, err = io.Copy(layerFile, t.TarStream) if err != nil { return fmt.Errorf("error getting layer: %v", err) } } return nil } tr := tar.NewReader(file) if err := tarball.Walk(*tr, fileWalker); err != nil { return nil, err } if layerFile == nil { return nil, fmt.Errorf("file %q not found", layerTarPath) } return layerFile, nil } func getAncestry(file *os.File, imgID string) ([]string, error) { var ancestry []string curImgID := imgID var err error for curImgID != "" { ancestry = append(ancestry, curImgID) curImgID, err = getParent(file, curImgID) if err != nil { return nil, err } } return ancestry, nil } func getParent(file *os.File, imgID string) (string, error) { var parent string _, err := file.Seek(0, 0) if err != nil { return "", fmt.Errorf("error seeking file: %v", err) } jsonPath := filepath.Join(imgID, "json") parentWalker := func(t *tarball.TarFile) error { if filepath.Clean(t.Name()) == jsonPath { jsonb, err := ioutil.ReadAll(t.TarStream) if err != nil { return fmt.Errorf("error reading layer json: %v", err) } var dockerData types.DockerImageData if err := json.Unmarshal(jsonb, &dockerData); err != nil { return fmt.Errorf("error unmarshaling layer data: %v", err) } parent = dockerData.Parent } return nil } tr := tar.NewReader(file) if err := tarball.Walk(*tr, parentWalker); err != nil { return "", err } return parent, nil } docker2aci-0.12.0/lib/internal/backend/repository/000077500000000000000000000000001273674375500220135ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/backend/repository/repository.go000066400000000000000000000117461273674375500245720ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // Package repository is an implementation of Docker2ACIBackend for Docker // remote registries. // // Note: this package is an implementation detail and shouldn't be used outside // of docker2aci. package repository import ( "fmt" "net/http" "path" "github.com/appc/docker2aci/lib/common" "github.com/appc/docker2aci/lib/internal/docker" "github.com/appc/docker2aci/lib/internal/types" "github.com/appc/docker2aci/lib/internal/typesV2" "github.com/appc/docker2aci/lib/internal/util" "github.com/appc/spec/schema" ) type registryVersion int const ( registryV1 registryVersion = iota registryV2 ) type RepositoryBackend struct { repoData *RepoData username string password string insecure common.InsecureConfig hostsV2Support map[string]bool hostsV2AuthTokens map[string]map[string]string schema string imageManifests map[types.ParsedDockerURL]v2Manifest imageV2Manifests map[types.ParsedDockerURL]*typesV2.ImageManifest imageConfigs map[types.ParsedDockerURL]*typesV2.ImageConfig } func NewRepositoryBackend(username string, password string, insecure common.InsecureConfig) *RepositoryBackend { return &RepositoryBackend{ username: username, password: password, insecure: insecure, hostsV2Support: make(map[string]bool), hostsV2AuthTokens: make(map[string]map[string]string), imageManifests: make(map[types.ParsedDockerURL]v2Manifest), imageV2Manifests: make(map[types.ParsedDockerURL]*typesV2.ImageManifest), imageConfigs: make(map[types.ParsedDockerURL]*typesV2.ImageConfig), } } func (rb *RepositoryBackend) GetImageInfo(url string) ([]string, *types.ParsedDockerURL, error) { dockerURL, err := docker.ParseDockerURL(url) if err != nil { return nil, nil, err } var supportsV2, ok bool var URLSchema string if supportsV2, ok = rb.hostsV2Support[dockerURL.IndexURL]; !ok { var err error URLSchema, supportsV2, err = rb.supportsRegistry(dockerURL.IndexURL, registryV2) if err != nil { return nil, nil, err } rb.schema = URLSchema + "://" rb.hostsV2Support[dockerURL.IndexURL] = supportsV2 } if supportsV2 { return rb.getImageInfoV2(dockerURL) } else { URLSchema, supportsV1, err := rb.supportsRegistry(dockerURL.IndexURL, registryV1) if err != nil { return nil, nil, err } if !supportsV1 { return nil, nil, fmt.Errorf("registry doesn't support API v2 nor v1") } rb.schema = URLSchema + "://" return rb.getImageInfoV1(dockerURL) } } func (rb *RepositoryBackend) BuildACI(layerIDs []string, dockerURL *types.ParsedDockerURL, outputDir string, tmpBaseDir string, compression common.Compression) ([]string, []*schema.ImageManifest, error) { if rb.hostsV2Support[dockerURL.IndexURL] { return rb.buildACIV2(layerIDs, dockerURL, outputDir, tmpBaseDir, compression) } else { return rb.buildACIV1(layerIDs, dockerURL, outputDir, tmpBaseDir, compression) } } func checkRegistryStatus(statusCode int, hdr http.Header, version registryVersion) (bool, error) { switch statusCode { case http.StatusOK, http.StatusUnauthorized: ok := true if version == registryV2 { // v2 API requires this check ok = hdr.Get("Docker-Distribution-API-Version") == "registry/2.0" } return ok, nil case http.StatusNotFound: return false, nil } return false, fmt.Errorf("unexpected http code: %d", statusCode) } func (rb *RepositoryBackend) supportsRegistry(indexURL string, version registryVersion) (schema string, ok bool, err error) { var URLPath string switch version { case registryV1: // the v1 API defines this URL to check if the registry's status URLPath = "v1/_ping" case registryV2: URLPath = "v2" } URLStr := path.Join(indexURL, URLPath) fetch := func(schema string) (res *http.Response, err error) { url := schema + "://" + URLStr req, err := http.NewRequest("GET", url, nil) if err != nil { return nil, err } rb.setBasicAuth(req) client := util.GetTLSClient(rb.insecure.SkipVerify) res, err = client.Do(req) return } schema = "https" res, err := fetch(schema) if err == nil { ok, err = checkRegistryStatus(res.StatusCode, res.Header, version) defer res.Body.Close() } if err != nil || !ok { if rb.insecure.AllowHTTP { schema = "http" res, err = fetch(schema) if err == nil { ok, err = checkRegistryStatus(res.StatusCode, res.Header, version) defer res.Body.Close() } } return schema, ok, err } return schema, ok, err } docker2aci-0.12.0/lib/internal/backend/repository/repository1.go000066400000000000000000000246151273674375500246520ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package repository import ( "encoding/json" "fmt" "io" "io/ioutil" "net/http" "os" "path" "strconv" "strings" "time" "github.com/appc/docker2aci/lib/common" "github.com/appc/docker2aci/lib/internal" "github.com/appc/docker2aci/lib/internal/types" "github.com/appc/docker2aci/lib/internal/util" "github.com/appc/docker2aci/pkg/log" "github.com/appc/spec/schema" "github.com/coreos/ioprogress" ) type RepoData struct { Tokens []string Endpoints []string Cookie []string } func (rb *RepositoryBackend) getImageInfoV1(dockerURL *types.ParsedDockerURL) ([]string, *types.ParsedDockerURL, error) { repoData, err := rb.getRepoDataV1(dockerURL.IndexURL, dockerURL.ImageName) if err != nil { return nil, nil, fmt.Errorf("error getting repository data: %v", err) } // TODO(iaguis) check more endpoints appImageID, err := rb.getImageIDFromTagV1(repoData.Endpoints[0], dockerURL.ImageName, dockerURL.Tag, repoData) if err != nil { return nil, nil, fmt.Errorf("error getting ImageID from tag %s: %v", dockerURL.Tag, err) } ancestry, err := rb.getAncestryV1(appImageID, repoData.Endpoints[0], repoData) if err != nil { return nil, nil, err } rb.repoData = repoData return ancestry, dockerURL, nil } func (rb *RepositoryBackend) buildACIV1(layerIDs []string, dockerURL *types.ParsedDockerURL, outputDir string, tmpBaseDir string, compression common.Compression) ([]string, []*schema.ImageManifest, error) { layerFiles := make([]*os.File, len(layerIDs)) layerDatas := make([]types.DockerImageData, len(layerIDs)) tmpParentDir, err := ioutil.TempDir(tmpBaseDir, "docker2aci-") if err != nil { return nil, nil, err } defer os.RemoveAll(tmpParentDir) var doneChannels []chan error for i, layerID := range layerIDs { doneChan := make(chan error) doneChannels = append(doneChannels, doneChan) // https://github.com/golang/go/wiki/CommonMistakes i := i // golang-- layerID := layerID go func() { tmpDir, err := ioutil.TempDir(tmpParentDir, "") if err != nil { doneChan <- fmt.Errorf("error creating dir: %v", err) return } j, size, err := rb.getJsonV1(layerID, rb.repoData.Endpoints[0], rb.repoData) if err != nil { doneChan <- fmt.Errorf("error getting image json: %v", err) return } layerDatas[i] = types.DockerImageData{} if err := json.Unmarshal(j, &layerDatas[i]); err != nil { doneChan <- fmt.Errorf("error unmarshaling layer data: %v", err) return } layerFiles[i], err = rb.getLayerV1(layerID, rb.repoData.Endpoints[0], rb.repoData, size, tmpDir) if err != nil { doneChan <- fmt.Errorf("error getting the remote layer: %v", err) return } doneChan <- nil }() } for _, doneChan := range doneChannels { err := <-doneChan if err != nil { return nil, nil, err } } var aciLayerPaths []string var aciManifests []*schema.ImageManifest var curPwl []string for i := len(layerIDs) - 1; i >= 0; i-- { log.Debug("Generating layer ACI...") aciPath, manifest, err := internal.GenerateACI(i, layerDatas[i], dockerURL, outputDir, layerFiles[i], curPwl, compression) if err != nil { return nil, nil, fmt.Errorf("error generating ACI: %v", err) } aciLayerPaths = append(aciLayerPaths, aciPath) aciManifests = append(aciManifests, manifest) curPwl = manifest.PathWhitelist layerFiles[i].Close() } return aciLayerPaths, aciManifests, nil } func (rb *RepositoryBackend) getRepoDataV1(indexURL string, remote string) (*RepoData, error) { client := util.GetTLSClient(rb.insecure.SkipVerify) repositoryURL := rb.schema + path.Join(indexURL, "v1", "repositories", remote, "images") req, err := http.NewRequest("GET", repositoryURL, nil) if err != nil { return nil, err } if rb.username != "" && rb.password != "" { req.SetBasicAuth(rb.username, rb.password) } req.Header.Set("X-Docker-Token", "true") res, err := client.Do(req) if err != nil { return nil, err } defer res.Body.Close() if res.StatusCode != 200 { return nil, fmt.Errorf("HTTP code: %d, URL: %s", res.StatusCode, req.URL) } var tokens []string if res.Header.Get("X-Docker-Token") != "" { tokens = res.Header["X-Docker-Token"] } var cookies []string if res.Header.Get("Set-Cookie") != "" { cookies = res.Header["Set-Cookie"] } var endpoints []string if res.Header.Get("X-Docker-Endpoints") != "" { endpoints = makeEndpointsListV1(res.Header["X-Docker-Endpoints"]) } else { // Assume same endpoint endpoints = append(endpoints, indexURL) } return &RepoData{ Endpoints: endpoints, Tokens: tokens, Cookie: cookies, }, nil } func (rb *RepositoryBackend) getImageIDFromTagV1(registry string, appName string, tag string, repoData *RepoData) (string, error) { client := util.GetTLSClient(rb.insecure.SkipVerify) // we get all the tags instead of directly getting the imageID of the // requested one (.../tags/TAG) because even though it's specified in the // Docker API, some registries (e.g. Google Container Registry) don't // implement it. req, err := http.NewRequest("GET", rb.schema+path.Join(registry, "repositories", appName, "tags"), nil) if err != nil { return "", fmt.Errorf("failed to get Image ID: %s, URL: %s", err, req.URL) } setAuthTokenV1(req, repoData.Tokens) setCookieV1(req, repoData.Cookie) res, err := client.Do(req) if err != nil { return "", fmt.Errorf("failed to get Image ID: %s, URL: %s", err, req.URL) } defer res.Body.Close() if res.StatusCode != 200 { return "", fmt.Errorf("HTTP code: %d. URL: %s", res.StatusCode, req.URL) } j, err := ioutil.ReadAll(res.Body) if err != nil { return "", err } var tags map[string]string if err := json.Unmarshal(j, &tags); err != nil { return "", fmt.Errorf("error unmarshaling: %v", err) } imageID, ok := tags[tag] if !ok { return "", fmt.Errorf("tag %s not found", tag) } return imageID, nil } func (rb *RepositoryBackend) getAncestryV1(imgID, registry string, repoData *RepoData) ([]string, error) { client := util.GetTLSClient(rb.insecure.SkipVerify) req, err := http.NewRequest("GET", rb.schema+path.Join(registry, "images", imgID, "ancestry"), nil) if err != nil { return nil, err } setAuthTokenV1(req, repoData.Tokens) setCookieV1(req, repoData.Cookie) res, err := client.Do(req) if err != nil { return nil, err } defer res.Body.Close() if res.StatusCode != 200 { return nil, fmt.Errorf("HTTP code: %d. URL: %s", res.StatusCode, req.URL) } var ancestry []string j, err := ioutil.ReadAll(res.Body) if err != nil { return nil, fmt.Errorf("Failed to read downloaded json: %s (%s)", err, j) } if err := json.Unmarshal(j, &ancestry); err != nil { return nil, fmt.Errorf("error unmarshaling: %v", err) } return ancestry, nil } func (rb *RepositoryBackend) getJsonV1(imgID, registry string, repoData *RepoData) ([]byte, int64, error) { client := util.GetTLSClient(rb.insecure.SkipVerify) req, err := http.NewRequest("GET", rb.schema+path.Join(registry, "images", imgID, "json"), nil) if err != nil { return nil, -1, err } setAuthTokenV1(req, repoData.Tokens) setCookieV1(req, repoData.Cookie) res, err := client.Do(req) if err != nil { return nil, -1, err } defer res.Body.Close() if res.StatusCode != 200 { return nil, -1, fmt.Errorf("HTTP code: %d, URL: %s", res.StatusCode, req.URL) } imageSize := int64(-1) if hdr := res.Header.Get("X-Docker-Size"); hdr != "" { imageSize, err = strconv.ParseInt(hdr, 10, 64) if err != nil { return nil, -1, err } } b, err := ioutil.ReadAll(res.Body) if err != nil { return nil, -1, fmt.Errorf("failed to read downloaded json: %v (%s)", err, b) } return b, imageSize, nil } func (rb *RepositoryBackend) getLayerV1(imgID, registry string, repoData *RepoData, imgSize int64, tmpDir string) (*os.File, error) { client := util.GetTLSClient(rb.insecure.SkipVerify) req, err := http.NewRequest("GET", rb.schema+path.Join(registry, "images", imgID, "layer"), nil) if err != nil { return nil, err } setAuthTokenV1(req, repoData.Tokens) setCookieV1(req, repoData.Cookie) res, err := client.Do(req) if err != nil { return nil, err } defer res.Body.Close() if res.StatusCode != 200 { res.Body.Close() return nil, fmt.Errorf("HTTP code: %d. URL: %s", res.StatusCode, req.URL) } // if we didn't receive the size via X-Docker-Size when we retrieved the // layer's json, try Content-Length if imgSize == -1 { if hdr := res.Header.Get("Content-Length"); hdr != "" { imgSize, err = strconv.ParseInt(hdr, 10, 64) if err != nil { return nil, err } } } prefix := "Downloading " + imgID[:12] fmtBytesSize := 18 barSize := int64(80 - len(prefix) - fmtBytesSize) bar := ioprogress.DrawTextFormatBarForW(barSize, os.Stderr) fmtfunc := func(progress, total int64) string { return fmt.Sprintf( "%s: %s %s", prefix, bar(progress, total), ioprogress.DrawTextFormatBytes(progress, total), ) } progressReader := &ioprogress.Reader{ Reader: res.Body, Size: imgSize, DrawFunc: ioprogress.DrawTerminalf(os.Stderr, fmtfunc), DrawInterval: 500 * time.Millisecond, } layerFile, err := ioutil.TempFile(tmpDir, "dockerlayer-") if err != nil { return nil, err } _, err = io.Copy(layerFile, progressReader) if err != nil { return nil, err } if err := layerFile.Sync(); err != nil { return nil, err } return layerFile, nil } func setAuthTokenV1(req *http.Request, token []string) { if req.Header.Get("Authorization") == "" { req.Header.Set("Authorization", "Token "+strings.Join(token, ",")) } } func setCookieV1(req *http.Request, cookie []string) { if req.Header.Get("Cookie") == "" { req.Header.Set("Cookie", strings.Join(cookie, "")) } } func makeEndpointsListV1(headers []string) []string { var endpoints []string for _, ep := range headers { endpointsList := strings.Split(ep, ",") for _, endpointEl := range endpointsList { endpoints = append( endpoints, path.Join(strings.TrimSpace(endpointEl), "v1")) } } return endpoints } docker2aci-0.12.0/lib/internal/backend/repository/repository2.go000066400000000000000000000425771273674375500246620ustar00rootroot00000000000000// Copyright 2016 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package repository import ( "encoding/json" "errors" "fmt" "io" "io/ioutil" "net/http" "os" "path" "regexp" "strconv" "strings" "sync" "time" "github.com/appc/docker2aci/lib/common" "github.com/appc/docker2aci/lib/internal" "github.com/appc/docker2aci/lib/internal/types" "github.com/appc/docker2aci/lib/internal/typesV2" "github.com/appc/docker2aci/lib/internal/util" "github.com/appc/docker2aci/pkg/log" "github.com/appc/spec/schema" "github.com/coreos/pkg/progressutil" ) const ( defaultIndexURL = "registry-1.docker.io" ) var validHex = regexp.MustCompile(`^([a-f0-9]{64})$`) // A manifest conforming to the docker v2.1 spec type v2Manifest struct { Name string `json:"name"` Tag string `json:"tag"` FSLayers []struct { BlobSum string `json:"blobSum"` } `json:"fsLayers"` History []struct { V1Compatibility string `json:"v1Compatibility"` } `json:"history"` Signature []byte `json:"signature"` } func (rb *RepositoryBackend) getImageInfoV2(dockerURL *types.ParsedDockerURL) ([]string, *types.ParsedDockerURL, error) { layers, err := rb.getManifestV2(dockerURL) if err != nil { return nil, nil, err } return layers, dockerURL, nil } func (rb *RepositoryBackend) buildACIV2(layerIDs []string, dockerURL *types.ParsedDockerURL, outputDir string, tmpBaseDir string, compression common.Compression) ([]string, []*schema.ImageManifest, error) { _, isVersion22 := rb.imageV2Manifests[*dockerURL] if isVersion22 { return rb.buildACIV22(layerIDs, dockerURL, outputDir, tmpBaseDir, compression) } return rb.buildACIV21(layerIDs, dockerURL, outputDir, tmpBaseDir, compression) } func (rb *RepositoryBackend) buildACIV21(layerIDs []string, dockerURL *types.ParsedDockerURL, outputDir string, tmpBaseDir string, compression common.Compression) ([]string, []*schema.ImageManifest, error) { layerFiles := make([]*os.File, len(layerIDs)) layerDatas := make([]types.DockerImageData, len(layerIDs)) tmpParentDir, err := ioutil.TempDir(tmpBaseDir, "docker2aci-") if err != nil { return nil, nil, err } defer os.RemoveAll(tmpParentDir) copier := progressutil.NewCopyProgressPrinter() var errChannels []chan error closers := make([]io.ReadCloser, len(layerIDs)) var wg sync.WaitGroup for i, layerID := range layerIDs { wg.Add(1) errChan := make(chan error, 1) errChannels = append(errChannels, errChan) // https://github.com/golang/go/wiki/CommonMistakes i := i // golang-- layerID := layerID go func() { defer wg.Done() manifest := rb.imageManifests[*dockerURL] layerIndex, err := getLayerIndex(layerID, manifest) if err != nil { errChan <- err return } if len(manifest.History) <= layerIndex { errChan <- fmt.Errorf("history not found for layer %s", layerID) return } layerDatas[i] = types.DockerImageData{} if err := json.Unmarshal([]byte(manifest.History[layerIndex].V1Compatibility), &layerDatas[i]); err != nil { errChan <- fmt.Errorf("error unmarshaling layer data: %v", err) return } tmpDir, err := ioutil.TempDir(tmpParentDir, "") if err != nil { errChan <- fmt.Errorf("error creating dir: %v", err) return } layerFiles[i], closers[i], err = rb.getLayerV2(layerID, dockerURL, tmpDir, copier) if err != nil { errChan <- fmt.Errorf("error getting the remote layer: %v", err) return } errChan <- nil }() } // Need to wait for all of the readers to be added to the copier (which happens during rb.getLayerV2) wg.Wait() err = copier.PrintAndWait(os.Stderr, 500*time.Millisecond, nil) if err != nil { return nil, nil, err } for _, closer := range closers { if closer != nil { closer.Close() } } for _, errChan := range errChannels { err := <-errChan if err != nil { return nil, nil, err } } for _, layerFile := range layerFiles { err := layerFile.Sync() if err != nil { return nil, nil, err } } var aciLayerPaths []string var aciManifests []*schema.ImageManifest var curPwl []string for i := len(layerIDs) - 1; i >= 0; i-- { log.Debug("Generating layer ACI...") aciPath, aciManifest, err := internal.GenerateACI(i, layerDatas[i], dockerURL, outputDir, layerFiles[i], curPwl, compression) if err != nil { return nil, nil, fmt.Errorf("error generating ACI: %v", err) } aciLayerPaths = append(aciLayerPaths, aciPath) aciManifests = append(aciManifests, aciManifest) curPwl = aciManifest.PathWhitelist layerFiles[i].Close() } return aciLayerPaths, aciManifests, nil } type layer struct { index int file *os.File closer io.Closer err error } func (rb *RepositoryBackend) buildACIV22(layerIDs []string, dockerURL *types.ParsedDockerURL, outputDir string, tmpBaseDir string, compression common.Compression) ([]string, []*schema.ImageManifest, error) { layerFiles := make([]*os.File, len(layerIDs)) tmpParentDir, err := ioutil.TempDir(tmpBaseDir, "docker2aci-") if err != nil { return nil, nil, err } defer os.RemoveAll(tmpParentDir) copier := progressutil.NewCopyProgressPrinter() resultChan := make(chan layer, len(layerIDs)) for i, layerID := range layerIDs { // https://github.com/golang/go/wiki/CommonMistakes i := i // golang-- layerID := layerID go func() { tmpDir, err := ioutil.TempDir(tmpParentDir, "") if err != nil { resultChan <- layer{ index: i, err: fmt.Errorf("error creating dir: %v", err), } return } layerFile, closer, err := rb.getLayerV2(layerID, dockerURL, tmpDir, copier) if err != nil { resultChan <- layer{ index: i, err: fmt.Errorf("error getting the remote layer: %v", err), } return } resultChan <- layer{ index: i, file: layerFile, closer: closer, err: nil, } }() } var errs []error for i := 0; i < len(layerIDs); i++ { res := <-resultChan if res.closer != nil { defer res.closer.Close() } if res.file != nil { defer res.file.Close() } if res.err != nil { errs = append(errs, res.err) } layerFiles[res.index] = res.file } if len(errs) > 0 { return nil, nil, errs[0] } err = copier.PrintAndWait(os.Stderr, 500*time.Millisecond, nil) if err != nil { return nil, nil, err } for _, layerFile := range layerFiles { err := layerFile.Sync() if err != nil { return nil, nil, err } } var aciLayerPaths []string var aciManifests []*schema.ImageManifest var curPwl []string for i := len(layerIDs) - 1; i > 0; i-- { log.Debug("Generating layer ACI...") aciPath, aciManifest, err := internal.GenerateACI22LowerLayer(dockerURL, layerIDs[i], outputDir, layerFiles[i], curPwl, compression) if err != nil { return nil, nil, fmt.Errorf("error generating ACI: %v", err) } aciLayerPaths = append(aciLayerPaths, aciPath) aciManifests = append(aciManifests, aciManifest) curPwl = aciManifest.PathWhitelist } aciPath, aciManifest, err := internal.GenerateACI22TopLayer(dockerURL, rb.imageConfigs[*dockerURL], layerIDs[0], outputDir, layerFiles[0], curPwl, compression, aciManifests) if err != nil { return nil, nil, fmt.Errorf("error generating ACI: %v", err) } aciLayerPaths = append(aciLayerPaths, aciPath) aciManifests = append(aciManifests, aciManifest) return aciLayerPaths, aciManifests, nil } func (rb *RepositoryBackend) getManifestV2(dockerURL *types.ParsedDockerURL) ([]string, error) { var reference string if dockerURL.Digest != "" { reference = dockerURL.Digest } else { reference = dockerURL.Tag } url := rb.schema + path.Join(dockerURL.IndexURL, "v2", dockerURL.ImageName, "manifests", reference) req, err := http.NewRequest("GET", url, nil) if err != nil { return nil, err } rb.setBasicAuth(req) accepting := []string{ typesV2.MediaTypeOCIManifest, typesV2.MediaTypeDockerV22Manifest, typesV2.MediaTypeDockerV21Manifest, } res, err := rb.makeRequest(req, dockerURL.ImageName, accepting) if err != nil { return nil, err } defer res.Body.Close() if res.StatusCode != http.StatusOK { return nil, fmt.Errorf("unexpected http code: %d, URL: %s", res.StatusCode, req.URL) } switch res.Header.Get("content-type") { case typesV2.MediaTypeDockerV22Manifest, typesV2.MediaTypeOCIManifest: return rb.getManifestV22(dockerURL, res) case typesV2.MediaTypeDockerV21Manifest: return rb.getManifestV21(dockerURL, res) } return rb.getManifestV21(dockerURL, res) } func (rb *RepositoryBackend) getManifestV21(dockerURL *types.ParsedDockerURL, res *http.Response) ([]string, error) { manblob, err := ioutil.ReadAll(res.Body) if err != nil { return nil, err } manifest := &v2Manifest{} err = json.Unmarshal(manblob, manifest) if err != nil { return nil, err } if manifest.Name != dockerURL.ImageName { return nil, fmt.Errorf("name doesn't match what was requested, expected: %s, downloaded: %s", dockerURL.ImageName, manifest.Name) } if manifest.Tag != dockerURL.Tag { return nil, fmt.Errorf("tag doesn't match what was requested, expected: %s, downloaded: %s", dockerURL.Tag, manifest.Tag) } if err := fixManifestLayers(manifest); err != nil { return nil, err } //TODO: verify signature here layers := make([]string, len(manifest.FSLayers)) for i, layer := range manifest.FSLayers { layers[i] = layer.BlobSum } rb.imageManifests[*dockerURL] = *manifest return layers, nil } func (rb *RepositoryBackend) getManifestV22(dockerURL *types.ParsedDockerURL, res *http.Response) ([]string, error) { manblob, err := ioutil.ReadAll(res.Body) if err != nil { return nil, err } manifest := &typesV2.ImageManifest{} err = json.Unmarshal(manblob, manifest) if err != nil { return nil, err } //TODO: verify signature here layers := make([]string, len(manifest.Layers)) for i, layer := range manifest.Layers { layers[i] = layer.Digest } err = rb.getConfigV22(dockerURL, manifest.Config.Digest) if err != nil { return nil, err } rb.imageV2Manifests[*dockerURL] = manifest return layers, nil } func (rb *RepositoryBackend) getConfigV22(dockerURL *types.ParsedDockerURL, configDigest string) error { url := rb.schema + path.Join(dockerURL.IndexURL, "v2", dockerURL.ImageName, "blobs", configDigest) req, err := http.NewRequest("GET", url, nil) if err != nil { return err } rb.setBasicAuth(req) accepting := []string{ typesV2.MediaTypeOCIConfig, typesV2.MediaTypeDockerV22Config, } res, err := rb.makeRequest(req, dockerURL.ImageName, accepting) if err != nil { return err } defer res.Body.Close() confblob, err := ioutil.ReadAll(res.Body) if err != nil { return err } config := &typesV2.ImageConfig{} err = json.Unmarshal(confblob, config) if err != nil { return err } rb.imageConfigs[*dockerURL] = config return nil } func fixManifestLayers(manifest *v2Manifest) error { type imageV1 struct { ID string Parent string } imgs := make([]*imageV1, len(manifest.FSLayers)) for i := range manifest.FSLayers { img := &imageV1{} if err := json.Unmarshal([]byte(manifest.History[i].V1Compatibility), img); err != nil { return err } imgs[i] = img if err := validateV1ID(img.ID); err != nil { return err } } if imgs[len(imgs)-1].Parent != "" { return errors.New("Invalid parent ID in the base layer of the image.") } // check general duplicates to error instead of a deadlock idmap := make(map[string]struct{}) var lastID string for _, img := range imgs { // skip IDs that appear after each other, we handle those later if _, exists := idmap[img.ID]; img.ID != lastID && exists { return fmt.Errorf("ID %+v appears multiple times in manifest", img.ID) } lastID = img.ID idmap[lastID] = struct{}{} } // backwards loop so that we keep the remaining indexes after removing items for i := len(imgs) - 2; i >= 0; i-- { if imgs[i].ID == imgs[i+1].ID { // repeated ID. remove and continue manifest.FSLayers = append(manifest.FSLayers[:i], manifest.FSLayers[i+1:]...) manifest.History = append(manifest.History[:i], manifest.History[i+1:]...) } else if imgs[i].Parent != imgs[i+1].ID { return fmt.Errorf("Invalid parent ID. Expected %v, got %v.", imgs[i+1].ID, imgs[i].Parent) } } return nil } func validateV1ID(id string) error { if ok := validHex.MatchString(id); !ok { return fmt.Errorf("image ID %q is invalid", id) } return nil } func getLayerIndex(layerID string, manifest v2Manifest) (int, error) { for i, layer := range manifest.FSLayers { if layer.BlobSum == layerID { return i, nil } } return -1, fmt.Errorf("layer not found in manifest: %s", layerID) } func (rb *RepositoryBackend) getLayerV2(layerID string, dockerURL *types.ParsedDockerURL, tmpDir string, copier *progressutil.CopyProgressPrinter) (*os.File, io.ReadCloser, error) { url := rb.schema + path.Join(dockerURL.IndexURL, "v2", dockerURL.ImageName, "blobs", layerID) req, err := http.NewRequest("GET", url, nil) if err != nil { return nil, nil, err } rb.setBasicAuth(req) accepting := []string{ typesV2.MediaTypeDockerV22RootFS, typesV2.MediaTypeOCIRootFS, } res, err := rb.makeRequest(req, dockerURL.ImageName, accepting) if err != nil { return nil, nil, err } if res.StatusCode == http.StatusTemporaryRedirect || res.StatusCode == http.StatusFound { location := res.Header.Get("Location") if location != "" { req, err = http.NewRequest("GET", location, nil) if err != nil { return nil, nil, err } res, err = rb.makeRequest(req, dockerURL.ImageName, accepting) if err != nil { return nil, nil, err } defer res.Body.Close() } } if res.StatusCode != http.StatusOK { return nil, nil, fmt.Errorf("HTTP code: %d. URL: %s", res.StatusCode, req.URL) } var in io.Reader in = res.Body var size int64 if hdr := res.Header.Get("Content-Length"); hdr != "" { size, err = strconv.ParseInt(hdr, 10, 64) if err != nil { return nil, nil, err } } name := "Downloading " + layerID[:18] layerFile, err := ioutil.TempFile(tmpDir, "dockerlayer-") if err != nil { return nil, nil, err } err = copier.AddCopy(in, name, size, layerFile) if err != nil { return nil, nil, err } return layerFile, res.Body, nil } func (rb *RepositoryBackend) makeRequest(req *http.Request, repo string, acceptHeaders []string) (*http.Response, error) { setBearerHeader := false hostAuthTokens, ok := rb.hostsV2AuthTokens[req.URL.Host] if ok { authToken, ok := hostAuthTokens[repo] if ok { req.Header.Set("Authorization", "Bearer "+authToken) setBearerHeader = true } } for _, acceptHeader := range acceptHeaders { req.Header.Add("Accept", acceptHeader) } client := util.GetTLSClient(rb.insecure.SkipVerify) res, err := client.Do(req) if err != nil { return nil, err } if res.StatusCode == http.StatusUnauthorized && setBearerHeader { return res, err } hdr := res.Header.Get("www-authenticate") if res.StatusCode != http.StatusUnauthorized || hdr == "" { return res, err } tokens := strings.Split(hdr, ",") if len(tokens) != 3 || !strings.HasPrefix(strings.ToLower(tokens[0]), "bearer realm") { return res, err } res.Body.Close() var realm, service, scope string for _, token := range tokens { if strings.HasPrefix(strings.ToLower(token), "bearer realm") { realm = strings.Trim(token[len("bearer realm="):], "\"") } if strings.HasPrefix(token, "service") { service = strings.Trim(token[len("service="):], "\"") } if strings.HasPrefix(token, "scope") { scope = strings.Trim(token[len("scope="):], "\"") } } if realm == "" { return nil, fmt.Errorf("missing realm in bearer auth challenge") } if service == "" { return nil, fmt.Errorf("missing service in bearer auth challenge") } // The scope can be empty if we're not getting a token for a specific repo if scope == "" && repo != "" { // If the scope is empty and it shouldn't be, we can infer it based on the repo scope = fmt.Sprintf("repository:%s:pull", repo) } authReq, err := http.NewRequest("GET", realm, nil) if err != nil { return nil, err } getParams := authReq.URL.Query() getParams.Add("service", service) if scope != "" { getParams.Add("scope", scope) } authReq.URL.RawQuery = getParams.Encode() rb.setBasicAuth(authReq) res, err = client.Do(authReq) if err != nil { return nil, err } defer res.Body.Close() switch res.StatusCode { case http.StatusUnauthorized: return nil, fmt.Errorf("unable to retrieve auth token: 401 unauthorized") case http.StatusOK: break default: return nil, fmt.Errorf("unexpected http code: %d, URL: %s", res.StatusCode, authReq.URL) } tokenBlob, err := ioutil.ReadAll(res.Body) if err != nil { return nil, err } tokenStruct := struct { Token string `json:"token"` }{} err = json.Unmarshal(tokenBlob, &tokenStruct) if err != nil { return nil, err } hostAuthTokens, ok = rb.hostsV2AuthTokens[req.URL.Host] if !ok { hostAuthTokens = make(map[string]string) rb.hostsV2AuthTokens[req.URL.Host] = hostAuthTokens } hostAuthTokens[repo] = tokenStruct.Token return rb.makeRequest(req, repo, acceptHeaders) } func (rb *RepositoryBackend) setBasicAuth(req *http.Request) { if rb.username != "" && rb.password != "" { req.SetBasicAuth(rb.username, rb.password) } } docker2aci-0.12.0/lib/internal/docker/000077500000000000000000000000001273674375500174545ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/docker/docker_functions.go000066400000000000000000000107301273674375500233430ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package docker import ( "encoding/base64" "encoding/json" "fmt" "io/ioutil" "os" "path" "runtime" "strings" "github.com/appc/docker2aci/lib/internal/types" "github.com/docker/distribution/reference" ) const ( dockercfgFileName = "config.json" dockercfgFileNameOld = ".dockercfg" defaultIndexURL = "registry-1.docker.io" defaultIndexURLAuth = "https://index.docker.io/v1/" defaultTag = "latest" defaultRepoPrefix = "library/" ) // SplitReposName breaks a repo name into an index name and remote name. func SplitReposName(name string) (indexName, remoteName string) { i := strings.IndexRune(name, '/') if i == -1 || (!strings.ContainsAny(name[:i], ".:") && name[:i] != "localhost") { indexName, remoteName = defaultIndexURL, name } else { indexName, remoteName = name[:i], name[i+1:] } if indexName == defaultIndexURL && !strings.ContainsRune(remoteName, '/') { remoteName = defaultRepoPrefix + remoteName } return } // Get a repos name and returns the right reposName + tag // The tag can be confusing because of a port in a repository name. // Ex: localhost.localdomain:5000/samalba/hipache:latest func parseRepositoryTag(repos string) (string, string) { n := strings.LastIndex(repos, ":") if n < 0 { return repos, "" } if tag := repos[n+1:]; !strings.Contains(tag, "/") { return repos[:n], tag } return repos, "" } func decodeDockerAuth(s string) (string, string, error) { decoded, err := base64.StdEncoding.DecodeString(s) if err != nil { return "", "", err } parts := strings.SplitN(string(decoded), ":", 2) if len(parts) != 2 { return "", "", fmt.Errorf("invalid auth configuration file") } user := parts[0] password := strings.Trim(parts[1], "\x00") return user, password, nil } func getHomeDir() string { if runtime.GOOS == "windows" { return os.Getenv("USERPROFILE") } return os.Getenv("HOME") } // GetDockercfgAuth reads a ~/.dockercfg file and returns the username and password // of the given docker index server. func GetAuthInfo(indexServer string) (string, string, error) { // official docker registry if indexServer == defaultIndexURL { indexServer = defaultIndexURLAuth } dockerCfgPath := path.Join(getHomeDir(), ".docker", dockercfgFileName) if _, err := os.Stat(dockerCfgPath); err == nil { j, err := ioutil.ReadFile(dockerCfgPath) if err != nil { return "", "", err } var dockerAuth types.DockerConfigFile if err := json.Unmarshal(j, &dockerAuth); err != nil { return "", "", err } // try the normal case if c, ok := dockerAuth.AuthConfigs[indexServer]; ok { return decodeDockerAuth(c.Auth) } } else if os.IsNotExist(err) { oldDockerCfgPath := path.Join(getHomeDir(), dockercfgFileNameOld) if _, err := os.Stat(oldDockerCfgPath); err != nil { return "", "", nil //missing file is not an error } j, err := ioutil.ReadFile(oldDockerCfgPath) if err != nil { return "", "", err } var dockerAuthOld map[string]types.DockerAuthConfigOld if err := json.Unmarshal(j, &dockerAuthOld); err != nil { return "", "", err } if c, ok := dockerAuthOld[indexServer]; ok { return decodeDockerAuth(c.Auth) } } else { // if file is there but we can't stat it for any reason other // than it doesn't exist then stop return "", "", fmt.Errorf("%s - %v", dockerCfgPath, err) } return "", "", nil } // ParseDockerURL takes a Docker URL and returns a ParsedDockerURL with its // index URL, image name, and tag. func ParseDockerURL(arg string) (*types.ParsedDockerURL, error) { r, err := reference.ParseNamed(arg) if err != nil { return nil, err } tag := defaultTag var digest string switch x := r.(type) { case reference.Canonical: digest = x.Digest().String() case reference.NamedTagged: tag = x.Tag() } indexURL, remoteName := SplitReposName(r.Name()) p := &types.ParsedDockerURL{ IndexURL: indexURL, ImageName: remoteName, Tag: tag, Digest: digest, } return p, nil } docker2aci-0.12.0/lib/internal/internal.go000066400000000000000000000520561273674375500203600ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // Package internal provides functions shared by different parts of docker2aci. // // Note: this package is an implementation detail and shouldn't be used outside // of docker2aci. package internal import ( "archive/tar" "encoding/json" "fmt" "io" "os" "path" "path/filepath" "runtime" "sort" "strconv" "strings" "time" "github.com/appc/docker2aci/lib/common" "github.com/appc/docker2aci/lib/internal/tarball" "github.com/appc/docker2aci/lib/internal/types" "github.com/appc/docker2aci/lib/internal/typesV2" "github.com/appc/docker2aci/lib/internal/util" "github.com/appc/docker2aci/pkg/log" "github.com/appc/spec/aci" "github.com/appc/spec/schema" appctypes "github.com/appc/spec/schema/types" gzip "github.com/klauspost/pgzip" ) // Docker2ACIBackend is the interface that abstracts converting Docker layers // to ACI from where they're stored (remote or file). // // GetImageInfo takes a Docker URL and returns a list of layers and the parsed // Docker URL. // // BuildACI takes a Docker layer, converts it to ACI and returns its output // path and its converted ImageManifest. type Docker2ACIBackend interface { GetImageInfo(dockerUrl string) ([]string, *types.ParsedDockerURL, error) BuildACI(layerIDs []string, dockerURL *types.ParsedDockerURL, outputDir string, tmpBaseDir string, compression common.Compression) ([]string, []*schema.ImageManifest, error) } // GenerateACI takes a Docker layer and generates an ACI from it. func GenerateACI(layerNumber int, layerData types.DockerImageData, dockerURL *types.ParsedDockerURL, outputDir string, layerFile *os.File, curPwl []string, compression common.Compression) (string, *schema.ImageManifest, error) { manifest, err := GenerateManifest(layerData, dockerURL) if err != nil { return "", nil, fmt.Errorf("error generating the manifest: %v", err) } imageName := strings.Replace(dockerURL.ImageName, "/", "-", -1) aciPath := generateACIPath(outputDir, imageName, layerData.ID, dockerURL.Tag, layerData.OS, layerData.Architecture, layerNumber) manifest, err = writeACI(layerFile, *manifest, curPwl, aciPath, compression) if err != nil { return "", nil, fmt.Errorf("error writing ACI: %v", err) } if err := ValidateACI(aciPath); err != nil { return "", nil, fmt.Errorf("invalid ACI generated: %v", err) } return aciPath, manifest, nil } func GenerateACI22LowerLayer(dockerURL *types.ParsedDockerURL, layerDigest string, outputDir string, layerFile *os.File, curPwl []string, compression common.Compression) (string, *schema.ImageManifest, error) { formattedDigest := strings.Replace(layerDigest, ":", "-", -1) aciName := fmt.Sprintf("%s/%s-%s", dockerURL.IndexURL, dockerURL.ImageName, formattedDigest) manifest, err := GenerateEmptyManifest(aciName) if err != nil { return "", nil, err } aciPath := generateACIPath(outputDir, aciName, layerDigest, dockerURL.Tag, runtime.GOOS, runtime.GOARCH, -1) manifest, err = writeACI(layerFile, *manifest, curPwl, aciPath, compression) if err != nil { return "", nil, err } err = ValidateACI(aciPath) if err != nil { return "", nil, fmt.Errorf("invalid ACI generated: %v", err) } return aciPath, manifest, nil } func GenerateACI22TopLayer(dockerURL *types.ParsedDockerURL, imageConfig *typesV2.ImageConfig, layerDigest string, outputDir string, layerFile *os.File, curPwl []string, compression common.Compression, lowerLayers []*schema.ImageManifest) (string, *schema.ImageManifest, error) { aciName := fmt.Sprintf("%s/%s", dockerURL.IndexURL, dockerURL.ImageName) manifest, err := GenerateManifestV22(aciName, dockerURL, imageConfig, layerDigest, lowerLayers) if err != nil { return "", nil, err } aciPath := generateACIPath(outputDir, aciName, layerDigest, dockerURL.Tag, runtime.GOOS, runtime.GOARCH, -1) manifest, err = writeACI(layerFile, *manifest, curPwl, aciPath, compression) if err != nil { return "", nil, err } err = ValidateACI(aciPath) if err != nil { return "", nil, fmt.Errorf("invalid ACI generated: %v", err) } return aciPath, manifest, nil } func generateACIPath(outputDir, imageName, digest, tag, osString, arch string, layerNum int) string { aciPath := imageName if tag != "" { aciPath += "-" + tag } if osString != "" { aciPath += "-" + osString if arch != "" { aciPath += "-" + arch } } if layerNum != -1 { aciPath += "-" + strconv.Itoa(layerNum) } aciPath += ".aci" return path.Join(outputDir, aciPath) } // GenerateManifest converts the docker manifest format to an appc // ImageManifest. func GenerateManifest(layerData types.DockerImageData, dockerURL *types.ParsedDockerURL) (*schema.ImageManifest, error) { dockerConfig := layerData.Config genManifest := &schema.ImageManifest{} appURL := "" appURL = dockerURL.IndexURL + "/" appURL += dockerURL.ImageName + "-" + layerData.ID appURL, err := appctypes.SanitizeACIdentifier(appURL) if err != nil { return nil, err } name := appctypes.MustACIdentifier(appURL) genManifest.Name = *name acVersion, err := appctypes.NewSemVer(schema.AppContainerVersion.String()) if err != nil { panic("invalid appc spec version") } genManifest.ACVersion = *acVersion genManifest.ACKind = appctypes.ACKind(schema.ImageManifestKind) var ( labels appctypes.Labels parentLabels appctypes.Labels annotations appctypes.Annotations ) layer := appctypes.MustACIdentifier("layer") labels = append(labels, appctypes.Label{Name: *layer, Value: layerData.ID}) tag := dockerURL.Tag version := appctypes.MustACIdentifier("version") labels = append(labels, appctypes.Label{Name: *version, Value: tag}) if layerData.OS != "" { os := appctypes.MustACIdentifier("os") labels = append(labels, appctypes.Label{Name: *os, Value: layerData.OS}) parentLabels = append(parentLabels, appctypes.Label{Name: *os, Value: layerData.OS}) if layerData.Architecture != "" { arch := appctypes.MustACIdentifier("arch") labels = append(labels, appctypes.Label{Name: *arch, Value: layerData.Architecture}) parentLabels = append(parentLabels, appctypes.Label{Name: *arch, Value: layerData.Architecture}) } } if layerData.Author != "" { authorsKey := appctypes.MustACIdentifier("authors") annotations = append(annotations, appctypes.Annotation{Name: *authorsKey, Value: layerData.Author}) } epoch := time.Unix(0, 0) if !layerData.Created.Equal(epoch) { createdKey := appctypes.MustACIdentifier("created") annotations = append(annotations, appctypes.Annotation{Name: *createdKey, Value: layerData.Created.Format(time.RFC3339)}) } if layerData.Comment != "" { commentKey := appctypes.MustACIdentifier("docker-comment") annotations = append(annotations, appctypes.Annotation{Name: *commentKey, Value: layerData.Comment}) } if dockerURL.IndexURL != "" { annotations = append(annotations, appctypes.Annotation{Name: *appctypes.MustACIdentifier(common.AppcDockerRegistryURL), Value: dockerURL.IndexURL}) } annotations = append(annotations, appctypes.Annotation{Name: *appctypes.MustACIdentifier(common.AppcDockerRepository), Value: dockerURL.ImageName}) annotations = append(annotations, appctypes.Annotation{Name: *appctypes.MustACIdentifier(common.AppcDockerImageID), Value: layerData.ID}) annotations = append(annotations, appctypes.Annotation{Name: *appctypes.MustACIdentifier(common.AppcDockerParentImageID), Value: layerData.Parent}) if dockerConfig != nil { if len(dockerConfig.Entrypoint) > 0 { entry, err := json.Marshal(dockerConfig.Entrypoint) if err != nil { return nil, err } annotations = append(annotations, appctypes.Annotation{Name: *appctypes.MustACIdentifier(common.AppcDockerEntrypoint), Value: string(entry)}) } if len(dockerConfig.Cmd) > 0 { cmd, err := json.Marshal(dockerConfig.Cmd) if err != nil { return nil, err } annotations = append(annotations, appctypes.Annotation{Name: *appctypes.MustACIdentifier(common.AppcDockerCmd), Value: string(cmd)}) } exec := getExecCommand(dockerConfig.Entrypoint, dockerConfig.Cmd) if exec != nil { user, group := parseDockerUser(dockerConfig.User) var env appctypes.Environment for _, v := range dockerConfig.Env { parts := strings.SplitN(v, "=", 2) env.Set(parts[0], parts[1]) } app := &appctypes.App{ Exec: exec, User: user, Group: group, Environment: env, WorkingDirectory: dockerConfig.WorkingDir, } app.MountPoints, err = convertVolumesToMPs(dockerConfig.Volumes) if err != nil { return nil, err } app.Ports, err = convertPorts(dockerConfig.ExposedPorts, dockerConfig.PortSpecs) if err != nil { return nil, err } genManifest.App = app } } if layerData.Parent != "" { indexPrefix := "" // omit docker hub index URL in app name indexPrefix = dockerURL.IndexURL + "/" parentImageNameString := indexPrefix + dockerURL.ImageName + "-" + layerData.Parent parentImageNameString, err := appctypes.SanitizeACIdentifier(parentImageNameString) if err != nil { return nil, err } parentImageName := appctypes.MustACIdentifier(parentImageNameString) genManifest.Dependencies = append(genManifest.Dependencies, appctypes.Dependency{ImageName: *parentImageName, Labels: parentLabels}) annotations = append(annotations, appctypes.Annotation{Name: *appctypes.MustACIdentifier(common.AppcDockerTag), Value: dockerURL.Tag}) } genManifest.Labels = labels genManifest.Annotations = annotations return genManifest, nil } func GenerateEmptyManifest(name string) (*schema.ImageManifest, error) { acid, err := appctypes.NewACIdentifier(name) if err != nil { return nil, err } return &schema.ImageManifest{ ACKind: schema.ImageManifestKind, ACVersion: schema.AppContainerVersion, Name: *acid, Labels: appctypes.Labels{ appctypes.Label{ *appctypes.MustACIdentifier("arch"), runtime.GOARCH, }, appctypes.Label{ *appctypes.MustACIdentifier("os"), runtime.GOOS, }, }, }, nil } func GenerateManifestV22(name string, dockerURL *types.ParsedDockerURL, config *typesV2.ImageConfig, imageDigest string, lowerLayers []*schema.ImageManifest) (*schema.ImageManifest, error) { manifest, err := GenerateEmptyManifest(name) if err != nil { return nil, err } labels := manifest.Labels.ToMap() annotations := manifest.Annotations addLabel := func(key, val string) { if key != "" && val != "" { labels[*appctypes.MustACIdentifier(key)] = val } } addAnno := func(key, val string) { if key != "" && val != "" { annotations.Set(*appctypes.MustACIdentifier(key), val) } } addLabel("version", dockerURL.Tag) addLabel("os", config.OS) addLabel("arch", config.Architecture) addAnno("author", config.Author) addAnno("created", config.Created) addAnno(common.AppcDockerRegistryURL, dockerURL.IndexURL) addAnno(common.AppcDockerRepository, dockerURL.ImageName) addAnno(common.AppcDockerImageID, imageDigest) addAnno("created", config.Created) if config.Config != nil { exec := getExecCommand(config.Config.Entrypoint, config.Config.Cmd) user, group := parseDockerUser(config.Config.User) var env appctypes.Environment for _, v := range config.Config.Env { parts := strings.SplitN(v, "=", 2) env.Set(parts[0], parts[1]) } manifest.App = &appctypes.App{ Exec: exec, User: user, Group: group, Environment: env, WorkingDirectory: config.Config.WorkingDir, } manifest.App.MountPoints, err = convertVolumesToMPs(config.Config.Volumes) if err != nil { return nil, err } manifest.App.Ports, err = convertPorts(config.Config.ExposedPorts, nil) if err != nil { return nil, err } } for _, lowerLayer := range lowerLayers { manifest.Dependencies = append(manifest.Dependencies, appctypes.Dependency{ ImageName: lowerLayer.Name, Labels: lowerLayer.Labels, }) } manifest.Labels, err = appctypes.LabelsFromMap(labels) if err != nil { return nil, err } manifest.Annotations = annotations return manifest, nil } // ValidateACI checks whether the ACI in aciPath is valid. func ValidateACI(aciPath string) error { aciFile, err := os.Open(aciPath) if err != nil { return err } defer aciFile.Close() tr, err := aci.NewCompressedTarReader(aciFile) if err != nil { return err } defer tr.Close() if err := aci.ValidateArchive(tr.Reader); err != nil { return err } return nil } type appcPortSorter []appctypes.Port func (s appcPortSorter) Len() int { return len(s) } func (s appcPortSorter) Swap(i, j int) { s[i], s[j] = s[j], s[i] } func (s appcPortSorter) Less(i, j int) bool { return s[i].Name.String() < s[j].Name.String() } func convertPorts(dockerExposedPorts map[string]struct{}, dockerPortSpecs []string) ([]appctypes.Port, error) { ports := []appctypes.Port{} for ep := range dockerExposedPorts { appcPort, err := parseDockerPort(ep) if err != nil { return nil, err } ports = append(ports, *appcPort) } if dockerExposedPorts == nil && dockerPortSpecs != nil { log.Debug("warning: docker image uses deprecated PortSpecs field") for _, ep := range dockerPortSpecs { appcPort, err := parseDockerPort(ep) if err != nil { return nil, err } ports = append(ports, *appcPort) } } sort.Sort(appcPortSorter(ports)) return ports, nil } func parseDockerPort(dockerPort string) (*appctypes.Port, error) { var portString string proto := "tcp" sp := strings.Split(dockerPort, "/") if len(sp) < 2 { portString = dockerPort } else { proto = sp[1] portString = sp[0] } port, err := strconv.ParseUint(portString, 10, 0) if err != nil { return nil, fmt.Errorf("error parsing port %q: %v", portString, err) } sn, err := appctypes.SanitizeACName(dockerPort) if err != nil { return nil, err } appcPort := &appctypes.Port{ Name: *appctypes.MustACName(sn), Protocol: proto, Port: uint(port), } return appcPort, nil } type appcVolSorter []appctypes.MountPoint func (s appcVolSorter) Len() int { return len(s) } func (s appcVolSorter) Swap(i, j int) { s[i], s[j] = s[j], s[i] } func (s appcVolSorter) Less(i, j int) bool { return s[i].Name.String() < s[j].Name.String() } func convertVolumesToMPs(dockerVolumes map[string]struct{}) ([]appctypes.MountPoint, error) { mps := []appctypes.MountPoint{} dup := make(map[string]int) for p := range dockerVolumes { n := filepath.Join("volume", p) sn, err := appctypes.SanitizeACName(n) if err != nil { return nil, err } // check for duplicate names if i, ok := dup[sn]; ok { dup[sn] = i + 1 sn = fmt.Sprintf("%s-%d", sn, i) } else { dup[sn] = 1 } mp := appctypes.MountPoint{ Name: *appctypes.MustACName(sn), Path: p, } mps = append(mps, mp) } sort.Sort(appcVolSorter(mps)) return mps, nil } func writeACI(layer io.ReadSeeker, manifest schema.ImageManifest, curPwl []string, output string, compression common.Compression) (*schema.ImageManifest, error) { dir, _ := path.Split(output) err := os.MkdirAll(dir, 0755) if err != nil { return nil, fmt.Errorf("error creating ACI parent dir: %v", err) } aciFile, err := os.Create(output) if err != nil { return nil, fmt.Errorf("error creating ACI file: %v", err) } defer aciFile.Close() var w io.WriteCloser = aciFile if compression == common.GzipCompression { w = gzip.NewWriter(aciFile) defer w.Close() } trw := tar.NewWriter(w) defer trw.Close() if err := WriteRootfsDir(trw); err != nil { return nil, fmt.Errorf("error writing rootfs entry: %v", err) } fileMap := make(map[string]struct{}) var whiteouts []string convWalker := func(t *tarball.TarFile) error { name := t.Name() if name == "./" { return nil } t.Header.Name = path.Join("rootfs", name) absolutePath := strings.TrimPrefix(t.Header.Name, "rootfs") if filepath.Clean(absolutePath) == "/dev" && t.Header.Typeflag != tar.TypeDir { return fmt.Errorf(`invalid layer: "/dev" is not a directory`) } fileMap[absolutePath] = struct{}{} if strings.Contains(t.Header.Name, "/.wh.") { whiteouts = append(whiteouts, strings.Replace(absolutePath, ".wh.", "", 1)) return nil } if t.Header.Typeflag == tar.TypeLink { t.Header.Linkname = path.Join("rootfs", t.Linkname()) } if err := trw.WriteHeader(t.Header); err != nil { return err } if _, err := io.Copy(trw, t.TarStream); err != nil { return err } if !util.In(curPwl, absolutePath) { curPwl = append(curPwl, absolutePath) } return nil } tr, err := aci.NewCompressedTarReader(layer) if err == nil { defer tr.Close() // write files in rootfs/ if err := tarball.Walk(*tr.Reader, convWalker); err != nil { return nil, err } } else { // ignore errors: empty layers in tars generated by docker save are not // valid tar files so we ignore errors trying to open them. Converted // ACIs will have the manifest and an empty rootfs directory in any // case. } newPwl := subtractWhiteouts(curPwl, whiteouts) newPwl, err = writeStdioSymlinks(trw, fileMap, newPwl) if err != nil { return nil, err } // Let's copy the newly generated PathWhitelist to avoid unintended // side-effects manifest.PathWhitelist = make([]string, len(newPwl)) copy(manifest.PathWhitelist, newPwl) if err := WriteManifest(trw, manifest); err != nil { return nil, fmt.Errorf("error writing manifest: %v", err) } return &manifest, nil } func getExecCommand(entrypoint []string, cmd []string) appctypes.Exec { return append(entrypoint, cmd...) } func parseDockerUser(dockerUser string) (string, string) { // if the docker user is empty assume root user and group if dockerUser == "" { return "0", "0" } dockerUserParts := strings.Split(dockerUser, ":") // when only the user is given, the docker spec says that the default and // supplementary groups of the user in /etc/passwd should be applied. // Assume root group for now in this case. if len(dockerUserParts) < 2 { return dockerUserParts[0], "0" } return dockerUserParts[0], dockerUserParts[1] } func subtractWhiteouts(pathWhitelist []string, whiteouts []string) []string { matchPaths := []string{} for _, path := range pathWhitelist { // If one of the parent dirs of the current path matches the // whiteout then also this path should be removed curPath := path for curPath != "/" { for _, whiteout := range whiteouts { if curPath == whiteout { matchPaths = append(matchPaths, path) } } curPath = filepath.Dir(curPath) } } for _, matchPath := range matchPaths { idx := util.IndexOf(pathWhitelist, matchPath) if idx != -1 { pathWhitelist = append(pathWhitelist[:idx], pathWhitelist[idx+1:]...) } } sort.Sort(sort.StringSlice(pathWhitelist)) return pathWhitelist } // WriteManifest writes a schema.ImageManifest entry on a tar.Writer. func WriteManifest(outputWriter *tar.Writer, manifest schema.ImageManifest) error { b, err := json.Marshal(manifest) if err != nil { return err } hdr := getGenericTarHeader() hdr.Name = "manifest" hdr.Mode = 0644 hdr.Size = int64(len(b)) hdr.Typeflag = tar.TypeReg if err := outputWriter.WriteHeader(hdr); err != nil { return err } if _, err := outputWriter.Write(b); err != nil { return err } return nil } // WriteRootfsDir writes a "rootfs" dir entry on a tar.Writer. func WriteRootfsDir(tarWriter *tar.Writer) error { hdr := getGenericTarHeader() hdr.Name = "rootfs" hdr.Mode = 0755 hdr.Size = int64(0) hdr.Typeflag = tar.TypeDir return tarWriter.WriteHeader(hdr) } type symlink struct { linkname string target string } // writeStdioSymlinks adds the /dev/stdin, /dev/stdout, /dev/stderr, and // /dev/fd symlinks expected by Docker to the converted ACIs so apps can find // them as expected func writeStdioSymlinks(tarWriter *tar.Writer, fileMap map[string]struct{}, pwl []string) ([]string, error) { stdioSymlinks := []symlink{ {"/dev/stdin", "/proc/self/fd/0"}, // Docker makes /dev/{stdout,stderr} point to /proc/self/fd/{1,2} but // we point to /dev/console instead in order to support the case when // stdout/stderr is a Unix socket (e.g. for the journal). {"/dev/stdout", "/dev/console"}, {"/dev/stderr", "/dev/console"}, {"/dev/fd", "/proc/self/fd"}, } for _, s := range stdioSymlinks { name := s.linkname target := s.target if _, exists := fileMap[name]; exists { continue } hdr := &tar.Header{ Name: filepath.Join("rootfs", name), Mode: 0777, Typeflag: tar.TypeSymlink, Linkname: target, } if err := tarWriter.WriteHeader(hdr); err != nil { return nil, err } if !util.In(pwl, name) { pwl = append(pwl, name) } } return pwl, nil } func getGenericTarHeader() *tar.Header { // FIXME(iaguis) Use docker image time instead of the Unix Epoch? hdr := &tar.Header{ Uid: 0, Gid: 0, ModTime: time.Unix(0, 0), Uname: "0", Gname: "0", ChangeTime: time.Unix(0, 0), } return hdr } docker2aci-0.12.0/lib/internal/tarball/000077500000000000000000000000001273674375500176265ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/tarball/tarfile.go000066400000000000000000000025221273674375500216040ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // Package tarball provides functions to manipulate tar files. // // Note: this package is an implementation detail and shouldn't be used outside // of docker2aci. package tarball import ( "archive/tar" "io" ) // TarFile is a representation of a file in a tarball. It consists of two parts, // the Header and the Stream. The Header is a regular tar header, the Stream // is a byte stream that can be used to read the file's contents. type TarFile struct { Header *tar.Header TarStream io.Reader } // Name returns the name of the file as reported by the header. func (t *TarFile) Name() string { return t.Header.Name } // Linkname returns the Linkname of the file as reported by the header. func (t *TarFile) Linkname() string { return t.Header.Linkname } docker2aci-0.12.0/lib/internal/tarball/walk.go000066400000000000000000000023741273674375500211210ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package tarball import ( "archive/tar" "fmt" "io" ) // WalkFunc is a func for handling each file (header and byte stream) in a tarball type WalkFunc func(t *TarFile) error // Walk walks through the files in the tarball represented by tarstream and // passes each of them to the WalkFunc provided as an argument func Walk(tarReader tar.Reader, walkFunc func(t *TarFile) error) error { for { hdr, err := tarReader.Next() if err == io.EOF { // end of tar archive break } if err != nil { return fmt.Errorf("Error reading tar entry: %v", err) } if err := walkFunc(&TarFile{Header: hdr, TarStream: &tarReader}); err != nil { return err } } return nil } docker2aci-0.12.0/lib/internal/types/000077500000000000000000000000001273674375500173515ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/types/docker_types.go000066400000000000000000000073001273674375500223730ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package types import "time" // DockerImageData stores the JSON structure of a Docker image. // Taken and adapted from upstream Docker. type DockerImageData struct { ID string `json:"id"` Parent string `json:"parent,omitempty"` Comment string `json:"comment,omitempty"` Created time.Time `json:"created"` Container string `json:"container,omitempty"` ContainerConfig DockerImageConfig `json:"container_config,omitempty"` DockerVersion string `json:"docker_version,omitempty"` Author string `json:"author,omitempty"` Config *DockerImageConfig `json:"config,omitempty"` Architecture string `json:"architecture,omitempty"` OS string `json:"os,omitempty"` Checksum string `json:"checksum"` } // Note: the Config structure should hold only portable information about the container. // Here, "portable" means "independent from the host we are running on". // Non-portable information *should* appear in HostConfig. // Taken and adapted from upstream Docker. type DockerImageConfig struct { Hostname string Domainname string User string Memory int64 // Memory limit (in bytes) MemorySwap int64 // Total memory usage (memory + swap); set `-1' to disable swap CpuShares int64 // CPU shares (relative weight vs. other containers) Cpuset string // Cpuset 0-2, 0,1 AttachStdin bool AttachStdout bool AttachStderr bool PortSpecs []string // Deprecated - Can be in the format of 8080/tcp ExposedPorts map[string]struct{} Tty bool // Attach standard streams to a tty, including stdin if it is not closed. OpenStdin bool // Open stdin StdinOnce bool // If true, close stdin after the 1 attached client disconnects. Env []string Cmd []string Image string // Name of the image as it was passed by the operator (eg. could be symbolic) Volumes map[string]struct{} WorkingDir string Entrypoint []string NetworkDisabled bool MacAddress string OnBuild []string } // DockerAuthConfigOld represents the deprecated ~/.dockercfg auth // configuration. // Taken from upstream Docker. type DockerAuthConfigOld struct { Username string `json:"username,omitempty"` Password string `json:"password,omitempty"` Auth string `json:"auth"` Email string `json:"email"` ServerAddress string `json:"serveraddress,omitempty"` } // DockerAuthConfig represents a config.json auth entry. // Taken from upstream Docker. type DockerAuthConfig struct { Username string `json:"username,omitempty"` Password string `json:"password,omitempty"` Auth string `json:"auth,omitempty"` ServerAddress string `json:"serveraddress,omitempty"` RegistryToken string `json:"registrytoken,omitempty"` } // DockerConfigFile represents a config.json auth file. // Taken from upstream docker. type DockerConfigFile struct { AuthConfigs map[string]DockerAuthConfig `json:"auths"` } docker2aci-0.12.0/lib/internal/types/types.go000066400000000000000000000016451273674375500210520ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // Package types defines Docker image, URL and configuration types. // // Note: this package is an implementation detail and shouldn't be used outside // of docker2aci. package types // ParsedDockerURL represents a parsed Docker URL. type ParsedDockerURL struct { IndexURL string ImageName string Tag string Digest string } docker2aci-0.12.0/lib/internal/typesV2/000077500000000000000000000000001273674375500175615ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/typesV2/docker_types.go000066400000000000000000000107621273674375500226110ustar00rootroot00000000000000// Copyright 2016 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package typesV2 import ( "encoding/json" "errors" ) const ( MediaTypeDockerV21Manifest = "application/vnd.docker.distribution.manifest.v1+json" MediaTypeDockerV21SignedManifest = "application/vnd.docker.distribution.manifest.v1+prettyjws" MediaTypeDockerV21ManifestLayer = "application/vnd.docker.container.image.rootfs.diff+x-gtar" MediaTypeDockerV22Manifest = "application/vnd.docker.distribution.manifest.v2+json" MediaTypeDockerV22ManifestList = "application/vnd.docker.distribution.manifest.list.v2+json" MediaTypeDockerV22Config = "application/vnd.docker.container.image.v1+json" MediaTypeDockerV22RootFS = "application/vnd.docker.image.rootfs.diff.tar.gzip" MediaTypeOCIManifest = "application/vnd.oci.image.manifest.v1+json" MediaTypeOCIManifestList = "application/vnd.oci.image.manifest.list.v1+json" MediaTypeOCIConfig = "application/vnd.oci.image.serialization.config.v1+json" MediaTypeOCIRootFS = "application/vnd.oci.image.serialization.rootfs.tar.gzip" MediaTypeOCICombined = "application/vnd.oci.image.serialization.combined.v1+json" ) var ( ErrIncorrectMediaType = errors.New("incorrect mediaType") ErrMissingConfig = errors.New("the config field is empty") ErrMissingLayers = errors.New("the layers field is empty") ) type ImageManifest struct { SchemaVersion int `json:"schemaVersion"` MediaType string `json:"mediaType"` Config *ImageManifestDigest `json:"config"` Layers []*ImageManifestDigest `json:"layers"` Annotations map[string]string `json:"annotations"` } type ImageManifestDigest struct { MediaType string `json:"mediaType"` Size int `json:"size"` Digest string `json:"digest"` } func (im *ImageManifest) String() string { manblob, err := json.Marshal(im) if err != nil { return err.Error() } return string(manblob) } func (im *ImageManifest) PrettyString() string { manblob, err := json.MarshalIndent(im, "", " ") if err != nil { return err.Error() } return string(manblob) } func (im *ImageManifest) Validate() error { if im.MediaType != MediaTypeDockerV22Manifest && im.MediaType != MediaTypeOCIManifest { return ErrIncorrectMediaType } if im.Config == nil { return ErrMissingConfig } if len(im.Layers) == 0 { return ErrMissingLayers } return nil } type ImageConfig struct { Created string `json:"created"` Author string `json:"author"` Architecture string `json:"architecture"` OS string `json:"os"` Config *ImageConfigConfig `json:"config"` RootFS *ImageConfigRootFS `json:"rootfs"` History []*ImageConfigHistory `json:"history"` } type ImageConfigConfig struct { User string `json:"User"` Memory int `json:"Memory"` MemorySwap int `json:"MemorySwap"` CpuShares int `json:"CpuShares"` ExposedPorts map[string]struct{} `json:"ExposedPorts"` Env []string `json:"Env"` Entrypoint []string `json:"Entrypoint"` Cmd []string `json:"Cmd"` Volumes map[string]struct{} `json:"Volumes"` WorkingDir string `json:"WorkingDir"` } type ImageConfigRootFS struct { DiffIDs []string `json:"diff_ids"` Type string `json:"type"` } type ImageConfigHistory struct { Created string `json:"created,omitempty"` Author string `json:"author,omitempty"` CreatedBy string `json:"created_by,omitempty"` Comment string `json:"comment,omitempty"` EmptyLayer bool `json:"empty_layer,omitempty"` } func (ic *ImageConfig) String() string { manblob, err := json.Marshal(ic) if err != nil { return err.Error() } return string(manblob) } func (ic *ImageConfig) PrettyString() string { manblob, err := json.MarshalIndent(ic, "", " ") if err != nil { return err.Error() } return string(manblob) } docker2aci-0.12.0/lib/internal/util/000077500000000000000000000000001273674375500171625ustar00rootroot00000000000000docker2aci-0.12.0/lib/internal/util/util.go000066400000000000000000000043701273674375500204720ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // Package util defines convenience functions for handling slices and debugging. // // Note: this package is an implementation detail and shouldn't be used outside // of docker2aci. package util import ( "crypto/tls" "fmt" "net/http" "github.com/appc/spec/pkg/acirenderer" ) // Quote takes a slice of strings and returns another slice with them quoted. func Quote(l []string) []string { var quoted []string for _, s := range l { quoted = append(quoted, fmt.Sprintf("%q", s)) } return quoted } // ReverseImages takes an acirenderer.Images and reverses it. func ReverseImages(s acirenderer.Images) acirenderer.Images { var o acirenderer.Images for i := len(s) - 1; i >= 0; i-- { o = append(o, s[i]) } return o } // In checks whether el is in list. func In(list []string, el string) bool { return IndexOf(list, el) != -1 } // IndexOf returns the index of el in list, or -1 if it's not found. func IndexOf(list []string, el string) int { for i, x := range list { if el == x { return i } } return -1 } // GetTLSClient gets an HTTP client that behaves like the default HTTP // client, but optionally skips the TLS certificate verification. func GetTLSClient(skipTLSCheck bool) *http.Client { if !skipTLSCheck { return http.DefaultClient } client := *http.DefaultClient // Default transport is hidden behind the RoundTripper // interface, so we can't easily make a copy of it. If this // ever panics, we will have to adapt. realTransport := http.DefaultTransport.(*http.Transport) tr := *realTransport if tr.TLSClientConfig == nil { tr.TLSClientConfig = &tls.Config{} } tr.TLSClientConfig.InsecureSkipVerify = true client.Transport = &tr return &client } docker2aci-0.12.0/lib/version.go000066400000000000000000000013201273674375500164010ustar00rootroot00000000000000// Copyright 2016 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package docker2aci import "github.com/appc/spec/schema" var Version = "0.12.0" var AppcVersion = schema.AppContainerVersion docker2aci-0.12.0/main.go000066400000000000000000000140161273674375500151000ustar00rootroot00000000000000// Copyright 2015 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package main import ( "flag" "fmt" "net/url" "os" "strings" "github.com/appc/docker2aci/lib" "github.com/appc/docker2aci/lib/common" "github.com/appc/docker2aci/pkg/log" "github.com/appc/spec/aci" "github.com/appc/spec/schema" ) var ( flagNoSquash bool flagImage string flagDebug bool flagInsecureSkipVerify bool flagInsecureAllowHTTP bool flagCompression string flagVersion bool ) func init() { flag.BoolVar(&flagNoSquash, "nosquash", false, "Don't squash layers and output every layer as ACI") flag.StringVar(&flagImage, "image", "", "When converting a local file, it selects a particular image to convert. Format: IMAGE_NAME[:TAG]") flag.BoolVar(&flagDebug, "debug", false, "Enables debug messages") flag.BoolVar(&flagInsecureSkipVerify, "insecure-skip-verify", false, "Don't verify certificates when fetching images") flag.BoolVar(&flagInsecureAllowHTTP, "insecure-allow-http", false, "Uses unencrypted connections when fetching images") flag.StringVar(&flagCompression, "compression", "gzip", "Type of compression to use; allowed values: gzip, none") flag.BoolVar(&flagVersion, "version", false, "Print version") } func printVersion() { fmt.Println("docker2aci version", docker2aci.Version) fmt.Println("appc version", docker2aci.AppcVersion) } func runDocker2ACI(arg string) error { if flagDebug { log.InitDebug() } squash := !flagNoSquash var aciLayerPaths []string // try to convert a local file u, err := url.Parse(arg) if err != nil { return fmt.Errorf("error parsing argument: %v", err) } var compression common.Compression switch flagCompression { case "none": compression = common.NoCompression case "gzip": compression = common.GzipCompression default: return fmt.Errorf("unknown compression method: %s", flagCompression) } cfg := docker2aci.CommonConfig{ Squash: squash, OutputDir: ".", TmpDir: os.TempDir(), Compression: compression, } if u.Scheme == "docker" { if flagImage != "" { return fmt.Errorf("flag --image works only with files.") } dockerURL := strings.TrimPrefix(arg, "docker://") indexServer := docker2aci.GetIndexName(dockerURL) var username, password string username, password, err = docker2aci.GetDockercfgAuth(indexServer) if err != nil { return fmt.Errorf("error reading .dockercfg file: %v", err) } remoteConfig := docker2aci.RemoteConfig{ CommonConfig: cfg, Username: username, Password: password, Insecure: common.InsecureConfig{ SkipVerify: flagInsecureSkipVerify, AllowHTTP: flagInsecureAllowHTTP, }, } aciLayerPaths, err = docker2aci.ConvertRemoteRepo(dockerURL, remoteConfig) } else { fileConfig := docker2aci.FileConfig{ CommonConfig: cfg, DockerURL: flagImage, } aciLayerPaths, err = docker2aci.ConvertSavedFile(arg, fileConfig) if serr, ok := err.(*common.ErrSeveralImages); ok { err = fmt.Errorf("%s, use option --image with one of:\n\n%s", serr, strings.Join(serr.Images, "\n")) } } if err != nil { return fmt.Errorf("conversion error: %v", err) } // we get last layer's manifest, this will include all the elements in the // previous layers. If we're squashing, the last element of aciLayerPaths // will be the squashed image. manifest, err := getManifest(aciLayerPaths[len(aciLayerPaths)-1]) if err != nil { return err } if err := printConvertedVolumes(*manifest); err != nil { return err } if err := printConvertedPorts(*manifest); err != nil { return err } fmt.Printf("\nGenerated ACI(s):\n") for _, aciFile := range aciLayerPaths { fmt.Println(aciFile) } return nil } func printConvertedVolumes(manifest schema.ImageManifest) error { if manifest.App != nil && manifest.App.MountPoints != nil { mps := manifest.App.MountPoints if len(mps) > 0 { fmt.Printf("\nConverted volumes:\n") for _, mp := range mps { fmt.Printf("\tname: %q, path: %q, readOnly: %v\n", mp.Name, mp.Path, mp.ReadOnly) } } } return nil } func printConvertedPorts(manifest schema.ImageManifest) error { if manifest.App != nil && manifest.App.Ports != nil { ports := manifest.App.Ports if len(ports) > 0 { fmt.Printf("\nConverted ports:\n") for _, port := range ports { fmt.Printf("\tname: %q, protocol: %q, port: %v, count: %v, socketActivated: %v\n", port.Name, port.Protocol, port.Port, port.Count, port.SocketActivated) } } } return nil } func getManifest(aciPath string) (*schema.ImageManifest, error) { f, err := os.Open(aciPath) if err != nil { return nil, fmt.Errorf("error opening converted image: %v", err) } defer f.Close() manifest, err := aci.ManifestFromImage(f) if err != nil { return nil, fmt.Errorf("error reading manifest from converted image: %v", err) } return manifest, nil } func usage() { fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0]) fmt.Fprintf(os.Stderr, "docker2aci [--debug] [--nosquash] [--compression=(gzip|none)] IMAGE\n") fmt.Fprintf(os.Stderr, " Where IMAGE is\n") fmt.Fprintf(os.Stderr, " [--image=IMAGE_NAME[:TAG]] FILEPATH\n") fmt.Fprintf(os.Stderr, " or\n") fmt.Fprintf(os.Stderr, " docker://[REGISTRYURL/]IMAGE_NAME[:TAG]\n") fmt.Fprintf(os.Stderr, "Flags:\n") flag.PrintDefaults() } func main() { flag.Usage = usage flag.Parse() args := flag.Args() if flagVersion { printVersion() return } if len(args) < 1 { usage() os.Exit(2) } if err := runDocker2ACI(args[0]); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) os.Exit(1) } } docker2aci-0.12.0/pkg/000077500000000000000000000000001273674375500144045ustar00rootroot00000000000000docker2aci-0.12.0/pkg/log/000077500000000000000000000000001273674375500151655ustar00rootroot00000000000000docker2aci-0.12.0/pkg/log/log.go000066400000000000000000000021161273674375500162750ustar00rootroot00000000000000// Copyright 2016 The appc Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package log import ( "fmt" "io" "os" "strings" ) var debugEnabled bool func printTo(w io.Writer, i ...interface{}) { s := fmt.Sprint(i...) fmt.Fprintln(w, strings.TrimSuffix(s, "\n")) } // Info prints a message to stderr. func Info(i ...interface{}) { printTo(os.Stderr, i...) } // Debug prints a message to stderr if debug is enabled. func Debug(i ...interface{}) { if debugEnabled { printTo(os.Stderr, i...) } } // InitDebug enables debug output. func InitDebug() { debugEnabled = true } docker2aci-0.12.0/scripts/000077500000000000000000000000001273674375500153125ustar00rootroot00000000000000docker2aci-0.12.0/scripts/bump-release000077500000000000000000000023501273674375500176210ustar00rootroot00000000000000#!/bin/bash -e # # Attempt to bump the docker2aci release to the specified version by replacing # all occurrences of the current/previous version. # # Generates two commits: the release itself and the bump to the next +git # version # # YMMV, no disclaimer or warranty, etc. # make sure we are running in a toplevel directory if ! [[ "$0" =~ "scripts/bump-release" ]]; then echo "This script must be run in a toplevel docker2aci directory" exit 255 fi if ! [[ "$1" =~ ^v[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]$ ]]; then echo "Usage: scripts/bump-release " echo " where VERSION must be vX.Y.Z" exit 255 fi function replace_stuff() { local FROM local TO local REPLACE FROM=$1 TO=$2 # escape special characters REPLACE=$(sed -e 's/[]\/$*.^|[]/\\&/g'<<< $FROM) shift 2 echo $* | xargs sed -i --follow-symlinks -e "s/$REPLACE/$TO/g" } function replace_version() { replace_stuff $1 $2 lib/version.go } NEXT=${1:1} # 0.2.3 NEXTGIT="${NEXT}+git" # 0.2.3+git PREVGIT=$(grep -Po 'var Version = "\K[^"]*(?=")' lib/version.go) # 0.1.2+git PREV=${PREVGIT::-4} # 0.1.2 replace_version $PREVGIT $NEXT git commit -am "version: bump to v${NEXT}" replace_version $NEXT $NEXTGIT git commit -am "version: bump to v${NEXTGIT}" docker2aci-0.12.0/tests/000077500000000000000000000000001273674375500147655ustar00rootroot00000000000000docker2aci-0.12.0/tests/README.md000066400000000000000000000012721273674375500162460ustar00rootroot00000000000000# docker2aci tests ## Semaphore The tests run on the [Semaphore](https://semaphoreci.com/) CI system. The tests are executed on Semaphore at each Pull Request (PR). Each GitHub PR page should have a link to the [test results on Semaphore](https://semaphoreci.com/appc/docker2aci). ### Build settings The tests will run on two VMs. The "Setup" and "Post thread" sections will be executed on both VMs. The "Thread 1" and "Thread 2" will be executed in parallel in separate VMs. #### Setup ``` ./build.sh ``` #### Thread 1 ``` ./tests/test.sh ``` ### Platform Select `Ubuntu 14.04 LTS v1503 (beta with Docker support)`. The platform with *Docker support* means the tests will run in a VM. docker2aci-0.12.0/tests/rkt-v1.1.0.md5sum000066400000000000000000000000731273674375500175420ustar00rootroot00000000000000MD5 (rkt-v1.1.0.tar.gz) = d3d9d62429e53d8f631dbec93e4e719f docker2aci-0.12.0/tests/test-basic/000077500000000000000000000000001273674375500170235ustar00rootroot00000000000000docker2aci-0.12.0/tests/test-basic/Dockerfile000066400000000000000000000002061273674375500210130ustar00rootroot00000000000000FROM busybox COPY check.sh / RUN echo file1 > file1 ; ln file1 file2 RUN echo file3 > file3 RUN echo file4 > file4 CMD /check.sh 2>&1 docker2aci-0.12.0/tests/test-basic/check.sh000077500000000000000000000006071273674375500204420ustar00rootroot00000000000000#!/bin/sh set -e set -x grep -q file1 file1 grep -q file1 file2 grep -q file3 file3 grep -q file4 file4 if [ "$CHECK" != "rkt-rendered" ] ; then # Skip this test because of: # https://github.com/coreos/rkt/issues/1774 test $(ls -i file1 |awk '{print $1}') -eq $(ls -i file2 |awk '{print $1}') test $(ls -i file3 |awk '{print $1}') -ne $(ls -i file4 |awk '{print $1}') fi echo "SUCCESS" docker2aci-0.12.0/tests/test-pwl/000077500000000000000000000000001273674375500165445ustar00rootroot00000000000000docker2aci-0.12.0/tests/test-pwl/Dockerfile000066400000000000000000000001241273674375500205330ustar00rootroot00000000000000FROM gcr.io/google_containers/nginx:1.7.9 COPY check.sh / ENTRYPOINT /check.sh 2>&1 docker2aci-0.12.0/tests/test-pwl/check.sh000077500000000000000000000000671273674375500201630ustar00rootroot00000000000000#!/bin/sh set -e set -x ls -l /var/run echo "SUCCESS" docker2aci-0.12.0/tests/test-whiteouts/000077500000000000000000000000001273674375500177755ustar00rootroot00000000000000docker2aci-0.12.0/tests/test-whiteouts/Dockerfile000066400000000000000000000022131273674375500217650ustar00rootroot00000000000000FROM busybox COPY check.sh / RUN echo yes > layer0-file1 ; ln layer0-file1 layer0-file2 ; ln layer0-file1 layer0-file3 RUN echo yes > layer1-file1 ; ln layer1-file1 layer1-file2 ; ln layer1-file1 layer1-file3 RUN echo yes > layer2-file1 ; ln layer2-file1 layer2-file2 ; ln layer2-file1 layer2-file3 RUN echo yes > layer3-file1 ; ln layer3-file1 layer3-file2 ; ln layer3-file1 layer3-file3 RUN rm -f layer1-file1 layer2-file2 layer3-file3 RUN echo yes > layer4-file1 ; ln layer4-file1 layer4-file2 ; ln layer4-file1 layer4-file3 RUN echo yes > layer5-file1 ; ln layer5-file1 layer5-file2 ; ln layer5-file1 layer5-file3 RUN echo yes > layer6-file1 ; ln layer6-file1 layer6-file2 ; ln layer6-file1 layer6-file3 RUN rm -f layer4-file2 layer5-file1 layer6-file1 layer4-file3 layer5-file3 layer6-file2 RUN echo OLD > layer10-file1 ; ln layer10-file1 layer10-file2 ; ln layer10-file1 layer10-file3 RUN echo NEW > layer10-file1 ; ln -f layer10-file1 layer10-file2 ; ln -f layer10-file1 layer10-file3 ; echo foo > foo RUN echo line1 > layer11-file1 ; ln layer11-file1 layer11-file2 ; ln layer11-file1 layer11-file3 RUN echo line2 >> layer11-file1 CMD /check.sh 2>&1 docker2aci-0.12.0/tests/test-whiteouts/check.sh000077500000000000000000000023501273674375500214110ustar00rootroot00000000000000#!/bin/sh set -e set -x grep -q yes layer0-file1 grep -q yes layer0-file2 grep -q yes layer0-file3 test ! -e layer1-file1 test -e layer1-file2 test -e layer1-file3 test -e layer2-file1 test ! -e layer2-file2 test -e layer2-file3 test -e layer3-file1 test -e layer3-file2 test ! -e layer3-file3 grep -q yes layer1-file2 grep -q yes layer1-file3 grep -q yes layer2-file1 grep -q yes layer2-file3 grep -q yes layer3-file1 grep -q yes layer3-file2 test -e layer4-file1 test ! -e layer4-file2 test ! -e layer4-file3 test ! -e layer5-file1 test -e layer5-file2 test ! -e layer5-file3 test ! -e layer6-file1 test ! -e layer6-file2 test -e layer6-file3 grep -q yes layer4-file1 grep -q yes layer5-file2 grep -q yes layer6-file3 grep -q NEW layer10-file1 grep -q NEW layer10-file2 grep -q NEW layer10-file3 grep -q line1 layer11-file1 grep -q line1 layer11-file2 grep -q line1 layer11-file3 # # Docker with AUFS or overlay storage backend does not handle this test # # correctly and Semaphore uses AUFS if [ "$DOCKER_STORAGE_BACKEND" == devicemapper ] ; then grep -q line2 layer11-file1 grep -q line2 layer11-file2 grep -q line2 layer11-file3 cmp layer11-file1 layer11-file2 cmp layer11-file1 layer11-file3 fi echo "SUCCESS" docker2aci-0.12.0/tests/test.sh000077500000000000000000000042571273674375500163130ustar00rootroot00000000000000#!/bin/bash set -e DOCKER2ACI=../bin/docker2aci PREFIX=docker2aci-tests TESTDIR=$(dirname $(realpath $0)) RKTVERSION=v1.1.0 cd $TESTDIR # install rkt in Semaphore if ! which rkt > /dev/null ; then if [ "$SEMAPHORE" != "true" ] ; then echo "Please install rkt" exit 1 fi pushd $SEMAPHORE_CACHE_DIR if ! md5sum -c $TESTDIR/rkt-$RKTVERSION.md5sum; then wget https://github.com/coreos/rkt/releases/download/$RKTVERSION/rkt-$RKTVERSION.tar.gz fi md5sum -c $TESTDIR/rkt-$RKTVERSION.md5sum tar xf rkt-$RKTVERSION.tar.gz export PATH=$PATH:$PWD/rkt-$RKTVERSION/ popd fi RKT=$(which rkt) DOCKER_STORAGE_BACKEND=$(sudo docker info|grep '^Storage Driver:'|sed 's/Storage Driver: //') for i in $(find . -maxdepth 1 -type d -name 'test-*') ; do TESTNAME=$(basename $i) echo "### Test case ${TESTNAME}: build..." sudo docker build --tag=$PREFIX/${TESTNAME} --no-cache=true ${TESTNAME} echo "### Test case ${TESTNAME}: test in Docker..." sudo docker run --rm \ --env=CHECK=docker-run \ --env=DOCKER_STORAGE_BACKEND=$DOCKER_STORAGE_BACKEND \ $PREFIX/${TESTNAME} echo "### Test case ${TESTNAME}: converting to ACI..." sudo docker save -o ${TESTNAME}.docker $PREFIX/${TESTNAME} # Docker now writes files as root, so make them readable sudo chmod o+rx ${TESTNAME}.docker $DOCKER2ACI ${TESTNAME}.docker echo "### Test case ${TESTNAME}: test in rkt..." sudo $RKT prepare --insecure-options=image \ --set-env=CHECK=rkt-run \ --set-env=DOCKER_STORAGE_BACKEND=$DOCKER_STORAGE_BACKEND \ ./${PREFIX}-${TESTNAME}-latest.aci \ > rkt-uuid-${TESTNAME} sudo $RKT run-prepared $(cat rkt-uuid-${TESTNAME}) sudo $RKT status $(cat rkt-uuid-${TESTNAME}) | grep app-${TESTNAME}=0 sudo $RKT rm $(cat rkt-uuid-${TESTNAME}) echo "### Test case ${TESTNAME}: test with 'rkt image render'..." sudo $RKT image render --overwrite ${PREFIX}/${TESTNAME} ./rendered-${TESTNAME} pushd rendered-${TESTNAME}/rootfs CHECK=rkt-rendered DOCKER_STORAGE_BACKEND=$DOCKER_STORAGE_BACKEND $TESTDIR/${TESTNAME}/check.sh popd echo "### Test case ${TESTNAME}: SUCCESS" sudo docker rmi $PREFIX/${TESTNAME} done