1
0
Fork 0

@actions/artifact download artifacts (#340)

* Download Artifacts using @actions/artifact
pull/344/head
Konrad Pabjan 2020-02-13 18:24:11 -05:00 committed by GitHub
parent 84f1e31b69
commit f383109dc3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 1416 additions and 18 deletions

View File

@ -0,0 +1,199 @@
# `@actions/artifact`
## Usage
You can use this package to interact with the actions artifacts.
- [Upload an Artifact](#Upload-an-Artifact)
- [Download a Single Artifact](#Download-a-Single-Artifact)
- [Download All Artifacts](#Download-all-Artifacts)
- [Additional Documentation](#Additional-Documentation)
Relative paths and absolute paths are both allowed. Relative paths are rooted against the current working directory.
## Upload an Artifact
Method Name: `uploadArtifact`
#### Inputs
- `name`
- The name of the artifact that is being uploaded
- Required
- `files`
- A list of file paths that describe what should be uploaded as part of the artifact
- If a path is provided that does not exist, an error will be thrown
- Can be absolute or relative. Internally everything is normalized and resolved
- Required
- `rootDirectory`
- A file path that denotes the root directory of the files being uploaded. This path is used to strip the paths provided in `files` to control how they are uploaded and structured
- If a file specified in `files` is not in the `rootDirectory`, an error will be thrown
- Required
- `options`
- Extra options that allow for the customization of the upload behavior
- Optional
#### Available Options
- `continueOnError`
- Indicates if the artifact upload should continue in the event a file fails to upload. If there is a error during upload, a partial artifact will always be created and available for download at the end. The `size` reported will be the amount of storage that the user or org will be charged for the partial artifact.
- If set to `false`, and an error is encountered, all other uploads will stop and any files that were queued will not be attempted to be uploaded. The partial artifact available will only include files up until the failure.
- If set to `true` and an error is encountered, the failed file will be skipped and ignored and all other queued files will be attempted to be uploaded. There will be an artifact available for download at the end with everything excluding the file that failed to upload
- Optional, defaults to `true` if not specified
#### Example using Absolute File Paths
```js
const artifact = require('@actions/artifact');
const artifactClient = artifact.create()
const artifactName = 'my-artifact';
const files = [
'/home/user/files/plz-upload/file1.txt',
'/home/user/files/plz-upload/file2.txt',
'/home/user/files/plz-upload/dir/file3.txt'
]
const rootDirectory = '/home/user/files/plz-upload'
const options = {
continueOnError: true
}
const uploadResult = await artifactClient.uploadArtifact(artifactName, files, rootDirectory, options)
```
#### Example using Relative File Paths
```js
// Assuming the current working directory is /home/user/files/plz-upload
const artifact = require('@actions/artifact');
const artifactClient = artifact.create()
const artifactName = 'my-artifact';
const files = [
'file1.txt',
'file2.txt',
'dir/file3.txt'
]
const rootDirectory = '.' // Also possible to use __dirname
const options = {
continueOnError: false
}
const uploadResponse = await artifactClient.uploadArtifact(artifactName, files, rootDirectory, options)
```
#### Upload Result
The returned `UploadResponse` will contain the following information
- `artifactName`
- The name of the artifact that was uploaded
- `artifactItems`
- A list of all files that describe what is uploaded if there are no errors encountered. Usually this will be equal to the inputted `files` with the exception of empty directories (will not be uploaded)
- `size`
- Total size of the artifact that was uploaded in bytes
- `failedItems`
- A list of items that were not uploaded successfully (this will include queued items that were not uploaded if `continueOnError` is set to false). This is a subset of `artifactItems`
## Download a Single Artifact
Method Name: `downloadArtifact`
#### Inputs
- `name`
- The name of the artifact to download
- Required
- `path`
- Path that denotes where the artifact will be downloaded to
- Optional. Defaults to the GitHub workspace directory(`$GITHUB_WORKSPACE`) if not specified
- `options`
- Extra options that allow for the customization of the download behavior
- Optional
#### Available Options
- `createArtifactFolder`
- Specifies if a folder (the artifact name) is created for the artifact that is downloaded (contents downloaded into this folder),
- Optional. Defaults to false if not specified
#### Example
```js
const artifact = require('@actions/artifact');
const artifactClient = artifact.create()
const artifactName = 'my-artifact';
const path = 'some/directory'
const options = {
createArtifactFolder: false
}
const downloadResponse = await artifactClient.downloadArtifact(artifactName, path, options)
// Post download, the directory structure will look like this
/some
/directory
/file1.txt
/file2.txt
/dir
/file3.txt
// If createArtifactFolder is set to true, the directory structure will look like this
/some
/directory
/my-artifact
/file1.txt
/file2.txt
/dir
/file3.txt
```
#### Download Response
The returned `DownloadResponse` will contain the following information
- `artifactName`
- The name of the artifact that was downloaded
- `downloadPath`
- The full Path to where the artifact was downloaded
## Download All Artifacts
Method Name: `downloadAllArtifacts`
#### Inputs
- `path`
- Path that denotes where the artifact will be downloaded to
- Optional. Defaults to the GitHub workspace directory(`$GITHUB_WORKSPACE`) if not specified
```js
const artifact = require('@actions/artifact');
const artifactClient = artifact.create();
const downloadResponse = await artifactClient.downloadAllArtifacts();
// output result
for (response in downloadResponse) {
console.log(response.artifactName);
console.log(response.downloadPath);
}
```
Because there are multiple artifacts, an extra directory (denoted by the name of the artifact) will be created for each artifact in the path. With 2 artifacts(`my-artifact-1` and `my-artifact-2` for example) and the default path, the directory structure will be as follows:
```js
/GITHUB_WORKSPACE
/my-artifact-1
/ .. contents of `my-artifact-1`
/my-artifact-2
/ .. contents of `my-artifact-2`
```
#### Download Result
An array will be returned that describes the results for downloading all artifacts. The number of items in the array indicates the number of artifacts that were downloaded.
Each artifact will have the same `DownloadResponse` as if it was individually downloaded
- `artifactName`
- The name of the artifact that was downloaded
- `downloadPath`
- The full Path to where the artifact was downloaded
## Additional Documentation
Check out [additional-information](additional-information.md) for extra documentation.

View File

@ -0,0 +1,5 @@
# @actions/artifact Releases
### 0.1.0
- Initial release

View File

@ -0,0 +1,499 @@
import * as path from 'path'
import * as core from '@actions/core'
import {URL} from 'url'
import {getDownloadSpecification} from '../src/internal-download-specification'
import {ContainerEntry} from '../src/internal-contracts'
const artifact1Name = 'my-artifact'
const artifact2Name = 'my-artifact-extra'
// Populating with only the information that is necessary
function getPartialContainerEntry(): ContainerEntry {
return {
containerId: 10,
scopeIdentifier: '00000000-0000-0000-0000-000000000000',
path: 'ADD_INFORMATION',
itemType: 'ADD_INFORMATION',
status: 'created',
dateCreated: '2020-02-06T22:13:35.373Z',
dateLastModified: '2020-02-06T22:13:35.453Z',
createdBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
lastModifiedBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
itemLocation: 'ADD_INFORMATION',
contentLocation: 'ADD_INFORMATION',
contentId: ''
}
}
function createFileEntry(entryPath: string): ContainerEntry {
const newFileEntry = getPartialContainerEntry()
newFileEntry.path = entryPath
newFileEntry.itemType = 'file'
newFileEntry.itemLocation = createItemLocation(entryPath)
newFileEntry.contentLocation = createContentLocation(entryPath)
return newFileEntry
}
function createDirectoryEntry(directoryPath: string): ContainerEntry {
const newDirectoryEntry = getPartialContainerEntry()
newDirectoryEntry.path = directoryPath
newDirectoryEntry.itemType = 'folder'
newDirectoryEntry.itemLocation = createItemLocation(directoryPath)
newDirectoryEntry.contentLocation = createContentLocation(directoryPath)
return newDirectoryEntry
}
function createItemLocation(relativePath: string): string {
const itemLocation = new URL(
'https://testing/_apis/resources/Containers/10000'
)
itemLocation.searchParams.append('itemPath', relativePath)
itemLocation.searchParams.append('metadata', 'true')
return itemLocation.toString()
}
function createContentLocation(relativePath: string): string {
const itemLocation = new URL(
'https://testing/_apis/resources/Containers/10000'
)
itemLocation.searchParams.append('itemPath', relativePath)
return itemLocation.toString()
}
/*
Represents a set of container entries for two artifacts with the following directory structure
/my-artifact
/file1.txt
/file2.txt
/dir1
/file3.txt
/dir2
/dir3
/dir4
file4.txt
file5.txt
/my-artifact-extra
/file1.txt
*/
// main artfact
const file1Path = path.join(artifact1Name, 'file1.txt')
const file2Path = path.join(artifact1Name, 'file2.txt')
const dir1Path = path.join(artifact1Name, 'dir1')
const file3Path = path.join(dir1Path, 'file3.txt')
const dir2Path = path.join(dir1Path, 'dir2')
const dir3Path = path.join(dir2Path, 'dir3')
const dir4Path = path.join(dir3Path, 'dir4')
const file4Path = path.join(dir4Path, 'file4.txt')
const file5Path = path.join(dir4Path, 'file5.txt')
const rootDirectoryEntry = createDirectoryEntry(artifact1Name)
const directoryEntry1 = createDirectoryEntry(dir1Path)
const directoryEntry2 = createDirectoryEntry(dir2Path)
const directoryEntry3 = createDirectoryEntry(dir3Path)
const directoryEntry4 = createDirectoryEntry(dir4Path)
const fileEntry1 = createFileEntry(file1Path)
const fileEntry2 = createFileEntry(file2Path)
const fileEntry3 = createFileEntry(file3Path)
const fileEntry4 = createFileEntry(file4Path)
const fileEntry5 = createFileEntry(file5Path)
// extra artifact
const artifact2File1Path = path.join(artifact2Name, 'file1.txt')
const rootDirectoryEntry2 = createDirectoryEntry(artifact2Name)
const extraFileEntry = createFileEntry(artifact2File1Path)
const artifactContainerEntries: ContainerEntry[] = [
rootDirectoryEntry,
fileEntry1,
fileEntry2,
directoryEntry1,
fileEntry3,
directoryEntry2,
directoryEntry3,
directoryEntry4,
fileEntry4,
fileEntry5,
rootDirectoryEntry2,
extraFileEntry
]
describe('Search', () => {
beforeAll(async () => {
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
})
it('Download Specification - Absolute Path with no root directory', () => {
const testDownloadPath = path.join(
__dirname,
'some',
'destination',
'folder'
)
const specification = getDownloadSpecification(
artifact1Name,
artifactContainerEntries,
testDownloadPath,
false
)
expect(specification.rootDownloadLocation).toEqual(testDownloadPath)
expect(specification.filesToDownload.length).toEqual(5)
const item1ExpectedTargetPath = path.join(testDownloadPath, 'file1.txt')
const item2ExpectedTargetPath = path.join(testDownloadPath, 'file2.txt')
const item3ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'file3.txt'
)
const item4ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file4.txt'
)
const item5ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file5.txt'
)
const targetLocations = specification.filesToDownload.map(
item => item.targetPath
)
expect(targetLocations).toContain(item1ExpectedTargetPath)
expect(targetLocations).toContain(item2ExpectedTargetPath)
expect(targetLocations).toContain(item3ExpectedTargetPath)
expect(targetLocations).toContain(item4ExpectedTargetPath)
expect(targetLocations).toContain(item5ExpectedTargetPath)
for (const downloadItem of specification.filesToDownload) {
if (downloadItem.targetPath === item1ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file1Path)
)
} else if (downloadItem.targetPath === item2ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file2Path)
)
} else if (downloadItem.targetPath === item3ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file3Path)
)
} else if (downloadItem.targetPath === item4ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file4Path)
)
} else if (downloadItem.targetPath === item5ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file5Path)
)
} else {
throw new Error('this should never be reached')
}
}
expect(specification.directoryStructure.length).toEqual(3)
expect(specification.directoryStructure).toContain(testDownloadPath)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, 'dir1')
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, 'dir1', 'dir2', 'dir3', 'dir4')
)
})
it('Download Specification - Relative Path with no root directory', () => {
const testDownloadPath = path.join('some', 'destination', 'folder')
const specification = getDownloadSpecification(
artifact1Name,
artifactContainerEntries,
testDownloadPath,
false
)
expect(specification.rootDownloadLocation).toEqual(testDownloadPath)
expect(specification.filesToDownload.length).toEqual(5)
const item1ExpectedTargetPath = path.join(testDownloadPath, 'file1.txt')
const item2ExpectedTargetPath = path.join(testDownloadPath, 'file2.txt')
const item3ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'file3.txt'
)
const item4ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file4.txt'
)
const item5ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file5.txt'
)
const targetLocations = specification.filesToDownload.map(
item => item.targetPath
)
expect(targetLocations).toContain(item1ExpectedTargetPath)
expect(targetLocations).toContain(item2ExpectedTargetPath)
expect(targetLocations).toContain(item3ExpectedTargetPath)
expect(targetLocations).toContain(item4ExpectedTargetPath)
expect(targetLocations).toContain(item5ExpectedTargetPath)
for (const downloadItem of specification.filesToDownload) {
if (downloadItem.targetPath === item1ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file1Path)
)
} else if (downloadItem.targetPath === item2ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file2Path)
)
} else if (downloadItem.targetPath === item3ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file3Path)
)
} else if (downloadItem.targetPath === item4ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file4Path)
)
} else if (downloadItem.targetPath === item5ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file5Path)
)
} else {
throw new Error('this should never be reached')
}
}
expect(specification.directoryStructure.length).toEqual(3)
expect(specification.directoryStructure).toContain(testDownloadPath)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, 'dir1')
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, 'dir1', 'dir2', 'dir3', 'dir4')
)
})
it('Download Specification - Absolute Path with root directory', () => {
const testDownloadPath = path.join(
__dirname,
'some',
'destination',
'folder'
)
const specification = getDownloadSpecification(
artifact1Name,
artifactContainerEntries,
testDownloadPath,
true
)
expect(specification.rootDownloadLocation).toEqual(
path.join(testDownloadPath, artifact1Name)
)
expect(specification.filesToDownload.length).toEqual(5)
const item1ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'file1.txt'
)
const item2ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'file2.txt'
)
const item3ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'file3.txt'
)
const item4ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file4.txt'
)
const item5ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file5.txt'
)
const targetLocations = specification.filesToDownload.map(
item => item.targetPath
)
expect(targetLocations).toContain(item1ExpectedTargetPath)
expect(targetLocations).toContain(item2ExpectedTargetPath)
expect(targetLocations).toContain(item3ExpectedTargetPath)
expect(targetLocations).toContain(item4ExpectedTargetPath)
expect(targetLocations).toContain(item5ExpectedTargetPath)
for (const downloadItem of specification.filesToDownload) {
if (downloadItem.targetPath === item1ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file1Path)
)
} else if (downloadItem.targetPath === item2ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file2Path)
)
} else if (downloadItem.targetPath === item3ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file3Path)
)
} else if (downloadItem.targetPath === item4ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file4Path)
)
} else if (downloadItem.targetPath === item5ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file5Path)
)
} else {
throw new Error('this should never be reached')
}
}
expect(specification.directoryStructure.length).toEqual(3)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, artifact1Name)
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, dir1Path)
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, dir4Path)
)
})
it('Download Specification - Relative Path with root directory', () => {
const testDownloadPath = path.join('some', 'destination', 'folder')
const specification = getDownloadSpecification(
artifact1Name,
artifactContainerEntries,
testDownloadPath,
true
)
expect(specification.rootDownloadLocation).toEqual(
path.join(testDownloadPath, artifact1Name)
)
expect(specification.filesToDownload.length).toEqual(5)
const item1ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'file1.txt'
)
const item2ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'file2.txt'
)
const item3ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'file3.txt'
)
const item4ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file4.txt'
)
const item5ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file5.txt'
)
const targetLocations = specification.filesToDownload.map(
item => item.targetPath
)
expect(targetLocations).toContain(item1ExpectedTargetPath)
expect(targetLocations).toContain(item2ExpectedTargetPath)
expect(targetLocations).toContain(item3ExpectedTargetPath)
expect(targetLocations).toContain(item4ExpectedTargetPath)
expect(targetLocations).toContain(item5ExpectedTargetPath)
for (const downloadItem of specification.filesToDownload) {
if (downloadItem.targetPath === item1ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file1Path)
)
} else if (downloadItem.targetPath === item2ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file2Path)
)
} else if (downloadItem.targetPath === item3ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file3Path)
)
} else if (downloadItem.targetPath === item4ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file4Path)
)
} else if (downloadItem.targetPath === item5ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file5Path)
)
} else {
throw new Error('this should never be reached')
}
}
expect(specification.directoryStructure.length).toEqual(3)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, artifact1Name)
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, dir1Path)
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, dir4Path)
)
})
})

View File

@ -0,0 +1,236 @@
import * as core from '@actions/core'
import * as http from 'http'
import * as io from '../../io/src/io'
import * as net from 'net'
import * as path from 'path'
import * as configVariables from '../src/internal-config-variables'
import {HttpClient, HttpClientResponse} from '@actions/http-client'
import * as downloadClient from '../src/internal-download-http-client'
import {
ListArtifactsResponse,
QueryArtifactResponse
} from '../src/internal-contracts'
const root = path.join(__dirname, '_temp', 'artifact-download')
jest.mock('../src/internal-config-variables')
jest.mock('@actions/http-client')
describe('Download Tests', () => {
beforeAll(async () => {
await io.rmRF(root)
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
})
/**
* Test Listing Artifacts
*/
it('List Artifacts - Success', async () => {
setupSuccessfulListArtifactsResponse()
const artifacts = await downloadClient.listArtifacts()
expect(artifacts.count).toEqual(2)
const artifactNames = artifacts.value.map(item => item.name)
expect(artifactNames).toContain('artifact1-name')
expect(artifactNames).toContain('artifact2-name')
for (const artifact of artifacts.value) {
if (artifact.name === 'artifact1-name') {
expect(artifact.url).toEqual(
`${configVariables.getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=artifact1-name`
)
} else if (artifact.name === 'artifact2-name') {
expect(artifact.url).toEqual(
`${configVariables.getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=artifact2-name`
)
} else {
throw new Error(
'Invalid artifact combination. This should never be reached'
)
}
}
})
it('List Artifacts - Failure', async () => {
setupFailedResponse()
expect(downloadClient.listArtifacts()).rejects.toThrow(
'Unable to list artifacts for the run'
)
})
/**
* Test Container Items
*/
it('Container Items - Success', async () => {
setupSuccessfulContainerItemsResponse()
const response = await downloadClient.getContainerItems(
'artifact-name',
configVariables.getRuntimeUrl()
)
expect(response.count).toEqual(2)
const itemPaths = response.value.map(item => item.path)
expect(itemPaths).toContain('artifact-name')
expect(itemPaths).toContain('artifact-name/file1.txt')
for (const containerEntry of response.value) {
if (containerEntry.path === 'artifact-name') {
expect(containerEntry.itemType).toEqual('folder')
} else if (containerEntry.path === 'artifact-name/file1.txt') {
expect(containerEntry.itemType).toEqual('file')
} else {
throw new Error(
'Invalid container combination. This should never be reached'
)
}
}
})
it('Container Items - Failure', async () => {
setupFailedResponse()
expect(
downloadClient.getContainerItems(
'artifact-name',
configVariables.getRuntimeUrl()
)
).rejects.toThrow(
`Unable to get ContainersItems from ${configVariables.getRuntimeUrl()}`
)
})
/**
* Helper used to setup mocking for the HttpClient
*/
async function emptyMockReadBody(): Promise<string> {
return new Promise(resolve => {
resolve()
})
}
/**
* Setups up HTTP GET response for a successful listArtifacts() call
*/
function setupSuccessfulListArtifactsResponse(): void {
jest.spyOn(HttpClient.prototype, 'get').mockImplementationOnce(async () => {
const mockMessage = new http.IncomingMessage(new net.Socket())
let mockReadBody = emptyMockReadBody
mockMessage.statusCode = 201
const response: ListArtifactsResponse = {
count: 2,
value: [
{
containerId: '13',
size: -1,
signedContent: 'false',
fileContainerResourceUrl: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13`,
type: 'actions_storage',
name: 'artifact1-name',
url: `${configVariables.getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=artifact1-name`
},
{
containerId: '13',
size: -1,
signedContent: 'false',
fileContainerResourceUrl: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13`,
type: 'actions_storage',
name: 'artifact2-name',
url: `${configVariables.getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=artifact2-name`
}
]
}
const returnData: string = JSON.stringify(response, null, 2)
mockReadBody = async function(): Promise<string> {
return new Promise(resolve => {
resolve(returnData)
})
}
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: mockReadBody
})
})
})
}
/**
* Setups up HTTP GET response when querying for container items
*/
function setupSuccessfulContainerItemsResponse(): void {
jest.spyOn(HttpClient.prototype, 'get').mockImplementationOnce(async () => {
const mockMessage = new http.IncomingMessage(new net.Socket())
let mockReadBody = emptyMockReadBody
mockMessage.statusCode = 201
const response: QueryArtifactResponse = {
count: 2,
value: [
{
containerId: 10000,
scopeIdentifier: '00000000-0000-0000-0000-000000000000',
path: 'artifact-name',
itemType: 'folder',
status: 'created',
dateCreated: '2020-02-06T22:13:35.373Z',
dateLastModified: '2020-02-06T22:13:35.453Z',
createdBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
lastModifiedBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
itemLocation: `${configVariables.getRuntimeUrl()}/_apis/resources/Containers/10000?itemPath=artifact-name&metadata=True`,
contentLocation: `${configVariables.getRuntimeUrl()}/_apis/resources/Containers/10000?itemPath=artifact-name`,
contentId: ''
},
{
containerId: 10000,
scopeIdentifier: '00000000-0000-0000-0000-000000000000',
path: 'artifact-name/file1.txt',
itemType: 'file',
status: 'created',
dateCreated: '2020-02-06T22:13:35.373Z',
dateLastModified: '2020-02-06T22:13:35.453Z',
createdBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
lastModifiedBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
itemLocation: `${configVariables.getRuntimeUrl()}/_apis/resources/Containers/10000?itemPath=artifact-name%2Ffile1.txt&metadata=True`,
contentLocation: `${configVariables.getRuntimeUrl()}/_apis/resources/Containers/10000?itemPath=artifact-name%2Ffile1.txt`,
contentId: ''
}
]
}
const returnData: string = JSON.stringify(response, null, 2)
mockReadBody = async function(): Promise<string> {
return new Promise(resolve => {
resolve(returnData)
})
}
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: mockReadBody
})
})
})
}
/**
* Setups up HTTP GET response for a generic failed request
*/
function setupFailedResponse(): void {
jest.spyOn(HttpClient.prototype, 'get').mockImplementationOnce(async () => {
const mockMessage = new http.IncomingMessage(new net.Socket())
mockMessage.statusCode = 500
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: emptyMockReadBody
})
})
})
}
})

View File

@ -1,3 +1,6 @@
import * as fs from 'fs'
import * as io from '../../io/src/io'
import * as path from 'path'
import * as utils from '../src/internal-utils' import * as utils from '../src/internal-utils'
import * as core from '@actions/core' import * as core from '@actions/core'
import {HttpCodes} from '@actions/http-client' import {HttpCodes} from '@actions/http-client'
@ -96,4 +99,22 @@ describe('Utils', () => {
expect(utils.isRetryableStatusCode(HttpCodes.NotFound)).toEqual(false) expect(utils.isRetryableStatusCode(HttpCodes.NotFound)).toEqual(false)
expect(utils.isRetryableStatusCode(HttpCodes.Forbidden)).toEqual(false) expect(utils.isRetryableStatusCode(HttpCodes.Forbidden)).toEqual(false)
}) })
it('Test Creating Artifact Directories', async () => {
const root = path.join(__dirname, '_temp', 'artifact-download')
// remove directory before starting
await io.rmRF(root)
const directory1 = path.join(root, 'folder2', 'folder3')
const directory2 = path.join(directory1, 'folder1')
// Initially should not exist
expect(fs.existsSync(directory1)).toEqual(false)
expect(fs.existsSync(directory2)).toEqual(false)
const directoryStructure = [directory1, directory2]
await utils.createDirectoriesForArtifact(directoryStructure)
// directories should now be created
expect(fs.existsSync(directory1)).toEqual(true)
expect(fs.existsSync(directory2)).toEqual(true)
})
}) })

View File

@ -0,0 +1,45 @@
# Additional Information
Extra information
- [Non-Supported Characters](#Non-Supported-Characters)
- [Permission loss](#Permission-Loss)
- [Considerations](#Considerations)
## Non-Supported Characters
When uploading an artifact, the inputted `name` parameter along with the files specified in `files` cannot contain any of the following characters. They will be rejected by the server if attempted to be sent over and the upload will fail. These characters are not allowed due to limitations and restrictions with certain file systems such as NTFS. To maintain platform-agnostic behavior, all characters that are not supported by an individual filesystem/platform will not be supported on all filesystems/platforms.
- "
- :
- <
- \>
- |
- \*
- ?
- empty space
In addition to the aforementioned characters, the inputted `name` also cannot include the following
- \
- /
## Permission Loss
File permissions are not maintained between uploaded and downloaded artifacts. If file permissions are something that need to be maintained (such as an executable), consider archiving all of the files using something like `tar` and then uploading the single archive. After downloading the artifact, you can `un-tar` the individual file and permissions will be preserved.
```js
const artifact = require('@actions/artifact');
const artifactClient = artifact.create()
const artifactName = 'my-artifact';
const files = [
'/home/user/files/plz-upload/my-archive.tgz',
]
const rootDirectory = '/home/user/files/plz-upload'
const uploadResult = await artifactClient.uploadArtifact(artifactName, files, rootDirectory)
```
## Considerations
During upload, each file is uploaded concurrently in 4MB chunks using a separate HTTPS connection per file. Chunked uploads are used so that in the event of a failure (which is entirely possible because the internet is not perfect), the upload can be retried. If there is an error, a retry will be attempted after a certain period of time.
Uploading will be generally be faster if there are fewer files that are larger in size vs if there are lots of smaller files. Depending on the types and quantities of files being uploaded, it might be beneficial to separately compress and archive everything into a single archive (using something like `tar` or `zip`) before starting and artifact upload to speed things up.

View File

@ -10,8 +10,8 @@
], ],
"homepage": "https://github.com/actions/toolkit/tree/master/packages/artifact", "homepage": "https://github.com/actions/toolkit/tree/master/packages/artifact",
"license": "MIT", "license": "MIT",
"main": "lib/artifact.js", "main": "lib/artifact-client.js",
"types": "lib/artifact.d.ts", "types": "lib/artifact-client.d.ts",
"directories": { "directories": {
"lib": "lib", "lib": "lib",
"test": "__tests__" "test": "__tests__"

View File

@ -10,9 +10,22 @@ import {
} from './internal-upload-http-client' } from './internal-upload-http-client'
import {UploadResponse} from './internal-upload-response' import {UploadResponse} from './internal-upload-response'
import {UploadOptions} from './internal-upload-options' import {UploadOptions} from './internal-upload-options'
import {checkArtifactName} from './internal-utils' import {DownloadOptions} from './internal-download-options'
import {DownloadResponse} from './internal-download-response'
import {checkArtifactName, createDirectoriesForArtifact} from './internal-utils'
import {
listArtifacts,
downloadSingleArtifact,
getContainerItems
} from './internal-download-http-client'
import {getDownloadSpecification} from './internal-download-specification'
import {
getWorkSpaceDirectory,
getDownloadArtifactConcurrency
} from './internal-config-variables'
import {normalize, resolve} from 'path'
export {UploadResponse, UploadOptions} export {UploadResponse, UploadOptions, DownloadResponse, DownloadOptions}
export interface ArtifactClient { export interface ArtifactClient {
/** /**
@ -30,6 +43,25 @@ export interface ArtifactClient {
rootDirectory: string, rootDirectory: string,
options?: UploadOptions options?: UploadOptions
): Promise<UploadResponse> ): Promise<UploadResponse>
/**
* Downloads a single artifact associated with a run
*
* @param name the name of the artifact being downloaded
* @param path optional path that denotes where the artifact will be downloaded to
* @param options extra options that allow for the customization of the download behavior
*/
downloadArtifact(
name: string,
path?: string,
options?: DownloadOptions
): Promise<DownloadResponse>
/**
* Downloads all artifacts associated with a run. Because there are multiple artifacts being downloaded, a folder will be created for each one in the specified or default directory
* @param path optional path that denotes where the artifacts will be downloaded to
*/
downloadAllArtifacts(path?: string): Promise<DownloadResponse[]>
} }
export class DefaultArtifactClient implements ArtifactClient { export class DefaultArtifactClient implements ArtifactClient {
@ -100,25 +132,118 @@ export class DefaultArtifactClient implements ArtifactClient {
return uploadResponse return uploadResponse
} }
/* async downloadArtifact(
Downloads a single artifact associated with a run
export async function downloadArtifact(
name: string, name: string,
path?: string, path?: string | undefined,
options?: DownloadOptions options?: DownloadOptions | undefined
): Promise<DownloadResponse> { ): Promise<DownloadResponse> {
const artifacts = await listArtifacts()
TODO if (artifacts.count === 0) {
throw new Error(
`Unable to find any artifacts for the associated workflow`
)
} }
Downloads all artifacts associated with a run. Because there are multiple artifacts being downloaded, a folder will be created for each one in the specified or default directory const artifactToDownload = artifacts.value.find(artifact => {
return artifact.name === name
})
if (!artifactToDownload) {
throw new Error(`Unable to find an artifact with the name: ${name}`)
}
export async function downloadAllArtifacts( const items = await getContainerItems(
path?: string artifactToDownload.name,
artifactToDownload.fileContainerResourceUrl
)
if (!path) {
path = getWorkSpaceDirectory()
}
path = normalize(path)
path = resolve(path)
// During upload, empty directories are rejected by the remote server so there should be no artifacts that consist of only empty directories
const downloadSpecification = getDownloadSpecification(
name,
items.value,
path,
options?.createArtifactFolder || false
)
if (downloadSpecification.filesToDownload.length === 0) {
core.info(
`No downloadable files were found for the artifact: ${artifactToDownload.name}`
)
} else {
// Create all necessary directories recursively before starting any download
await createDirectoriesForArtifact(
downloadSpecification.directoryStructure
)
await downloadSingleArtifact(downloadSpecification.filesToDownload)
}
return {
artifactName: name,
downloadPath: downloadSpecification.rootDownloadLocation
}
}
async downloadAllArtifacts(
path?: string | undefined
): Promise<DownloadResponse[]> { ): Promise<DownloadResponse[]> {
const response: DownloadResponse[] = []
const artifacts = await listArtifacts()
if (artifacts.count === 0) {
core.info('Unable to find any artifacts for the associated workflow')
return response
}
TODO if (!path) {
path = getWorkSpaceDirectory()
}
path = normalize(path)
path = resolve(path)
const ARTIFACT_CONCURRENCY = getDownloadArtifactConcurrency()
const parallelDownloads = [...new Array(ARTIFACT_CONCURRENCY).keys()]
let downloadedArtifacts = 0
await Promise.all(
parallelDownloads.map(async () => {
while (downloadedArtifacts < artifacts.count) {
const currentArtifactToDownload = artifacts.value[downloadedArtifacts]
downloadedArtifacts += 1
// Get container entries for the specific artifact
const items = await getContainerItems(
currentArtifactToDownload.name,
currentArtifactToDownload.fileContainerResourceUrl
)
// Promise.All is not correctly inferring that 'path' is no longer possibly undefined: https://github.com/microsoft/TypeScript/issues/34925
const downloadSpecification = getDownloadSpecification(
currentArtifactToDownload.name,
items.value,
path!, // eslint-disable-line @typescript-eslint/no-non-null-assertion
true
)
if (downloadSpecification.filesToDownload.length === 0) {
core.info(
`No downloadable files were found for any artifact ${currentArtifactToDownload.name}`
)
} else {
await createDirectoriesForArtifact(
downloadSpecification.directoryStructure
)
await downloadSingleArtifact(downloadSpecification.filesToDownload)
}
response.push({
artifactName: currentArtifactToDownload.name,
downloadPath: downloadSpecification.rootDownloadLocation
})
}
})
)
return response
} }
*/
} }

View File

@ -10,6 +10,15 @@ export function getUploadChunkSize(): number {
return 4 * 1024 * 1024 // 4 MB Chunks return 4 * 1024 * 1024 // 4 MB Chunks
} }
export function getDownloadFileConcurrency(): number {
return 2
}
export function getDownloadArtifactConcurrency(): number {
// when downloading all artifact at once, this is number of concurrent artifacts being downloaded
return 1
}
export function getRuntimeToken(): string { export function getRuntimeToken(): string {
const token = process.env['ACTIONS_RUNTIME_TOKEN'] const token = process.env['ACTIONS_RUNTIME_TOKEN']
if (!token) { if (!token) {
@ -33,3 +42,11 @@ export function getWorkFlowRunId(): string {
} }
return workFlowRunId return workFlowRunId
} }
export function getWorkSpaceDirectory(): string {
const workspaceDirectory = process.env['GITHUB_WORKSPACE']
if (!workspaceDirectory) {
throw new Error('Unable to get GITHUB_WORKSPACE env variable')
}
return workspaceDirectory
}

View File

@ -31,3 +31,32 @@ export interface UploadResults {
size: number size: number
failedItems: string[] failedItems: string[]
} }
export interface ListArtifactsResponse {
count: number
value: ArtifactResponse[]
}
export interface QueryArtifactResponse {
count: number
value: ContainerEntry[]
}
export interface ContainerEntry {
containerId: number
scopeIdentifier: string
path: string
itemType: string
status: string
fileLength?: number
fileEncoding?: number
fileType?: number
dateCreated: string
dateLastModified: string
createdBy: string
lastModifiedBy: string
itemLocation: string
contentLocation: string
fileId?: number
contentId: string
}

View File

@ -0,0 +1,133 @@
import * as fs from 'fs'
import {
createHttpClient,
getArtifactUrl,
getRequestOptions,
isSuccessStatusCode,
isRetryableStatusCode
} from './internal-utils'
import {URL} from 'url'
import {
ListArtifactsResponse,
QueryArtifactResponse
} from './internal-contracts'
import {IHttpClientResponse} from '@actions/http-client/interfaces'
import {HttpClient} from '@actions/http-client'
import {DownloadItem} from './internal-download-specification'
import {getDownloadFileConcurrency} from './internal-config-variables'
import {warning} from '@actions/core'
/**
* Gets a list of all artifacts that are in a specific container
*/
export async function listArtifacts(): Promise<ListArtifactsResponse> {
const artifactUrl = getArtifactUrl()
const client = createHttpClient()
const requestOptions = getRequestOptions('application/json')
const rawResponse = await client.get(artifactUrl, requestOptions)
const body: string = await rawResponse.readBody()
if (isSuccessStatusCode(rawResponse.message.statusCode) && body) {
return JSON.parse(body)
}
// eslint-disable-next-line no-console
console.log(rawResponse)
throw new Error(`Unable to list artifacts for the run`)
}
/**
* Fetches a set of container items that describe the contents of an artifact
* @param artifactName the name of the artifact
* @param containerUrl the artifact container URL for the run
*/
export async function getContainerItems(
artifactName: string,
containerUrl: string
): Promise<QueryArtifactResponse> {
// The itemPath search parameter controls which containers will be returned
const resourceUrl = new URL(containerUrl)
resourceUrl.searchParams.append('itemPath', artifactName)
const client = createHttpClient()
const rawResponse = await client.get(resourceUrl.toString())
const body: string = await rawResponse.readBody()
if (isSuccessStatusCode(rawResponse.message.statusCode) && body) {
return JSON.parse(body)
}
// eslint-disable-next-line no-console
console.log(rawResponse)
throw new Error(`Unable to get ContainersItems from ${resourceUrl}`)
}
/**
* Concurrently downloads all the files that are part of an artifact
* @param downloadItems information about what items to download and where to save them
*/
export async function downloadSingleArtifact(
downloadItems: DownloadItem[]
): Promise<void> {
const DOWNLOAD_CONCURRENCY = getDownloadFileConcurrency()
// Limit the number of files downloaded at a single time
const parallelDownloads = [...new Array(DOWNLOAD_CONCURRENCY).keys()]
const client = createHttpClient()
let downloadedFiles = 0
await Promise.all(
parallelDownloads.map(async () => {
while (downloadedFiles < downloadItems.length) {
const currentFileToDownload = downloadItems[downloadedFiles]
downloadedFiles += 1
await downloadIndividualFile(
client,
currentFileToDownload.sourceLocation,
currentFileToDownload.targetPath
)
}
})
)
}
/**
* Downloads an individual file
* @param client http client that will be used to make the necessary calls
* @param artifactLocation origin location where a file will be downloaded from
* @param downloadPath destination location for the file being downloaded
*/
export async function downloadIndividualFile(
client: HttpClient,
artifactLocation: string,
downloadPath: string
): Promise<void> {
const stream = fs.createWriteStream(downloadPath)
const response = await client.get(artifactLocation)
if (isSuccessStatusCode(response.message.statusCode)) {
await pipeResponseToStream(response, stream)
} else if (isRetryableStatusCode(response.message.statusCode)) {
warning(
`Received http ${response.message.statusCode} during file download, will retry ${artifactLocation} after 10 seconds`
)
await new Promise(resolve => setTimeout(resolve, 10000))
const retryResponse = await client.get(artifactLocation)
if (isSuccessStatusCode(retryResponse.message.statusCode)) {
await pipeResponseToStream(response, stream)
} else {
// eslint-disable-next-line no-console
console.log(retryResponse)
throw new Error(`Unable to download ${artifactLocation}`)
}
} else {
// eslint-disable-next-line no-console
console.log(response)
throw new Error(`Unable to download ${artifactLocation}`)
}
}
export async function pipeResponseToStream(
response: IHttpClientResponse,
stream: NodeJS.WritableStream
): Promise<void> {
return new Promise(resolve => {
response.message.pipe(stream).on('close', () => {
resolve()
})
})
}

View File

@ -0,0 +1,78 @@
import * as path from 'path'
import {ContainerEntry} from './internal-contracts'
export interface DownloadSpecification {
// root download location for the artifact
rootDownloadLocation: string
// directories that need to be created for all the items in the artifact
directoryStructure: string[]
// individual files that need to be downloaded as part of the artifact
filesToDownload: DownloadItem[]
}
export interface DownloadItem {
// Url that denotes where to download the item from
sourceLocation: string
// Information about where the file should be downloaded to
targetPath: string
}
/**
* Creates a specification for a set of files that will be downloaded
* @param artifactName the name of the artifact
* @param artifactEntries a set of container entries that describe that files that make up an artifact
* @param downloadPath the path where the artifact will be downloaded to
* @param includeRootDirectory specifies if there should be an extra directory (denoted by the artifact name) where the artifact files should be downloaded to
*/
export function getDownloadSpecification(
artifactName: string,
artifactEntries: ContainerEntry[],
downloadPath: string,
includeRootDirectory: boolean
): DownloadSpecification {
const directories = new Set<string>()
const specifications: DownloadSpecification = {
rootDownloadLocation: includeRootDirectory
? path.join(downloadPath, artifactName)
: downloadPath,
directoryStructure: [],
filesToDownload: []
}
for (const entry of artifactEntries) {
// Ignore artifacts in the container that don't begin with the same name
if (
entry.path.startsWith(`${artifactName}/`) ||
entry.path.startsWith(`${artifactName}\\`)
) {
// normalize all separators to the local OS
const normalizedPathEntry = path.normalize(entry.path)
// entry.path always starts with the artifact name, if includeRootDirectory is false, remove the name from the beginning of the path
const filePath = path.join(
downloadPath,
includeRootDirectory
? normalizedPathEntry
: normalizedPathEntry.replace(artifactName, '')
)
// Case insensitive folder structure maintained in the backend, not every folder is created so the 'folder'
// itemType cannot be relied upon. The file must be used to determine the directory structure
if (entry.itemType === 'file') {
// Get the directories that we need to create from the filePath for each individual file
directories.add(path.dirname(filePath))
specifications.filesToDownload.push({
sourceLocation: entry.contentLocation,
targetPath: filePath
})
}
}
}
specifications.directoryStructure = Array.from(directories)
return specifications
}

View File

@ -1,4 +1,5 @@
import {debug} from '@actions/core' import {debug} from '@actions/core'
import {promises as fs} from 'fs'
import {HttpCodes, HttpClient} from '@actions/http-client' import {HttpCodes, HttpClient} from '@actions/http-client'
import {BearerCredentialHandler} from '@actions/http-client/auth' import {BearerCredentialHandler} from '@actions/http-client/auth'
import {IHeaders} from '@actions/http-client/interfaces' import {IHeaders} from '@actions/http-client/interfaces'
@ -113,3 +114,13 @@ export function checkArtifactName(name: string): void {
} }
} }
} }
export async function createDirectoriesForArtifact(
directories: string[]
): Promise<void> {
for (const directory of directories) {
await fs.mkdir(directory, {
recursive: true
})
}
}