From b6bad4eef62f663d32bc6c78ea9ff214b12bdd9c Mon Sep 17 00:00:00 2001 From: Jordi Boggiano Date: Thu, 9 Apr 2020 13:39:06 +0200 Subject: [PATCH 1/2] Add options to configure repository priorities --- CHANGELOG.md | 1 + UPGRADE-2.0.md | 2 +- doc/05-repositories.md | 19 +- doc/articles/repository-priorities.md | 94 +++++++++ src/Composer/DependencyResolver/Problem.php | 2 +- src/Composer/Repository/FilterRepository.php | 193 ++++++++++++++++++ src/Composer/Repository/RepositoryManager.php | 13 +- .../installer/repositories-priorities.test | 2 +- .../installer/repositories-priorities3.test | 49 +++++ .../Test/Repository/FilterRepositoryTest.php | 69 +++++++ .../Test/Repository/RepositoryManagerTest.php | 16 ++ 11 files changed, 455 insertions(+), 5 deletions(-) create mode 100644 doc/articles/repository-priorities.md create mode 100644 src/Composer/Repository/FilterRepository.php create mode 100644 tests/Composer/Test/Fixtures/installer/repositories-priorities3.test create mode 100644 tests/Composer/Test/Repository/FilterRepositoryTest.php diff --git a/CHANGELOG.md b/CHANGELOG.md index 71d55b32d..2fd7e2442 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ * Added support for parallel downloads of package metadata and zip files, this requires that the curl extension is present * Added much clearer dependency resolution error reporting for common error cases * Added support for TTY mode on Linux/OSX/WSL so that script handlers now run in interactive mode + * Added `only`, `exclude` and `canonical` options to all repositories, see [repository priorities](https://getcomposer.org/repoprio) for details * Added support for lib-zip platform package * Added `pre-operations-exec` event to be fired before the packages get installed/upgraded/removed * Added `pre-pool-create` event to be fired before the package pool for the dependency solver is created, which lets you modify the list of packages going in diff --git a/UPGRADE-2.0.md b/UPGRADE-2.0.md index f1fe337b7..618d311fc 100644 --- a/UPGRADE-2.0.md +++ b/UPGRADE-2.0.md @@ -2,7 +2,7 @@ ## For composer CLI users -- If a packages exists in a higher priority repository, it will now be entirely ignored in lower priority repositories. +- If a packages exists in a higher priority repository, it will now be entirely ignored in lower priority repositories. See [repository priorities](https://getcomposer.org/repoprio) for details. - Invalid PSR-0 / PSR-4 class configurations will not autoload anymore in optimized-autoloader mode, as per the warnings introduced in 1.10 - Package names now must comply to our naming guidelines or Composer will abort, as per the warnings introduced in 1.8.1 - Removed --no-suggest flag as it is not needed anymore diff --git a/doc/05-repositories.md b/doc/05-repositories.md index c551ff9e6..f6b1766e4 100644 --- a/doc/05-repositories.md +++ b/doc/05-repositories.md @@ -41,7 +41,7 @@ be preferred. A repository is a package source. It's a list of packages/versions. Composer will look in all your repositories to find the packages your project requires. -By default only the Packagist repository is registered in Composer. You can +By default only the Packagist.org repository is registered in Composer. You can add more repositories to your project by declaring them in `composer.json`. Repositories are only available to the root package and the repositories @@ -49,6 +49,12 @@ defined in your dependencies will not be loaded. Read the [FAQ entry](faqs/why-can't-composer-load-repositories-recursively.md) if you want to learn why. +When resolving dependencies, packages are looked up from repositories from +top to bottom, and by default as soon as a package is found in one Composer +stops looking in other repositories. Read the +[repository priorities](articles/repository-priorities.md) article for more +details and to see how to change this behavior. + ## Types ### Composer @@ -62,6 +68,17 @@ In the case of packagist, that file is located at `/packages.json`, so the URL o the repository would be `repo.packagist.org`. For `example.org/packages.json` the repository URL would be `example.org`. +```json +{ + "repositories": [ + { + "type": "composer", + "url": "https://example.org" + } + ] +} +``` + #### packages The only required field is `packages`. The JSON structure is as follows: diff --git a/doc/articles/repository-priorities.md b/doc/articles/repository-priorities.md new file mode 100644 index 000000000..031ca2b55 --- /dev/null +++ b/doc/articles/repository-priorities.md @@ -0,0 +1,94 @@ + + +# Repository priorities + +## Canonical repositories + +When Composer resolves dependencies it will look up a given package in the +topmost repository. If that repository does not contain the package, it +goes on to the next one, until one repository contains it and the process ends. + +Canonical repositories are better for a few reasons: + +- Performance wise, it is more efficient to stop looking for a package once it +has been found somewhere. It also avoids loading duplicate packages in case +the same package is present in several of your repositories. +- Security wise, it is safer to treat them canonically as it means that your most +important repositories will return the packages you expect them to always. Let's +say you have a private repository which is not canonical, and you require your +private package `foo/bar ^2.0` for example. Now if someone publishes +`foo/bar 2.999` to packagist.org, suddenly Composer will pick that package as it +has a higher version than your latest release (say 2.4.3), and you end up install +something you may not have meant to. If the private repository is canonical +however, that 2.999 version from packagist.org will not be considered at all. + +There are however a few cases where you may want to specifically load some packages +from a given repository, but not all. Or you may want a given repository to not be +canonical, and to be only preferred if it has higher package versions than the +repositories defined below. + +## Default behavior + +By default in Composer 2.x all repositories are canonical. Composer 1.x treated +all repositories as non-canonical. + +Another default is that the packagist.org repository is always added implicitly +as the last repository, unless you [disable it](../05-repositories.md#disabling-packagist-org). + +## Making repositories non-canonical + +You can add the canonical option to any repository to disable this default behavior +and make sure Composer keeps looking in other repositories, even if that repository +contains a given package. + +```json +{ + "repositories": [ + { + "type": "composer", + "url": "https://example.org", + "canonical": false + } + ] +} +``` + +## Filtering packages + +You can also filter packages which a repository will be able to load, either by +selecting which you want, or by excluding those you do not want. + +For example here we want to pick only the `foo/bar` and all the packages from +`some-vendor/` from this composer repository. + +```json +{ + "repositories": [ + { + "type": "composer", + "url": "https://example.org", + "only": ["foo/bar", "some-vendor/*"] + } + ] +} +``` + +And in this other example we exclude `toy/package` from a path repository, which +we may not want to load in this project. + +```json +{ + "repositories": [ + { + "type": "composer", + "url": "https://example.org", + "exclude": ["toy/package"] + } + ] +} +``` + +Both `only` and `exclude` should be array of package names, which can also +contain wildcards (`*`) which will match any characters. diff --git a/src/Composer/DependencyResolver/Problem.php b/src/Composer/DependencyResolver/Problem.php index f0d3d379c..6861b814c 100644 --- a/src/Composer/DependencyResolver/Problem.php +++ b/src/Composer/DependencyResolver/Problem.php @@ -257,7 +257,7 @@ class Problem } } - return array("- Root composer.json requires $packageName".self::constraintToText($constraint) . ', it is ', 'satisfiable by '.self::getPackageList($nextRepoPackages).' from '.$nextRepo->getRepoName().' but '.self::getPackageList($higherRepoPackages).' from '.reset($higherRepoPackages)->getRepository()->getRepoName().' has higher repository priority. The packages with higher priority do not match your constraint and are therefore not installable.'); + return array("- Root composer.json requires $packageName".self::constraintToText($constraint) . ', it is ', 'satisfiable by '.self::getPackageList($nextRepoPackages).' from '.$nextRepo->getRepoName().' but '.self::getPackageList($higherRepoPackages).' from '.reset($higherRepoPackages)->getRepository()->getRepoName().' has higher repository priority. The packages with higher priority do not match your constraint and are therefore not installable. See https://getcomposer.org/repoprio for details and assistance.'); } return array("- Root composer.json requires $packageName".self::constraintToText($constraint) . ', ', 'found '.self::getPackageList($packages).' but '.(self::hasMultipleNames($packages) ? 'these do' : 'it does').' not match your constraint.'); diff --git a/src/Composer/Repository/FilterRepository.php b/src/Composer/Repository/FilterRepository.php new file mode 100644 index 000000000..92c8283a8 --- /dev/null +++ b/src/Composer/Repository/FilterRepository.php @@ -0,0 +1,193 @@ + + * Jordi Boggiano + * + * For the full copyright and license information, please view the LICENSE + * file that was distributed with this source code. + */ + +namespace Composer\Repository; + +use Composer\Package\PackageInterface; +use Composer\Package\BasePackage; + +/** + * Filters which packages are seen as canonical on this repo by loadPackages + * + * @author Jordi Boggiano + */ +class FilterRepository implements RepositoryInterface +{ + private $only = array(); + private $exclude = array(); + private $canonical = true; + private $repo; + + public function __construct(RepositoryInterface $repo, array $options) + { + if (isset($options['only'])) { + if (!is_array($options['only'])) { + throw new \InvalidArgumentException('"only" key for repository '.$repo->getRepoName().' should be an array'); + } + $this->only = '{^'.implode('|', array_map(function ($val) { + return BasePackage::packageNameToRegexp($val, '%s'); + }, $options['only'])) .'$}iD'; + } + if (isset($options['exclude'])) { + if (!is_array($options['exclude'])) { + throw new \InvalidArgumentException('"exclude" key for repository '.$repo->getRepoName().' should be an array'); + } + $this->exclude = '{^'.implode('|', array_map(function ($val) { + return BasePackage::packageNameToRegexp($val, '%s'); + }, $options['exclude'])) .'$}iD'; + } + if ($this->exclude && $this->only) { + throw new \InvalidArgumentException('Only one of "only" and "exclude" can be specified for repository '.$repo->getRepoName()); + } + if (isset($options['canonical'])) { + if (!is_bool($options['canonical'])) { + throw new \InvalidArgumentException('"canonical" key for repository '.$repo->getRepoName().' should be a boolean'); + } + $this->canonical = $options['canonical']; + } + + $this->repo = $repo; + } + + public function getRepoName() + { + return $this->repo->getRepoName(); + } + + /** + * Returns the wrapped repositories + * + * @return RepositoryInterface + */ + public function getRepository() + { + return $this->repo; + } + + /** + * {@inheritdoc} + */ + public function hasPackage(PackageInterface $package) + { + return $this->repo->hasPackage($package); + } + + /** + * {@inheritdoc} + */ + public function findPackage($name, $constraint) + { + if (!$this->isAllowed($name)) { + return null; + } + + return $this->repo->findPackage($name, $constraint); + } + + /** + * {@inheritdoc} + */ + public function findPackages($name, $constraint = null) + { + if (!$this->isAllowed($name)) { + return array(); + } + + return $this->repo->findPackages($name, $constraint); + } + + /** + * {@inheritDoc} + */ + public function loadPackages(array $packageMap, array $acceptableStabilities, array $stabilityFlags) + { + foreach ($packageMap as $name => $constraint) { + if (!$this->isAllowed($name)) { + unset($packageMap[$name]); + } + } + + $result = $this->repo->loadPackages($packageMap, $acceptableStabilities, $stabilityFlags); + if (!$this->canonical) { + $result['namesFound'] = array(); + } + + return $result; + } + + /** + * {@inheritdoc} + */ + public function search($query, $mode = 0, $type = null) + { + return $this->repo->search($query, $mode, $type); + } + + /** + * {@inheritdoc} + */ + public function getPackages() + { + $result = array(); + foreach ($this->repo->getPackages() as $package) { + if ($this->isAllowed($package->getName())) { + $result[] = $package; + } + } + + return $result; + } + + /** + * {@inheritdoc} + */ + public function getProviders($packageName) + { + $result = array(); + foreach ($this->repo->getProviders($packageName) as $provider) { + if ($this->isAllowed($provider['name'])) { + $result[] = $provider; + } + } + + return $result; + } + + /** + * {@inheritdoc} + */ + public function removePackage(PackageInterface $package) + { + return $this->repo->removePackage($package); + } + + /** + * {@inheritdoc} + */ + public function count() + { + return $this->repo->count(); + } + + private function isAllowed($name) + { + if (!$this->only && !$this->exclude) { + return true; + } + + if ($this->only) { + return (bool) preg_match($this->only, $name); + } + + return !preg_match($this->exclude, $name); + } +} diff --git a/src/Composer/Repository/RepositoryManager.php b/src/Composer/Repository/RepositoryManager.php index 2dca57099..c5da49cdb 100644 --- a/src/Composer/Repository/RepositoryManager.php +++ b/src/Composer/Repository/RepositoryManager.php @@ -125,7 +125,18 @@ class RepositoryManager $class = $this->repositoryClasses[$type]; - return new $class($config, $this->io, $this->config, $this->httpDownloader, $this->eventDispatcher); + if (isset($config['only']) || isset($config['exclude']) || isset($config['canonical'])) { + $filterConfig = $config; + unset($config['only'], $config['exclude'], $config['canonical']); + } + + $repository = new $class($config, $this->io, $this->config, $this->httpDownloader, $this->eventDispatcher); + + if (isset($filterConfig)) { + $repository = new FilterRepository($repository, $filterConfig); + } + + return $repository; } /** diff --git a/tests/Composer/Test/Fixtures/installer/repositories-priorities.test b/tests/Composer/Test/Fixtures/installer/repositories-priorities.test index bc06179e0..f06efc5eb 100644 --- a/tests/Composer/Test/Fixtures/installer/repositories-priorities.test +++ b/tests/Composer/Test/Fixtures/installer/repositories-priorities.test @@ -28,7 +28,7 @@ Updating dependencies Your requirements could not be resolved to an installable set of packages. Problem 1 - - Root composer.json requires foo/a 2.*, it is satisfiable by foo/a[2.0.0] from package repo (defining 1 package) but foo/a[1.0.0] from package repo (defining 1 package) has higher repository priority. The packages with higher priority do not match your constraint and are therefore not installable. + - Root composer.json requires foo/a 2.*, it is satisfiable by foo/a[2.0.0] from package repo (defining 1 package) but foo/a[1.0.0] from package repo (defining 1 package) has higher repository priority. The packages with higher priority do not match your constraint and are therefore not installable. See https://getcomposer.org/repoprio for details and assistance. --EXPECT-- --EXPECT-EXIT-CODE-- diff --git a/tests/Composer/Test/Fixtures/installer/repositories-priorities3.test b/tests/Composer/Test/Fixtures/installer/repositories-priorities3.test new file mode 100644 index 000000000..35d66617c --- /dev/null +++ b/tests/Composer/Test/Fixtures/installer/repositories-priorities3.test @@ -0,0 +1,49 @@ +--TEST-- +Test that filter repositories apply correctly +--COMPOSER-- +{ + "repositories": [ + { + "type": "package", + "package": [ + { "name": "foo/a", "version": "1.0.0" } + ], + "canonical": false + }, + { + "type": "package", + "package": [ + { "name": "foo/a", "version": "1.0.0" }, + { "name": "foo/b", "version": "1.0.0" } + ], + "only": ["foo/b"] + }, + { + "type": "package", + "package": [ + { "name": "foo/a", "version": "1.2.0" }, + { "name": "foo/c", "version": "1.2.0" } + ], + "exclude": ["foo/c"] + }, + { + "type": "package", + "package": [ + { "name": "foo/a", "version": "1.1.0" }, + { "name": "foo/b", "version": "1.1.0" }, + { "name": "foo/c", "version": "1.1.0" } + ] + } + ], + "require": { + "foo/a": "1.*", + "foo/b": "1.*", + "foo/c": "1.*" + } +} +--RUN-- +update +--EXPECT-- +Installing foo/a (1.2.0) +Installing foo/b (1.0.0) +Installing foo/c (1.1.0) diff --git a/tests/Composer/Test/Repository/FilterRepositoryTest.php b/tests/Composer/Test/Repository/FilterRepositoryTest.php new file mode 100644 index 000000000..2973e445d --- /dev/null +++ b/tests/Composer/Test/Repository/FilterRepositoryTest.php @@ -0,0 +1,69 @@ + + * Jordi Boggiano + * + * For the full copyright and license information, please view the LICENSE + * file that was distributed with this source code. + */ + +namespace Composer\Test\Repository; + +use Composer\Test\TestCase; +use Composer\Repository\FilterRepository; +use Composer\Repository\ArrayRepository; +use Composer\Semver\Constraint\EmptyConstraint; +use Composer\Package\BasePackage; + +class FilterRepositoryTest extends TestCase +{ + private $arrayRepo; + + public function setUp() + { + $this->arrayRepo = new ArrayRepository(); + $this->arrayRepo->addPackage($this->getPackage('foo/aaa', '1.0.0')); + $this->arrayRepo->addPackage($this->getPackage('foo/bbb', '1.0.0')); + $this->arrayRepo->addPackage($this->getPackage('bar/xxx', '1.0.0')); + $this->arrayRepo->addPackage($this->getPackage('baz/yyy', '1.0.0')); + } + + /** + * @dataProvider repoMatchingTests + */ + public function testRepoMatching($expected, $config) + { + $repo = new FilterRepository($this->arrayRepo, $config); + $packages = $repo->getPackages(); + + $this->assertSame($expected, array_map(function ($p) { return $p->getName(); }, $packages)); + } + + public static function repoMatchingTests() + { + return array( + array(array('foo/aaa', 'foo/bbb'), array('only' => array('foo/*'))), + array(array('foo/aaa', 'baz/yyy'), array('only' => array('foo/aaa', 'baz/yyy'))), + array(array('bar/xxx'), array('exclude' => array('foo/*', 'baz/yyy'))), + ); + } + + public function testCanonicalDefaultTrue() + { + $repo = new FilterRepository($this->arrayRepo, array()); + $result = $repo->loadPackages(array('foo/aaa' => new EmptyConstraint), BasePackage::$stabilities, array()); + $this->assertCount(1, $result['packages']); + $this->assertCount(1, $result['namesFound']); + } + + public function testNonCanonical() + { + $repo = new FilterRepository($this->arrayRepo, array('canonical' => false)); + $result = $repo->loadPackages(array('foo/aaa' => new EmptyConstraint), BasePackage::$stabilities, array()); + $this->assertCount(1, $result['packages']); + $this->assertCount(0, $result['namesFound']); + } +} diff --git a/tests/Composer/Test/Repository/RepositoryManagerTest.php b/tests/Composer/Test/Repository/RepositoryManagerTest.php index dad0bd346..614dd226c 100644 --- a/tests/Composer/Test/Repository/RepositoryManagerTest.php +++ b/tests/Composer/Test/Repository/RepositoryManagerTest.php @@ -108,4 +108,20 @@ class RepositoryManagerTest extends TestCase return $cases; } + + public function testFilterRepoWrapping() + { + $rm = new RepositoryManager( + $this->getMockBuilder('Composer\IO\IOInterface')->getMock(), + $config = $this->getMockBuilder('Composer\Config')->setMethods(array('get'))->getMock(), + $this->getMockBuilder('Composer\Util\HttpDownloader')->disableOriginalConstructor()->getMock(), + $this->getMockBuilder('Composer\EventDispatcher\EventDispatcher')->disableOriginalConstructor()->getMock() + ); + + $rm->setRepositoryClass('path', 'Composer\Repository\PathRepository'); + $repo = $rm->createRepository('path', array('type' => 'path', 'url' => __DIR__, 'only' => array('foo/bar'))); + + $this->assertInstanceOf('Composer\Repository\FilterRepository', $repo); + $this->assertInstanceOf('Composer\Repository\PathRepository', $repo->getRepository()); + } } From 059c00917977577de0f1b1c70f743e5cae84e16a Mon Sep 17 00:00:00 2001 From: Jordi Boggiano Date: Thu, 9 Apr 2020 14:01:05 +0200 Subject: [PATCH 2/2] Docs fixes Co-Authored-By: Nils Adermann --- doc/05-repositories.md | 2 +- doc/articles/repository-priorities.md | 15 ++++++++------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/doc/05-repositories.md b/doc/05-repositories.md index f6b1766e4..b90c6e347 100644 --- a/doc/05-repositories.md +++ b/doc/05-repositories.md @@ -50,7 +50,7 @@ defined in your dependencies will not be loaded. Read the want to learn why. When resolving dependencies, packages are looked up from repositories from -top to bottom, and by default as soon as a package is found in one Composer +top to bottom, and by default, as soon as a package is found in one, Composer stops looking in other repositories. Read the [repository priorities](articles/repository-priorities.md) article for more details and to see how to change this behavior. diff --git a/doc/articles/repository-priorities.md b/doc/articles/repository-priorities.md index 031ca2b55..d9ca59393 100644 --- a/doc/articles/repository-priorities.md +++ b/doc/articles/repository-priorities.md @@ -6,7 +6,7 @@ ## Canonical repositories -When Composer resolves dependencies it will look up a given package in the +When Composer resolves dependencies, it will look up a given package in the topmost repository. If that repository does not contain the package, it goes on to the next one, until one repository contains it and the process ends. @@ -15,12 +15,13 @@ Canonical repositories are better for a few reasons: - Performance wise, it is more efficient to stop looking for a package once it has been found somewhere. It also avoids loading duplicate packages in case the same package is present in several of your repositories. -- Security wise, it is safer to treat them canonically as it means that your most -important repositories will return the packages you expect them to always. Let's +- Security wise, it is safer to treat them canonically as it means that packages you +expect to come from your most important repositories will never be loaded from +another repository instad. Let's say you have a private repository which is not canonical, and you require your private package `foo/bar ^2.0` for example. Now if someone publishes `foo/bar 2.999` to packagist.org, suddenly Composer will pick that package as it -has a higher version than your latest release (say 2.4.3), and you end up install +has a higher version than your latest release (say 2.4.3), and you end up installing something you may not have meant to. If the private repository is canonical however, that 2.999 version from packagist.org will not be considered at all. @@ -58,9 +59,9 @@ contains a given package. ## Filtering packages You can also filter packages which a repository will be able to load, either by -selecting which you want, or by excluding those you do not want. +selecting which ones you want, or by excluding those you do not want. -For example here we want to pick only the `foo/bar` and all the packages from +For example here we want to pick only the package `foo/bar` and all the packages from `some-vendor/` from this composer repository. ```json @@ -90,5 +91,5 @@ we may not want to load in this project. } ``` -Both `only` and `exclude` should be array of package names, which can also +Both `only` and `exclude` should be arrays of package names, which can also contain wildcards (`*`) which will match any characters.