Merge pull request #8755 from Seldaek/repo-filtering

Add options to configure repository priorities

CHANGELOG.md

@@ -8,6 +8,7 @@
   * Added support for parallel downloads of package metadata and zip files, this requires that the curl extension is present
   * Added much clearer dependency resolution error reporting for common error cases
   * Added support for TTY mode on Linux/OSX/WSL so that script handlers now run in interactive mode
+  * Added `only`, `exclude` and `canonical` options to all repositories, see [repository priorities](https://getcomposer.org/repoprio) for details
   * Added support for lib-zip platform package
   * Added `pre-operations-exec` event to be fired before the packages get installed/upgraded/removed
   * Added `pre-pool-create` event to be fired before the package pool for the dependency solver is created, which lets you modify the list of packages going in

UPGRADE-2.0.md

@@ -2,7 +2,7 @@
 
 ## For composer CLI users
 
-- If a packages exists in a higher priority repository, it will now be entirely ignored in lower priority repositories.
+- If a packages exists in a higher priority repository, it will now be entirely ignored in lower priority repositories. See [repository priorities](https://getcomposer.org/repoprio) for details.
 - Invalid PSR-0 / PSR-4 class configurations will not autoload anymore in optimized-autoloader mode, as per the warnings introduced in 1.10
 - Package names now must comply to our naming guidelines or Composer will abort, as per the warnings introduced in 1.8.1
 - Removed --no-suggest flag as it is not needed anymore

doc/05-repositories.md

@@ -41,7 +41,7 @@ be preferred.
 A repository is a package source. It's a list of packages/versions. Composer
 will look in all your repositories to find the packages your project requires.
 
-By default only the Packagist repository is registered in Composer. You can
+By default only the Packagist.org repository is registered in Composer. You can
 add more repositories to your project by declaring them in `composer.json`.
 
 Repositories are only available to the root package and the repositories
@@ -49,6 +49,12 @@ defined in your dependencies will not be loaded. Read the
 [FAQ entry](faqs/why-can't-composer-load-repositories-recursively.md) if you
 want to learn why.
 
+When resolving dependencies, packages are looked up from repositories from
+top to bottom, and by default, as soon as a package is found in one, Composer
+stops looking in other repositories. Read the
+[repository priorities](articles/repository-priorities.md) article for more
+details and to see how to change this behavior.
+
 ## Types
 
 ### Composer
@@ -62,6 +68,17 @@ In the case of packagist, that file is located at `/packages.json`, so the URL o
 the repository would be `repo.packagist.org`. For `example.org/packages.json` the
 repository URL would be `example.org`.
 
+```json
+{
+    "repositories": [
+        {
+            "type": "composer",
+            "url": "https://example.org"
+        }
+    ]
+}
+```
+
 #### packages
 
 The only required field is `packages`. The JSON structure is as follows:

doc/articles/repository-priorities.md

@@ -0,0 +1,95 @@
+<!--
+    tagline: Configure which packages are found in which repositories
+-->
+
+# Repository priorities
+
+## Canonical repositories
+
+When Composer resolves dependencies, it will look up a given package in the
+topmost repository. If that repository does not contain the package, it
+goes on to the next one, until one repository contains it and the process ends.
+
+Canonical repositories are better for a few reasons:
+
+- Performance wise, it is more efficient to stop looking for a package once it
+has been found somewhere. It also avoids loading duplicate packages in case
+the same package is present in several of your repositories.
+- Security wise, it is safer to treat them canonically as it means that packages you
+expect to come from your most important repositories will never be loaded from 
+another repository instad. Let's
+say you have a private repository which is not canonical, and you require your
+private package `foo/bar ^2.0` for example. Now if someone publishes
+`foo/bar 2.999` to packagist.org, suddenly Composer will pick that package as it
+has a higher version than your latest release (say 2.4.3), and you end up installing
+something you may not have meant to. If the private repository is canonical
+however, that 2.999 version from packagist.org will not be considered at all.
+
+There are however a few cases where you may want to specifically load some packages
+from a given repository, but not all. Or you may want a given repository to not be
+canonical, and to be only preferred if it has higher package versions than the
+repositories defined below.
+
+## Default behavior
+
+By default in Composer 2.x all repositories are canonical. Composer 1.x treated
+all repositories as non-canonical.
+
+Another default is that the packagist.org repository is always added implicitly
+as the last repository, unless you [disable it](../05-repositories.md#disabling-packagist-org).
+
+## Making repositories non-canonical
+
+You can add the canonical option to any repository to disable this default behavior
+and make sure Composer keeps looking in other repositories, even if that repository
+contains a given package.
+
+```json
+{
+    "repositories": [
+        {
+            "type": "composer",
+            "url": "https://example.org",
+            "canonical": false
+        }
+    ]
+}
+```
+
+## Filtering packages
+
+You can also filter packages which a repository will be able to load, either by
+selecting which ones you want, or by excluding those you do not want.
+
+For example here we want to pick only the package `foo/bar` and all the packages from
+`some-vendor/` from this composer repository.
+
+```json
+{
+    "repositories": [
+        {
+            "type": "composer",
+            "url": "https://example.org",
+            "only": ["foo/bar", "some-vendor/*"]
+        }
+    ]
+}
+```
+
+And in this other example we exclude `toy/package` from a path repository, which
+we may not want to load in this project.
+
+```json
+{
+    "repositories": [
+        {
+            "type": "composer",
+            "url": "https://example.org",
+            "exclude": ["toy/package"]
+        }
+    ]
+}
+```
+
+Both `only` and `exclude` should be arrays of package names, which can also
+contain wildcards (`*`) which will match any characters.

src/Composer/DependencyResolver/Problem.php

@@ -257,7 +257,7 @@ class Problem
                     }
                 }
 
-                return array("- Root composer.json requires $packageName".self::constraintToText($constraint) . ', it is ', 'satisfiable by '.self::getPackageList($nextRepoPackages).' from '.$nextRepo->getRepoName().' but '.self::getPackageList($higherRepoPackages).' from '.reset($higherRepoPackages)->getRepository()->getRepoName().' has higher repository priority. The packages with higher priority do not match your constraint and are therefore not installable.');
+                return array("- Root composer.json requires $packageName".self::constraintToText($constraint) . ', it is ', 'satisfiable by '.self::getPackageList($nextRepoPackages).' from '.$nextRepo->getRepoName().' but '.self::getPackageList($higherRepoPackages).' from '.reset($higherRepoPackages)->getRepository()->getRepoName().' has higher repository priority. The packages with higher priority do not match your constraint and are therefore not installable. See https://getcomposer.org/repoprio for details and assistance.');
             }
 
             return array("- Root composer.json requires $packageName".self::constraintToText($constraint) . ', ', 'found '.self::getPackageList($packages).' but '.(self::hasMultipleNames($packages) ? 'these do' : 'it does').' not match your constraint.');

src/Composer/Repository/FilterRepository.php

@@ -0,0 +1,193 @@
+<?php
+
+/*
+ * This file is part of Composer.
+ *
+ * (c) Nils Adermann <naderman@naderman.de>
+ *     Jordi Boggiano <j.boggiano@seld.be>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+namespace Composer\Repository;
+
+use Composer\Package\PackageInterface;
+use Composer\Package\BasePackage;
+
+/**
+ * Filters which packages are seen as canonical on this repo by loadPackages
+ *
+ * @author Jordi Boggiano <j.boggiano@seld.be>
+ */
+class FilterRepository implements RepositoryInterface
+{
+    private $only = array();
+    private $exclude = array();
+    private $canonical = true;
+    private $repo;
+
+    public function __construct(RepositoryInterface $repo, array $options)
+    {
+        if (isset($options['only'])) {
+            if (!is_array($options['only'])) {
+                throw new \InvalidArgumentException('"only" key for repository '.$repo->getRepoName().' should be an array');
+            }
+            $this->only = '{^'.implode('|', array_map(function ($val) {
+                return BasePackage::packageNameToRegexp($val, '%s');
+            }, $options['only'])) .'$}iD';
+        }
+        if (isset($options['exclude'])) {
+            if (!is_array($options['exclude'])) {
+                throw new \InvalidArgumentException('"exclude" key for repository '.$repo->getRepoName().' should be an array');
+            }
+            $this->exclude = '{^'.implode('|', array_map(function ($val) {
+                return BasePackage::packageNameToRegexp($val, '%s');
+            }, $options['exclude'])) .'$}iD';
+        }
+        if ($this->exclude && $this->only) {
+            throw new \InvalidArgumentException('Only one of "only" and "exclude" can be specified for repository '.$repo->getRepoName());
+        }
+        if (isset($options['canonical'])) {
+            if (!is_bool($options['canonical'])) {
+                throw new \InvalidArgumentException('"canonical" key for repository '.$repo->getRepoName().' should be a boolean');
+            }
+            $this->canonical = $options['canonical'];
+        }
+
+        $this->repo = $repo;
+    }
+
+    public function getRepoName()
+    {
+        return $this->repo->getRepoName();
+    }
+
+    /**
+     * Returns the wrapped repositories
+     *
+     * @return RepositoryInterface
+     */
+    public function getRepository()
+    {
+        return $this->repo;
+    }
+
+    /**
+     * {@inheritdoc}
+     */
+    public function hasPackage(PackageInterface $package)
+    {
+        return $this->repo->hasPackage($package);
+    }
+
+    /**
+     * {@inheritdoc}
+     */
+    public function findPackage($name, $constraint)
+    {
+        if (!$this->isAllowed($name)) {
+            return null;
+        }
+
+        return $this->repo->findPackage($name, $constraint);
+    }
+
+    /**
+     * {@inheritdoc}
+     */
+    public function findPackages($name, $constraint = null)
+    {
+        if (!$this->isAllowed($name)) {
+            return array();
+        }
+
+        return $this->repo->findPackages($name, $constraint);
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    public function loadPackages(array $packageMap, array $acceptableStabilities, array $stabilityFlags)
+    {
+        foreach ($packageMap as $name => $constraint) {
+            if (!$this->isAllowed($name)) {
+                unset($packageMap[$name]);
+            }
+        }
+
+        $result = $this->repo->loadPackages($packageMap, $acceptableStabilities, $stabilityFlags);
+        if (!$this->canonical) {
+            $result['namesFound'] = array();
+        }
+
+        return $result;
+    }
+
+    /**
+     * {@inheritdoc}
+     */
+    public function search($query, $mode = 0, $type = null)
+    {
+        return $this->repo->search($query, $mode, $type);
+    }
+
+    /**
+     * {@inheritdoc}
+     */
+    public function getPackages()
+    {
+        $result = array();
+        foreach ($this->repo->getPackages() as $package) {
+            if ($this->isAllowed($package->getName())) {
+                $result[] = $package;
+            }
+        }
+
+        return $result;
+    }
+
+    /**
+     * {@inheritdoc}
+     */
+    public function getProviders($packageName)
+    {
+        $result = array();
+        foreach ($this->repo->getProviders($packageName) as $provider) {
+            if ($this->isAllowed($provider['name'])) {
+                $result[] = $provider;
+            }
+        }
+
+        return $result;
+    }
+
+    /**
+     * {@inheritdoc}
+     */
+    public function removePackage(PackageInterface $package)
+    {
+        return $this->repo->removePackage($package);
+    }
+
+    /**
+     * {@inheritdoc}
+     */
+    public function count()
+    {
+        return $this->repo->count();
+    }
+
+    private function isAllowed($name)
+    {
+        if (!$this->only && !$this->exclude) {
+            return true;
+        }
+
+        if ($this->only) {
+            return (bool) preg_match($this->only, $name);
+        }
+
+        return !preg_match($this->exclude, $name);
+    }
+}

src/Composer/Repository/RepositoryManager.php

@@ -125,7 +125,18 @@ class RepositoryManager
 
         $class = $this->repositoryClasses[$type];
 
-        return new $class($config, $this->io, $this->config, $this->httpDownloader, $this->eventDispatcher);
+        if (isset($config['only']) || isset($config['exclude']) || isset($config['canonical'])) {
+            $filterConfig = $config;
+            unset($config['only'], $config['exclude'], $config['canonical']);
+        }
+
+        $repository = new $class($config, $this->io, $this->config, $this->httpDownloader, $this->eventDispatcher);
+
+        if (isset($filterConfig)) {
+            $repository = new FilterRepository($repository, $filterConfig);
+        }
+
+        return $repository;
     }
 
     /**

tests/Composer/Test/Fixtures/installer/repositories-priorities.test

@@ -28,7 +28,7 @@ Updating dependencies
 Your requirements could not be resolved to an installable set of packages.
 
   Problem 1
-    - Root composer.json requires foo/a 2.*, it is satisfiable by foo/a[2.0.0] from package repo (defining 1 package) but foo/a[1.0.0] from package repo (defining 1 package) has higher repository priority. The packages with higher priority do not match your constraint and are therefore not installable.
+    - Root composer.json requires foo/a 2.*, it is satisfiable by foo/a[2.0.0] from package repo (defining 1 package) but foo/a[1.0.0] from package repo (defining 1 package) has higher repository priority. The packages with higher priority do not match your constraint and are therefore not installable. See https://getcomposer.org/repoprio for details and assistance.
 
 --EXPECT--
 --EXPECT-EXIT-CODE--

tests/Composer/Test/Fixtures/installer/repositories-priorities3.test

@@ -0,0 +1,49 @@
+--TEST--
+Test that filter repositories apply correctly
+--COMPOSER--
+{
+    "repositories": [
+        {
+            "type": "package",
+            "package": [
+                { "name": "foo/a", "version": "1.0.0" }
+            ],
+            "canonical": false
+        },
+        {
+            "type": "package",
+            "package": [
+                { "name": "foo/a", "version": "1.0.0" },
+                { "name": "foo/b", "version": "1.0.0" }
+            ],
+            "only": ["foo/b"]
+        },
+        {
+            "type": "package",
+            "package": [
+                { "name": "foo/a", "version": "1.2.0" },
+                { "name": "foo/c", "version": "1.2.0" }
+            ],
+            "exclude": ["foo/c"]
+        },
+        {
+            "type": "package",
+            "package": [
+                { "name": "foo/a", "version": "1.1.0" },
+                { "name": "foo/b", "version": "1.1.0" },
+                { "name": "foo/c", "version": "1.1.0" }
+            ]
+        }
+    ],
+    "require": {
+        "foo/a": "1.*",
+        "foo/b": "1.*",
+        "foo/c": "1.*"
+    }
+}
+--RUN--
+update
+--EXPECT--
+Installing foo/a (1.2.0)
+Installing foo/b (1.0.0)
+Installing foo/c (1.1.0)

tests/Composer/Test/Repository/FilterRepositoryTest.php

@@ -0,0 +1,69 @@
+<?php
+
+/*
+ * This file is part of Composer.
+ *
+ * (c) Nils Adermann <naderman@naderman.de>
+ *     Jordi Boggiano <j.boggiano@seld.be>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+namespace Composer\Test\Repository;
+
+use Composer\Test\TestCase;
+use Composer\Repository\FilterRepository;
+use Composer\Repository\ArrayRepository;
+use Composer\Semver\Constraint\EmptyConstraint;
+use Composer\Package\BasePackage;
+
+class FilterRepositoryTest extends TestCase
+{
+    private $arrayRepo;
+
+    public function setUp()
+    {
+        $this->arrayRepo = new ArrayRepository();
+        $this->arrayRepo->addPackage($this->getPackage('foo/aaa', '1.0.0'));
+        $this->arrayRepo->addPackage($this->getPackage('foo/bbb', '1.0.0'));
+        $this->arrayRepo->addPackage($this->getPackage('bar/xxx', '1.0.0'));
+        $this->arrayRepo->addPackage($this->getPackage('baz/yyy', '1.0.0'));
+    }
+
+    /**
+     * @dataProvider repoMatchingTests
+     */
+    public function testRepoMatching($expected, $config)
+    {
+        $repo = new FilterRepository($this->arrayRepo, $config);
+        $packages = $repo->getPackages();
+
+        $this->assertSame($expected, array_map(function ($p) { return $p->getName(); }, $packages));
+    }
+
+    public static function repoMatchingTests()
+    {
+        return array(
+            array(array('foo/aaa', 'foo/bbb'), array('only' => array('foo/*'))),
+            array(array('foo/aaa', 'baz/yyy'), array('only' => array('foo/aaa', 'baz/yyy'))),
+            array(array('bar/xxx'), array('exclude' => array('foo/*', 'baz/yyy'))),
+        );
+    }
+
+    public function testCanonicalDefaultTrue()
+    {
+        $repo = new FilterRepository($this->arrayRepo, array());
+        $result = $repo->loadPackages(array('foo/aaa' => new EmptyConstraint), BasePackage::$stabilities, array());
+        $this->assertCount(1, $result['packages']);
+        $this->assertCount(1, $result['namesFound']);
+    }
+
+    public function testNonCanonical()
+    {
+        $repo = new FilterRepository($this->arrayRepo, array('canonical' => false));
+        $result = $repo->loadPackages(array('foo/aaa' => new EmptyConstraint), BasePackage::$stabilities, array());
+        $this->assertCount(1, $result['packages']);
+        $this->assertCount(0, $result['namesFound']);
+    }
+}

tests/Composer/Test/Repository/RepositoryManagerTest.php

@@ -108,4 +108,20 @@ class RepositoryManagerTest extends TestCase
 
         return $cases;
     }
+
+    public function testFilterRepoWrapping()
+    {
+        $rm = new RepositoryManager(
+            $this->getMockBuilder('Composer\IO\IOInterface')->getMock(),
+            $config = $this->getMockBuilder('Composer\Config')->setMethods(array('get'))->getMock(),
+            $this->getMockBuilder('Composer\Util\HttpDownloader')->disableOriginalConstructor()->getMock(),
+            $this->getMockBuilder('Composer\EventDispatcher\EventDispatcher')->disableOriginalConstructor()->getMock()
+        );
+
+        $rm->setRepositoryClass('path', 'Composer\Repository\PathRepository');
+        $repo = $rm->createRepository('path', array('type' => 'path', 'url' => __DIR__, 'only' => array('foo/bar')));
+
+        $this->assertInstanceOf('Composer\Repository\FilterRepository', $repo);
+        $this->assertInstanceOf('Composer\Repository\PathRepository', $repo->getRepository());
+    }
 }