Prepare 1.1.0 release (#23)

Author: f-dangel

.github/workflows/test.yaml

@@ -8,7 +8,6 @@ on:
     branches:
       - development
       - master
-      - release
 
 
 jobs:
@@ -30,8 +29,13 @@ jobs:
         python -m pip install --upgrade pip
         make install-test
     - name: Run test
+      if: contains('refs/heads/master refs/heads/development', github.ref)
       run: |
         make test
+    - name: Run test-light
+      if: contains('refs/heads/master refs/heads/development', github.ref) != 1
+      run: |
+        make test-light
 
     - name: Test coveralls - python ${{ matrix.python-version }}
       run: coveralls --service=github

README.md

@@ -11,6 +11,8 @@ for deep learning matrices, such as
 
 - the Hessian
 - the Fisher/generalized Gauss-Newton (GGN)
+- the Monte-Carlo approximated Fisher
+- the uncentered gradient covariance (aka empirical Fisher)
 
 Matrix-vector products are carried out in PyTorch, i.e. potentially on a GPU.
 The library supports defining these matrices not only on a mini-batch, but
@@ -44,20 +46,12 @@ Other features that could be supported in the future include:
 
 - Other matrices
 
-  - the un-centered gradient covariance (aka empirical Fisher)
   - the centered gradient covariance
   - terms of the [hierarchical GGN
     decomposition](https://arxiv.org/abs/2008.11865)
 
 - Block-diagonal approximations (via `param_groups`)
 
-- Inverse matrix-vector products by solving a linear system via conjugate
-  gradients
-
-  - This could allow computing generalization metrics like the Takeuchi
-    Information Criterion (TIC), using inverse matrix-vector products in
-    combination with Hutchinson trace estimation
-
 ###### Logo mage credits
 - SciPy logo: Unknown, [CC BY-SA
   4.0](https://creativecommons.org/licenses/by-sa/4.0), via Wikimedia Commons

changelog.md

@@ -6,9 +6,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.1.0] - 2023-02-19
+
+Adds various new features:
+
+### Added/New
+
+- Inverses of linear operators with multiplication via conjugate gradients
+  ([PR](https://github.com/f-dangel/curvlinops/pull/9),
+  [example](https://curvlinops.readthedocs.io/en/latest/basic_usage/example_inverses.html))
+
+- Spectral density estimation methods from [papyan2020traces](https://jmlr.org/beta/papers/v21/20-933.html)
+  ([PR](https://github.com/f-dangel/curvlinops/pull/14/files),
+  [basic example](https://curvlinops.readthedocs.io/en/latest/basic_usage/example_verification_spectral_density.html))
+
+  - Add caching to recycle Lanczos iterations between densities with different hyperparameters
+    ([PR](https://github.com/f-dangel/curvlinops/pull/15),
+     [demo 1](https://curvlinops.readthedocs.io/en/latest/basic_usage/example_verification_spectral_density.html#for-multiple-hyperparameters),
+     [demo 2](https://curvlinops.readthedocs.io/en/latest/basic_usage/example_verification_spectral_density.html#id1))
+
+- Example visualizing different supported curvature matrices
+  ([PR](https://github.com/f-dangel/curvlinops/pull/16),
+   [example](https://curvlinops.readthedocs.io/en/latest/basic_usage/example_visual_tour.html))
+
+- Linear operator for the uncentered gradient covariance matrix (aka 'empirical Fisher')
+  ([PR](https://github.com/f-dangel/curvlinops/pull/17))
+
+- Example for computing eigenvalues with `scipy.linalg.sparse.eigsh`
+  ([PR](https://github.com/f-dangel/curvlinops/pull/18),
+   [example](https://curvlinops.readthedocs.io/en/latest/basic_usage/example_eigenvalues.html))
+
+- Linear operator for a Monte-Carlo approximation of the Fisher
+  ([PR1](https://github.com/f-dangel/curvlinops/pull/20),
+   [PR2](https://github.com/f-dangel/curvlinops/pull/21),
+   [example](https://curvlinops.readthedocs.io/en/latest/basic_usage/example_fisher_monte_carlo.html))
+
+### Fixed/Removed
+
+### Internal
+
+- Refactor examples, extracting common `functorch` and array comparison methods
+  ([PR](https://github.com/f-dangel/curvlinops/pull/10))
+
+- Add description of the library on the RTD landing page
+  ([PR](https://github.com/f-dangel/curvlinops/pull/11))
+
+- Set up a proper test suite with cases
+  ([PR](https://github.com/f-dangel/curvlinops/pull/12))
+
+  - Add regression test cases
+    ([PR](https://github.com/f-dangel/curvlinops/pull/19))
+
+- Update code to latest versions of linting CI
+  ([PR](https://github.com/f-dangel/curvlinops/pull/22))
+
 ## [1.0.0] - 2022-09-30
 
 Initial release
 
-[Unreleased]: https://github.com/f-dangel/curvlinops/compare/1.0.0...HEAD
+[Unreleased]: https://github.com/f-dangel/curvlinops/compare/1.1.0...HEAD
+[1.1.0]: https://github.com/f-dangel/curvlinops/releases/tag/1.1.0
 [1.0.0]: https://github.com/f-dangel/curvlinops/releases/tag/1.0.0

curvlinops/__init__.py

@@ -1,9 +1,25 @@
 """``curvlinops`` library API."""
 
+from curvlinops.fisher import FisherMCLinearOperator
 from curvlinops.ggn import GGNLinearOperator
+from curvlinops.gradient_moments import EFLinearOperator
 from curvlinops.hessian import HessianLinearOperator
+from curvlinops.inverse import CGInverseLinearOperator
+from curvlinops.papyan2020traces.spectrum import (
+    LanczosApproximateLogSpectrumCached,
+    LanczosApproximateSpectrumCached,
+    lanczos_approximate_log_spectrum,
+    lanczos_approximate_spectrum,
+)
 
 __all__ = [
     "HessianLinearOperator",
     "GGNLinearOperator",
+    "EFLinearOperator",
+    "FisherMCLinearOperator",
+    "CGInverseLinearOperator",
+    "lanczos_approximate_spectrum",
+    "lanczos_approximate_log_spectrum",
+    "LanczosApproximateSpectrumCached",
+    "LanczosApproximateLogSpectrumCached",
 ]

curvlinops/_base.py

@@ -18,7 +18,7 @@
 from torch import device as torch_device
 from torch import from_numpy, tensor, zeros_like
 from torch.autograd import grad
-from torch.nn import Module
+from torch.nn import Module, Parameter
 from torch.nn.utils import parameters_to_vector
 from tqdm import tqdm
 
@@ -33,7 +33,7 @@ def __init__(
         self,
         model_func: Callable[[Tensor], Tensor],
         loss_func: Callable[[Tensor, Tensor], Tensor],
-        params: List[Tensor],
+        params: List[Parameter],
         data: Iterable[Tuple[Tensor, Tensor]],
         progressbar: bool = False,
         check_deterministic: bool = True,
@@ -87,7 +87,7 @@ def __init__(
                 self.to_device(old_device)
 
     @staticmethod
-    def _infer_device(params: List[Tensor]) -> torch_device:
+    def _infer_device(params: List[Parameter]) -> torch_device:
         """Infer the device on which to carry out matvecs.
 
         Args:
@@ -154,7 +154,7 @@ def _check_deterministic(self):
             self.print_nonclose(grad1, grad2, rtol, atol)
             raise RuntimeError("Check for deterministic gradient failed.")
 
-        v = rand(self.shape[0])
+        v = rand(self.shape[0]).astype(self.dtype)
         mat_v1 = self @ v
         mat_v2 = self @ v
 
@@ -196,7 +196,7 @@ def _matvec(self, x: ndarray) -> ndarray:
         x_list = self._preprocess(x)
         out_list = [zeros_like(x) for x in x_list]
 
-        for (X, y) in self._loop_over_data():
+        for X, y in self._loop_over_data():
             normalization_factor = self._get_normalization_factor(X, y)
 
             for mat_x, current in zip(out_list, self._matvec_batch(X, y, x_list)):
@@ -268,7 +268,7 @@ def _loop_over_data(self) -> Iterable[Tuple[Tensor, Tensor]]:
         if self._progressbar:
             data_iter = tqdm(data_iter, desc="matvec")
 
-        for (X, y) in data_iter:
+        for X, y in data_iter:
             X, y = X.to(self._device), y.to(self._device)
             yield (X, y)
 
@@ -283,7 +283,7 @@ def gradient_and_loss(self) -> Tuple[List[Tensor], Tensor]:
         total_loss = tensor([0.0], device=self._device)
         total_grad = [zeros_like(p) for p in self._params]
 
-        for (X, y) in self._loop_over_data():
+        for X, y in self._loop_over_data():
             loss = self._loss_func(self._model_func(X), y)
             normalization_factor = self._get_normalization_factor(X, y)
 

curvlinops/examples/__init__.py

@@ -0,0 +1 @@
+"""Contains functionality for examples in the documentation."""