Skip to content

Layer-Wise Distillation #1272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Jan 10, 2023
Merged

Layer-Wise Distillation #1272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
b87a66b
Initial Commit with Alex's Work
rahul-tuli Dec 19, 2022
8cb9da0
Update `student_names` -> `student_layer_names`
rahul-tuli Dec 19, 2022
2b9f6c0
Intermediate commit
rahul-tuli Dec 19, 2022
d634677
Styling
corey-nm Dec 19, 2022
5e9e037
Reorg initialize
corey-nm Dec 21, 2022
54c9f9b
More cleanups
corey-nm Dec 21, 2022
d4f498d
Update docstring
corey-nm Dec 21, 2022
dbaa81e
Moving finalize logic to update
corey-nm Dec 21, 2022
9e53ae9
Tests passing a bit
corey-nm Dec 21, 2022
fcda766
Fixing lifecycle tests
corey-nm Dec 21, 2022
65a6fa1
Changing projection to dict
corey-nm Dec 21, 2022
4eec356
Cleanup
corey-nm Dec 21, 2022
4db457d
Adding quantization hooks test
corey-nm Dec 21, 2022
a2b2d07
Add failing test for optimizer serialization
rahul-tuli Dec 21, 2022
3ee6f84
Monkey patching optimizer state_dict method
corey-nm Dec 21, 2022
5422e8e
Merge branch 'main' into layer-wise-distillation
corey-nm Dec 21, 2022
d68dda8
Apply suggestions from code review
rahul-tuli Dec 29, 2022
c8bf694
Update src/sparseml/pytorch/sparsification/distillation/modifier_per_…
corey-nm Jan 3, 2023
1bdc4a6
Merge branch 'main' into layer-wise-distillation
corey-nm Jan 3, 2023
f5cfce3
Merge branch 'main' into layer-wise-distillation
corey-nm Jan 5, 2023
2eedb93
Adding missing docstrings
corey-nm Jan 5, 2023
30084aa
Respond to review on modifier/optimizer state_dict
corey-nm Jan 5, 2023
96facf9
Add a test for modifier load before forward pass
rahul-tuli Jan 5, 2023
8f6a110
Updating comments
corey-nm Jan 5, 2023
1d65a32
Fix failing test
rahul-tuli Jan 5, 2023
6febf27
Add more asserts based on @bfineran 's comments
rahul-tuli Jan 5, 2023
64a851f
* Rename `_DISTILL_PARAM_GROUP_KEY` -> `DISTILL_PARAM_GROUP_KEY`
rahul-tuli Jan 10, 2023
350dbd8
Move state dict patching to a helper function
rahul-tuli Jan 10, 2023
52a6161
Merge remote-tracking branch 'origin/main' into layer-wise-distillation
rahul-tuli Jan 10, 2023
1078f00
Merge branch 'main' into layer-wise-distillation
corey-nm Jan 10, 2023
412eef6
Quality
corey-nm Jan 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/sparseml/pytorch/sparsification/distillation/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@
# limitations under the License.

from .modifier_distillation import *
from .modifier_per_layer import *
1 change: 1 addition & 0 deletions src/sparseml/pytorch/sparsification/distillation/modifier_distillation_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,7 @@ def loss_update(
teacher_outputs=teacher_outputs,
student_labels=student_labels,
teacher_labels=teacher_labels,
optimizer=optimizer,
)

total_loss = self.compute_total_loss(loss, distillation_loss)
Expand Down
Loading