---

%PDF-1.5 %���� 1 0 obj << /S /GoTo /D (section.1) >> endobj 4 0 obj (Introduction) endobj 5 0 obj << /S /GoTo /D (section.2) >> endobj 8 0 obj (Decoupling the Weight Decay from the Gradient-based Update) endobj 9 0 obj << /S /GoTo /D (section.3) >> endobj 12 0 obj (Justification of Decoupled Weight Decay via a View of Adaptive Gradient Methods as Bayesian Filtering) endobj 13 0 obj << /S /GoTo /D (section.4) >> endobj 16 0 obj (Experimental Validation) endobj 17 0 obj << /S /GoTo /D (subsection.4.1) >> endobj 20 0 obj (Evaluating Decoupled Weight Decay With Different Learning Rate Schedules) endobj 21 0 obj << /S /GoTo /D (subsection.4.2) >> endobj 24 0 obj (Decoupling the Weight Decay and Initial Learning Rate Parameters) endobj 25 0 obj << /S /GoTo /D (subsection.4.3) >> endobj 28 0 obj (Better Generalization of AdamW) endobj 29 0 obj << /S /GoTo /D (subsection.4.4) >> endobj 32 0 obj (AdamWR with Warm Restarts for Better Anytime Performance) endobj 33 0 obj << /S /GoTo /D (subsection.4.5) >> endobj 36 0 obj (Use of AdamW on other datasets and architectures) endobj 37 0 obj << /S /GoTo /D (section.5) >> endobj 40 0 obj (Conclusion and Future Work) endobj 41 0 obj << /S /GoTo /D (section.6) >> endobj 44 0 obj (Acknowledgments) endobj 45 0 obj << /S /GoTo /D (appendix.A) >> endobj 48 0 obj (Formal Analysis of Weight Decay vs L2 Regularization) endobj 49 0 obj << /S /GoTo /D (appendix.B) >> endobj 52 0 obj (Additional Practical Improvements of Adam) endobj 53 0 obj << /S /GoTo /D (subsection.B.1) >> endobj 56 0 obj (Normalized Weight Decay) endobj 57 0 obj << /S /GoTo /D (subsection.B.2) >> endobj 60 0 obj (Adam with Cosine Annealing and Warm Restarts) endobj 61 0 obj << /S /GoTo /D (appendix.C) >> endobj 64 0 obj (An Example Setting of the Schedule Multiplier) endobj 65 0 obj << /S /GoTo /D (appendix.D) >> endobj 68 0 obj (Additional Results) endobj 69 0 obj << /S /GoTo /D [70 0 R /Fit] >> endobj 99 0 obj << /Length 3623 /Filter /FlateDecode >> stream xڥZK�������
�����eɒe˶�Z���>�ĐD(�f����c@�i]��3����w��v���\}{{��u��27K�dv���^:[Ɖ���춘��?���ۙb�b��;��g��Ӛzm�}��U�^�o_��(��l���W��^}��o�늁�F�l����oV`����Y:�g��,�|�ه��_y�}���� �8�s�Q8K��M29�+Z[�Bw�B�� ��0�w�E�;/�8��������1,8#b�E��������������e�o��{����en�v��O3&�a|��< �NR��(�sn&�n��<#�������%��_9ѿ��t:�\�ϺR4�E,]L_�>�ʷ�ՃJۻ�ۭwe��}��S�����6������B{�yo��=7���u9b��cǴ]�?��fcI�r5<�������K�8�2��i�yM��I�\�f��o��!�ʞ��o�����u������u��r��%���-̉Y>(�
3+�\��"�� Z ��ˈ|������o�#�G���A���"�H�.nx�C��8'c,�w��r���2L 1q�l��(�)�`X�f
wBj ���X�m�߼/�z�����ׅ����z������89�|:�0�'�L�46M+@�c��-X~�z^v���oֻ���5ݚ�lۼ(�� ӭ�8�{�w�3��’��V��wF�+��uYo9� ��$h����5�6�{�3�o�Ŕ�v=�
V�a���e���� � �%�թ�~�O��e6��9(��N!�FZ"/�C?R;]Zy�mڲ��;:d�9�q��靠�(�����b�+ �G|�hО���N�?Tf�uX:�$m�/vM�%�
jG��<692>!�A�K^�}I�c/���&%k��4�2�GH����lyU�`$�S��������g-@g$�e-���\'�II�ke��� ss��|�;%Y#v��Ծ�a5�^?Ҡ���7S���ϡ�iN�e�چF��q�����']��(��`�F�%��f�m5�:�ә�붬�J�!�{ܼ%��0L�u�NϷF%��I� 4���A����Ls<0�I��Az�M�?�Epڅ�l�f/�f��-��J;x��0Hx���w$�l��p���v�'=K<��A8)��9�kZ�0��N�F�8�L(��m��â�Q�$A |�%rn�R�8�F!���$�)�NIR%(zզ��N��d{�["k�kʵ��w�s��wN�.+_�lըM<�Y+��MEC�+��rl�ѭ�9N�jX5�N Y7ĭo^ ��C�Q�������8��ʤ_�K������e;�$�y�������M�W�PRҬ��b�>�W�����|J��=R[j�NLo�q�;�=9/��7dЏ��[-��>�
�U�u�2��"���N���Ց�]�V�lR����I�Ъ�6Ǯz��ć z���{�+���Q� B�J,b��t~9�d�%,�E����l,�B���W��Ɓ�2��i�E�|���2 ( �t�C�|����9z �]-�^i��Y'�k��}�$���B8����,Q״�Uhd�$���-Y��y��$�-�IQ%b�VK��T�M���U��
��ȴ%��� ��%R�%.���Ue�\�q�-�z�����g϶��"�g6�FJ�q1_�|�@�?N��$�Py�H����C�}�r�@������g �)����^ұW�|��g*e;��A>B��M�D�(/<��go�]S�4�6�#y�4�`�4HA_
��Kw^������@�n~���s�|;��a�r��~_�}S_O�4K3L+r
"�#,�>��W��m~-mL����܄�}�������/~� 'EPȴoLQ���'%"rR!����T ��1c�7Ӳ��}+��+'d�}�g�>qC�D�P���nc���:�J@OKY��ض�n���ز��Pmz�Դw��ҩ�q��쑙�����6��

---