cloud-init generates ordering cycle via After=cloud-init in systemd-fsck

I can reproduce this fairly easily with a standard ubuntu image.

a.) launch an instance on openstack (or somewhere with an additional disk)

   upgrade cloud-init if necessary.

b.) create a filesystem and put it in /etc/fstab with simple options.
  on my opensstack cloud-init sets up an entry like:

   /dev/vdb /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

  but what we want is just a simple line:

   /dev/vdb /mnt auto defaults 0 2

c.) reboot

I'm attaching 'journal -o short-precise' output for first boot and after reboot (with the failure).

I suggest reverting the commit causing the regression and fixing LP: #1691489 in an other way.

This bug was fixed in the package cloud-init - 0.7.9-280-ge626966e-0ubuntu1

---------------
cloud-init (0.7.9-280-ge626966e-0ubuntu1) artful; urgency=medium

  * debian/rules: install rsyslog file with 0644 mode instead of 0755.
  * debian/rules, debian/apport-launcher.py: add an apport hook. (LP: #1607345)
  * New upstream snapshot.
    - cmdline: add collect-logs subcommand. [Chad Smith] (LP: #1607345)
    - CloudStack: consider dhclient lease files named with a hyphen.
      (LP: #1717147)
    - resizefs: Drop check for read-only device file, do not warn on
      overlayroot. [Chad Smith]
    - Do not provide systemd-fsck drop-in which could cause ordering cycles.
      [Balint Reczey] (LP: #1717477)
    - tests: Enable the NoCloud KVM platform [Joshua Powers]
    - resizefs: pass mount point to xfs_growfs [Dusty Mabe]
    - vmware: Enable nics before sending the SUCCESS event. [Sankar Tanguturi]
    - cloud-config modules: honor distros definitions in each module
      [Chad Smith] (LP: #1715738, #1715690)
    - chef: Add option to pin chef omnibus install version
      [Ethan Apodaca] (LP: #1462693)
    - tests: execute: support command as string [Joshua Powers]
    - schema and docs: Add jsonschema to resizefs and bootcmd modules
      [Chad Smith]
    - tools: Add xkvm script, wrapper around qemu-system [Joshua Powers]
    - vmware customization: return network config format
      [Sankar Tanguturi] (LP: #1675063)

 -- Scott Moser <email address hidden> Fri, 15 Sep 2017 16:09:07 -0400

Hello thermoman, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-233-ge586fe35-0ubuntu1~17.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Since this basically re-introduces LP: #1691489, we need to remember to revert the 'Fix Released' fields for each series in that bug once this goes out to -updates.

Hello thermoman, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-233-ge586fe35-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

## launched instance on Openstack
$ cat /etc/cloud/build.info
build_name: server
serial: 20170929

$ git clone http://git.launchpad.net/~smoser/cloud-init/+git/sru-info

$ sudo apt-get update -q && sudo apt-get install -q cloud-init
..
cloud-init is already the newest version (0.7.9-233-ge586fe35-0ubuntu1~16.04.1).

$ dpkg-query --show cloud-init
0.7.9-233-ge586fe35-0ubuntu1~16.04.1

$ ./sru-info/bin/save-old-data orig-boot

$ bdev="/dev/vdb";
$ sudo umount $bdev || :
$ sudo sed -i '/comment=cloudconfig/d' /etc/fstab
$ sudo mkdir -p /mnt
$ sudo mkfs.ext4 -F "$bdev"
$ echo "$bdev /mnt auto defaults 0 2" | sudo tee -a /etc/fstab

$ cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
/dev/vdb /mnt auto defaults 0 2

$ ./sru-info/bin/do-reboot

### ssh back in and see the bug.
$ journalctl -o short-precise | grep -i ordering.cycle
Oct 02 19:40:09.174479 xenial-1717477 systemd[1]: networking.service: Found ordering cycle on networking.service/start
Oct 02 19:40:09.174552 xenial-1717477 systemd[1]: networking.service: Breaking ordering cycle by deleting job apparmor.service/start
Oct 02 19:40:09.174565 xenial-1717477 systemd[1]: apparmor.service: Job apparmor.service/start deleted to break ordering cycle starting with networking.service/start
Oct 02 19:40:09.174576 xenial-1717477 systemd[1]: networking.service: Found ordering cycle on networking.service/start
Oct 02 19:40:09.174638 xenial-1717477 systemd[1]: networking.service: Breaking ordering cycle by deleting job local-fs.target/start
Oct 02 19:40:09.174650 xenial-1717477 systemd[1]: local-fs.target: Job local-fs.target/start deleted to break ordering cycle starting with networking.service/start

### enable proposed to show fix.
$ sudo ./sru-info/bin/enable-proposed
deb http://nova.clouds.archive.ubuntu.com/ubuntu/ xenial-proposed main universe

$ sudo apt-get update -q && sudo apt-get install -qy cloud-init
...
Unpacking cloud-init (0.7.9-233-ge586fe35-0ubuntu1~16.04.2) over (0.7.9-233-ge586fe35-0ubuntu1~16.04.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up cloud-init (0.7.9-233-ge586fe35-0ubuntu1~16.04.2) ...
Leaving 'diversion of /etc/init/ureadahead.conf to /etc/init/ureadahead.conf.disabled by cloud-init'

$ sudo ./sru-info/bin/save-old-data show-bug

$ sudo ./sru-info/bin/do-reboot

## ssh back in
$ journalctl -o short-precise | grep -i ordering.cycle || echo "no cycles"
no cycles

$ grep WARN /var/log/cloud-init || echo "no warnings"
no warnings

$ cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
/dev/vdb /mnt auto defaults 0 2

$ grep vdb /proc/mounts
/dev/vdb /mnt ext4 rw,relatime,data=ordered 0 0

## launch instance on OpenStack
$ cat /etc/cloud/build.info
build_name: server
serial: 20170922

$ git clone http://git.launchpad.net/~smoser/cloud-init/+git/sru-infy

$ sudo apt-get update -q && sudo apt-get install -q cloud-init
cloud-init is already the newest version (0.7.9-233-ge586fe35-0ubuntu1~17.04.1).

$ dpkg-query --show cloud-init
cloud-init 0.7.9-233-ge586fe35-0ubuntu1~17.04.1

$ sudo ./sru-info/bin/save-old-data orig-boot

$ bdev="/dev/vdb";
$ sudo umount $bdev || :
$ sudo sed -i '/comment=cloudconfig/d' /etc/fstab
$ sudo mkdir -p /mnt
$ sudo mkfs.ext4 -F "$bdev"
$ echo "$bdev /mnt auto defaults 0 2" | sudo tee -a /etc/fstab

LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
/dev/vdb /mnt auto defaults 0 2

$ sudo ./sru-info/bin/do-reboot
$ journalctl -o short-precise | grep -i ordering.cycle
Oct 02 19:53:10.852617 zesty-1717477 systemd[1]: apparmor.service: Found ordering cycle on apparmor.service/start

## enable proposed to show fix
$ sudo ./sru-info/bin/enable-proposed
deb http://nova.clouds.archive.ubuntu.com/ubuntu/ zesty-proposed main universe
ubuntu@zesty-1717477:~$

$ sudo apt-get update -q && sudo apt-get install -qy cloud-init
...
Setting up cloud-init (0.7.9-233-ge586fe35-0ubuntu1~17.04.2) ...

$ sudo ./sru-info/bin/save-old-data show-bug

$ sudo ./sru-info/bin/do-reboot

$ journalctl -o short-precise | grep -i ordering.cycle || echo "no cycles"
no cycles

$ grep WARN /var/log/cloud-init.log || echo "no warnings"
no warnings

$ cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
/dev/vdb /mnt auto defaults 0 2

$ grep vdb /proc/mounts
/dev/vdb /mnt ext4 rw,relatime,data=ordered 0 0

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package cloud-init - 0.7.9-233-ge586fe35-0ubuntu1~17.04.2

---------------
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~17.04.2) zesty; urgency=medium

  * cherry-pick a2f8ce9c: Do not provide systemd-fsck drop-in which
    could cause systemd ordering cycles (LP: #1717477).

 -- Scott Moser <email address hidden> Fri, 15 Sep 2017 15:30:01 -0400

This bug was fixed in the package cloud-init - 0.7.9-233-ge586fe35-0ubuntu1~16.04.2

---------------
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * cherry-pick a2f8ce9c: Do not provide systemd-fsck drop-in which
    could cause systemd ordering loops (LP: #1717477).

 -- Scott Moser <email address hidden> Fri, 15 Sep 2017 15:23:38 -0400

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3000