Python3

This page tracks the progress of Python 3 effort porting for OpenStack.

IRC: #openstack-dev

Join the #openstack-dev IRC channel on the Freenode network to discuss Python 3.

Current Status

Since Ussuri, OpenStack is Python3 only except Swift and Storlets. Python2.7 support and testing has been removed from all other projects. For more details, please refer to Drop py2.7 community goal

Python 3

Why should OpenStack move to Python 3 right now?

Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.

Python 3 Status of OpenStack projects

Common Libraries (Oslo Projects)

See programs.yaml for the list of Common Libraries.

Development tools

OpenStack clients

OpenStack Libraries

Dependencies: Environment markers

Example to not install the "futures" dependency on Python 3.3 and newer:

futures; python_version < '3.2'

OpenStack applications (tc:approved-release)

OpenStack applications approved by the OpenStack Technical Committee.

git clone https://bitbucket.org/tsg-/liberasurecode/
cd liberasurecode
/autogen.sh && /configure --prefix=/usr && make && sudo make install

Other OpenStack Applications and Projects

List of all OpenStack projects: projects.yaml.

Unsorted projects:

• App Catalog
• Chef OpenStack (Ruby)
• Kolla: ~2100 lines of Python, but not "tox -e py27" target
• OpenStack Ansible (Python): Ansible playbooks and roles for deployment
• Puppet OpenStack (Ruby)
• TripleO

Python 2: Python 2.6 support dropped, Python 2.7 only

OpenStack Liberty targets Python 2.7 and 3.4.

Python 2.6 support is being dropped in OpenStack since OpenStack Juno for servers. Python 2.6 support is currently kept in Oslo libraries and clients. See juno-cross-project-future-of-python etherpad.

Python 3.3 support is being dropped since OpenStack Liberty.

Python 2.6 is slowly being dropped in the whole OpenStack project, ex: Remove p26 job from DIB.

Python 2.6 support will be removed in Oslo and Clients clients for OpenStack Mitaka: Oslo libraries dropping python 2.6 compatability.

Port Python 2 code to Python 3

OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. As we are targeting Python 3.4 and up, there is no need to avoid u'unicode' syntax. Do not use six.u('unicode').

Before you begin

If you're doing development with Ubuntu/Debian (and not using devstack with the USE_PYTHON flag set), then you'll need the following packages installed to run the py34 tox unit test targets in the projects:

 sudo apt-get install python3.4 python3.4-dev

sixer tool

The sixer tool helps to replace most basic patterns to add Python 3 compatibility and it respects OpenStack coding style.

Common patterns

• Replace "for key in dict.iterkeys()" with "for key in dict"
• Replace dict.iteritems() with dict.items()
• Replace dict.itervalues() with dict.values()

Note: Replacing dict.iteritems()/.itervalues() with six.iteritems(dict)/six.itervalues(dict) was preferred in the past, but there was a discussion suggesting to avoid six for this. The overhead of creating a temporary list on Python 2 is negligible.

• Replace iterator.next() with next(iterator)
• Replace basestring with six.string_types
• Replace unicode with six.text_type
• Replace (str, unicode) with six.string_types
• Replace (int, long) with six.integer_types
• Replace func.func_name with func.__name__
• Replace exceptions.OSError with OSError and remove "import exceptions"
• map() and filter() if a list is needed on Python 3:
  • Replace map(func, data) with [func(item) for item in data]
  • Replace filter(lambda obj: test(obj), data) with [obj for obj in data if test(obj)]
  • Note: Usually, tests fail because map() or filter() objects have no length

Serialization: base64, JSON, etc.

• [with oslo.serialization 1.10 or newer] Get the base64 module from oslo_serialization (from oslo_serialization import base64) to get functions:
  • oslo_serialization.base64.decode_as_bytes(encoded)
  • oslo_serialization.base64.decode_as_text(encoded, encoding='utf-8')
  • oslo_serialization.base64.encode_as_bytes(s, encoding='utf-8')
  • oslo_serialization.base64.encode_as_text(s, encoding='utf-8')
• Replace text.encode('base64') and base64.b64encode(text) with:
  • base64.b64encode(text): only accept bytes and returns bytes,
  • or: oslo_serialization.base64.encode_as_bytes(text): accept bytes or Unicode and returns bytes
  • or: oslo_serialization.base64.encode_as_text(text): accept bytes and Unicode and returns Unicode
  • Warning: base64.encodestring(raw) adds a newline ("\n"), whereas encode_as_bytes() and encode_as_text() don't.
• Replace raw.decode('base64') with:
  • base64.b64decode(raw): return bytes
  • oslo_serialization.base64.decode_as_bytes(encoded): accept bytes and Unicode, returns bytes
  • oslo_serialization.base64.decode_as_text(encoded): accept bytes and Unicode, returns Unicode

Hexadecimal:

• Replace raw.decode('hex') with binascii.unhexlify(raw)
• Replace bytes.encode('hex') with binascii.hexlify(bytes)

JSON:

• [with oslo.serialization 1.10 or newer] Replace json.dumps(obj) with oslo_serialization.jsonutils.dump_as_bytes(obj)
• Replace json.loads(obj) with oslo_serialization.jsonutils.loads(obj): it accepts bytes and Unicode, bytes is decoded from UTF-8. It avoids "if isinstance(obj, bytes): obj = obj.decode('utf-8')" which may require a second temporary variable.

contextlib.nested

To replaced contextlib.nested is to use contextlib.ExitStack. It's available on Python 2 using contextlib2.ExitStack. For contextlib.nested, nova/test.py defines:

if six.PY3:
 @contextlib.contextmanager
 def nested(*contexts):
 with contextlib.ExitStack() as stack:
 yield [stack.enter_context(c) for c in contexts]
else:
 nested = contextlib.nested

Another option to replaced contextlib.nested is to use the @mock.patch decorator. Example with nested function:

def test_thing(self):
 @mock.patch(...)
 @mock.patch(...)
 @mock.patch(...)
 def do_test(...):
 ...

 do_test()

More options:

• Mock provides a context that patches multiple things so that no nesting is needed: mock.patch.multiple()
• oslotest provides fixtures for mock, so you don't need a context: oslotest.mockpatch.

oslo_utils.encodeutils.to_utf8

oslo.utils 3.5 has an oslo_utils.encodeutils.to_utf8() function to encode Unicode to UTF-8 and return bytes unchanged.

bytes.decode and unicode.encode

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write data.decode() or text.encode() without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: data.decode('utf-8') or text.encode('utf-8'). The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

safe_decode

Olso Incubator has a function safe_decode() which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is sys.stdin.encoding or sys.getdefaultencoding():

• Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
• Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

• safe_decode(data, 'utf-8'): decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:

• safe_decode(data)

By default, the decoder is strict. You can specify a different error handler using the optional errors parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

safe_encode

Olso Incubator has a function safe_encode() which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.

• safe_encode(text) encodes text to the output encoding
• safe_encode(bytes) may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (incomding parameter) is sys.stdin.encoding or sys.getdefaultencoding():

• Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
• Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (encoding parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

• safe_encode(data, incoming='utf-8'): encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:

• safe_encode(data)

Example:

• safe_encode(b'\xe9', incoming='latin-1') returns b'\xc3\xa9'.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional errors parameter. Example: safe_encode(b'[\xff]', incoming='ascii', errors='ignore') returns b'[]'.

logging module and format exceptions

The exception_to_unicode(exc) function of oslo_utils.encodeutils is the recommanded way to format an exception to Unicode. This function works on Python 2 and Python 3 and it should avoid mojibake is most cases.

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs b'hello' instead of 'hello'.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

• str(exc): native string, so use bytes on Python 2
• six.text_type(exc): always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: unicode(Exception("nonascii:\xe9")).
• six.u(str(exc)): unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: unicode(str(Exception("\xff")))
• LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using str(exc). Example: LOG.debug(str(exc)).

HTTP

The HTTP protocol is based on bytes:

• HTTP body contains bytes. For example, use io.BytesIO for a stream storing an HTTP body.
• HTTPConnection.getresponse().read() returns bytes (in Python 3, str which is bytes in Python 2)
• On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
• It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and HTTP header should use what character encoding?

References to port Python 2 code to Python 3

• Porting to Python 3 Book by Lennart Regebro, especially the Language differences and workarounds.
• HOWTO: Porting Python 2 Code to Python 3 by Brett Cannon
• Porting Python Code to 3.x
• python-incompatibility: Demonstrates incompatibilities between Python versions.

Common pitfalls

What is a string ?

You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:

 >>> type('foo')
 <type 'str'>
 >>> type(u'foo')
 <type 'unicode'>
 >>> type(b'foo')
 <type 'str'>
 >>> isinstance('foo', six.text_type)
 False
 >>> isinstance(u'foo', six.text_type)
 True
 >>> bytes is str
 True
 >>> b'foo'[0]
 'f'

Python 3:

 >>> type('foo')
 <class 'str'>
 >>> type(u'foo')
 <class 'str'>
 >>> type(b'foo')
 <class 'bytes'>
 >>> isinstance('foo', six.text_type)
 True
 >>> isinstance(b'foo', six.text_type)
 False
 >>> bytes is str
 False
 >>> b'foo'[0]
 102

tox/testr error: db type could not be determined

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/" and then run "tox -e py34" before running "tox -e py27". You only have to do this once. The problem only occurs with "tox -e py34" when .testrepository/ was created by Python 2.

Enable Python 3 in DevStack

• Fedora::

sudo dnf install python3-devel
sudo python3 -m pip install python-memcached # needed by keystonemiddleware

• localrc: add the following line

USE_PYTHON3=True

See Switch to python 3.5.

Functional and Integration Tests

Etherpad: https://etherpad.openstack.org/p/support-python3.5-functional-tests

Doug Hellmann wrote a cross-project specification to run integration tests on Python 3: Enabling Python 3 for Application Integration Tests (spec approved for Liberty).

Patch for DevStack: Enable optional Python 3 support.

Projects wanting to run their existing functional test suite against their services on Python 3 should:-

• Add the Python3.4 trove classifier to the setup.cfg configuration file of the respective service.
• set USE_PYTHON3=True in Devstack's localrc config file before running stack.sh after the DevStack change is merged or cherry-pick until then.
• set basepython = python3.5 in tox for the necessary functional/integration tests environment.

Then run tox -e<your functional test suite name>. Example:- tox -efunctional to run the functional the tests of a certain project.

Once the functional test suite is reliably passing against the service in a python 3.5 env, the trove classifiers must be updated along with a new gate job being added to the project.

Sirushti Murugesan's attempt on Heat: https://review.openstack.org/#/c/188033/

Python 3.5

• project-config: Add python35 jobs
• nova: Add a py35 environment to tox
• Neutron issues: https://bugs.launchpad.net/neutron/+bug/1559191
• There is no Jenkins job using Python 3.5 yet
• /usr/bin/python3 is Python 3.5 in Fedora 24, Ubuntu 16.04 LTS (Xenial Xerus) and Debian Stretch (Testing)

Python 3.6 status

Python 3.6 is scheduled to be the default/only version of Python available in Ubuntu 18.04 LTS (Bionic Beaver) and Debian 10 (Buster).

Known issues:

• eventlet<0.21.0 is broken with Python 3.6, see https://github.com/eventlet/eventlet/issues/371. There is also https://github.com/eventlet/eventlet/issues/401 which is fixed in 0.22.0, so it might seem feasible to upgrade to eventlet-0.22.1 once we can do proper testing.
• Horizon explicitly only claims support for Python 3.5, when running devstack with Python 3.6 one needs to set:

 ENABLED_PYTHON3_PACKAGES=horizon

Status of Python 3 in Linux distributions

• ArchLinux already switched to Python 3 by default in 2011
• Ubuntu wants to remove Python 2 from the default installation for Ubuntu 18.04 LTS: see (Python) Plans for 18.04.
• Fedora schedules the switch in Fedora 23 (october 2015): Python 3 as the Default Implementation
  • Python 3 Porting Database for Fedora and history of packages tracked by the Fedora portingdb
  • Python 3 in Fedora (Python 3 in Fedora)
• RHEL: Python 3.3 and Python 3.4 are available on RHEL6 and RHEL7 using SCL.
• CentOS: Python 3.3 and Python 3.4 are available on CentOS 6 and 7 using SCL (no Red Hat subscription needed, CentOS has its own flavor of SCL)
• In Debian, the plan is to deprecate Python 2 in Stretch (aka: Debian 9, the next Stable Debian after Jessie) and to completely remove Python 2 for the Buster release of Debian (aka: Debian 10, to be released in approximatively 2019).

Check the Project Testing Interface for updates about the platforms each release is tested on.

Status of Python 2 Support Being Dropped in Dependencies

• Django 2.0 will not include Python 2 support (23 Jan 2017 we use Django>=1.8,<1.9)
  • https://github.com/django/django/pull/7867/files
  • https://github.com/django/django/pull/7871/files
• Pylons/Pyramid are also considering dropping Python 2
  • https://github.com/Pylons/pyramid/issues/2903
  • This is the same team that maintains webob, but it isn't clear if they intend to also drop support there.

OpenStack plans to start dropping Python 2 support at the start of the U cycle

Progress reports, meetings and sprints

• Pike PTG: https://etherpad.openstack.org/p/ptg-pike-python35
• Ocata Summit
  • https://etherpad.openstack.org/p/ocata-python-3
• openstack-dev: Status of the OpenStack port to Python 3 (Jun 22, 2016). 80 projects/83 (96%) have unit tests passing on Python 3, TODO (3): Nova, Swift and Trove.
• Mitaka Summit
  • Cross-project session: Python 3 was proposed, https://etherpad.openstack.org/p/mitaka-cross-project-session-planning
• Liberty Summit
  • Cross-project session on enabling app integration testing: https://etherpad.openstack.org/p/liberty-cross-project-python3
• Kilo Summit
  • Doug Hellmann: "Moving our applications to Python 3" cross-project session
  • Victor Stinner: "Port nova to Python 3", https://etherpad.openstack.org/p/liberty-nova-summit-ideas
• Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)
• Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
• Havana summit notes: https://etherpad.openstack.org/p/havana-python3

Articles

• Status of Python 3 in OpenStack Mitaka (Victor Stinner, March 2016)
• Python 3 Status in OpenStack Liberty (Cyril Roelandt and Victor Stinner, September 2015)
• Status of the OpenStack port to Python 3 (Cyril Roelandt, February 2014)
• Why should OpenStack move to Python 3 right now? (Victor Stinner, December 2013)

Pycon Montreal 2014: Sprint Port OpenStack to Python 3

Enovance organized a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday) 2014. See the page Python3/SprintPycon2014.