← Back to team overview

launchpad-reviewers team mailing list archive

[Merge] ~cjwatson/launchpad:bs4 into launchpad:master

 

Colin Watson has proposed merging ~cjwatson/launchpad:bs4 into launchpad:master with ~cjwatson/launchpad:bs4-feeds as a prerequisite.

Commit message:
Port lp.testing.pages to Beautiful Soup 4

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/378001

For now, the various pagetest helper functions still support taking input from Beautiful Soup 3, but if they are given text then they'll parse it using Beautiful Soup 4.

This entails a large number of changes to individual tests that aren't practical to split up, mainly because of different output formatting: Beautiful Soup 4 orders tag attributes differently, omits the space before "/>", and converts HTML entities to Unicode characters.  There are also some changes due to the "class" attribute now being treated as a multi-valued attribute.  For the most part I've gone with the flow on these, though there are still quite a few places where we're using old-style constructions.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:bs4 into launchpad:master.
diff --git a/lib/lp/answers/browser/tests/test_questiontarget.py b/lib/lp/answers/browser/tests/test_questiontarget.py
index b450029..2c842c0 100644
--- a/lib/lp/answers/browser/tests/test_questiontarget.py
+++ b/lib/lp/answers/browser/tests/test_questiontarget.py
@@ -15,6 +15,7 @@ from lazr.restful.interfaces import (
     IWebServiceClientRequest,
     )
 from simplejson import dumps
+import six
 from zope.component import getUtility
 from zope.security.proxy import removeSecurityProxy
 from zope.traversing.browser import absoluteURL
@@ -249,7 +250,7 @@ class QuestionSetViewTestCase(TestCaseWithFactory):
         target_widget = view.widgets['scope'].target_widget
         self.assertIsNot(
             None, content.find(True, id=target_widget.show_widget_id))
-        text = str(content)
+        text = six.text_type(content)
         picker_vocab = "DistributionOrProductOrProjectGroup"
         self.assertIn(picker_vocab, text)
         focus_script = "setFocusByName('field.search_text')"
diff --git a/lib/lp/answers/browser/tests/views.txt b/lib/lp/answers/browser/tests/views.txt
index b8cf752..6056091 100644
--- a/lib/lp/answers/browser/tests/views.txt
+++ b/lib/lp/answers/browser/tests/views.txt
@@ -893,7 +893,7 @@ Summary, Created, Submitter, Assignee, and Status.
     >>> for row in table.findAll('tr'):
     ...     print(extract_text(row))
     Summary                Created     Submitter      Assignee  Status
-    6 Newly installed...  2005-10-14   Sample Person  —   Answered ...
+    6 Newly installed...  2005-10-14   Sample Person  —         Answered ...
 
 Distribution display the "Source Package" column. The name of the source
 package is displayed if it exists.
@@ -910,8 +910,8 @@ package is displayed if it exists.
     >>> for row in table.findAll('tr'):
     ...     print(extract_text(row))
     Summary  Created     Submitter      Source Package   Assignee  Status ...
-    8 ...    2006-07-20  Sample Person  mozilla-firefox  —   Answered
-    7 ...    2005-10-14  Foo Bar        —          —   Needs ...
+    8 ...    2006-07-20  Sample Person  mozilla-firefox  —         Answered
+    7 ...    2005-10-14  Foo Bar        —                —         Needs ...
 
 ProjectGroups display the "In" column to show the product name.
 
@@ -930,7 +930,7 @@ ProjectGroups display the "In" column to show the product name.
     >>> for row in table.findAll('tr'):
     ...     print(extract_text(row))
     Summary  Created     Submitter      In               Assignee  Status
-    6 ...    2005-10-14  Sample Person  Mozilla Firefox  —   Answered...
+    6 ...    2005-10-14  Sample Person  Mozilla Firefox  —         Answered...
 
 The Assignee column is always displayed. It contains The person assigned
 to the question, or an m-dash if there is no assignee.
@@ -951,7 +951,7 @@ to the question, or an m-dash if there is no assignee.
     ...     print(extract_text(row))
     Summary  Created     Submitter      Assignee  Status
     6 ...    2005-10-14  Sample Person  Bob       Answered
-    4 ...    2005-09-05  Foo Bar        —   Open ...
+    4 ...    2005-09-05  Foo Bar        —         Open ...
 
 
 ManageAnswerContactView
diff --git a/lib/lp/answers/stories/faq-browse-and-search.txt b/lib/lp/answers/stories/faq-browse-and-search.txt
index 949bd50..4dd94b5 100644
--- a/lib/lp/answers/stories/faq-browse-and-search.txt
+++ b/lib/lp/answers/stories/faq-browse-and-search.txt
@@ -127,8 +127,7 @@ matches.)
 
     >>> message = find_main_content(browser.contents).find('p')
     >>> print(extract_text(message))
-    You can also consult the list of 1 question(s)
-        matching “plugin”.
+    You can also consult the list of 1 question(s) matching “plugin”.
 
 Following the link will show the questions results:
 
@@ -146,8 +145,7 @@ the same keywords.
 
     >>> message = find_tag_by_id(browser.contents, 'found-matching-faqs')
     >>> print(extract_text(message))
-    You can also consult the list of 1 FAQ(s)
-        matching “plugin”.
+    You can also consult the list of 1 FAQ(s) matching “plugin”.
 
 Following the link will show the questions results:
 
diff --git a/lib/lp/answers/stories/question-browse-and-search.txt b/lib/lp/answers/stories/question-browse-and-search.txt
index 24ae67e..8ce1c90 100644
--- a/lib/lp/answers/stories/question-browse-and-search.txt
+++ b/lib/lp/answers/stories/question-browse-and-search.txt
@@ -261,11 +261,11 @@ source package's question listing.
     >>> browser.title
     'Questions : mozilla-firefox package : Ubuntu'
     >>> soup = find_main_content(browser.contents)
-    >>> soup('table', 'listing')
-    [<table...
+    >>> print(soup.find('table', 'listing'))
+    <table...
     ...mailto: problem in webpage...2006-07-20...
     ...Installation of Java Runtime Environment for Mozilla...2006-07-20...
-    </table>]
+    </table>
 
 Average Joe wants to see all questions but listed from the oldest to the
 newest. Again, he adds the 'Invalid' status to the selection and
@@ -276,12 +276,12 @@ selects the 'oldest first' sort order.
     >>> browser.getControl('Search', index=0).click()
 
     >>> soup = find_main_content(browser.contents)
-    >>> soup('table', 'listing')
-    [<table...
+    >>> print(soup.find('table', 'listing'))
+    <table...
     ...Firefox is slow and consumes too much RAM...2005-09-05...
     ...Installation of Java Runtime Environment for Mozilla...2006-07-20...
     ...mailto: problem in webpage...2006-07-20...
-    </table>]
+    </table>
 
 
 == Common Reports ==
@@ -770,7 +770,7 @@ Entering an invalid project also displays an error message:
 
     >>> for message in find_tags_by_class(anon_browser.contents, 'message'):
     ...     print(message.renderContents())
-    There is no project named &#x27;invalid&#x27; registered in Launchpad
+    There is no project named 'invalid' registered in Launchpad
 
 If the browser supports javascript, there is a 'Choose' link available
 to help the user find an existing project. Since the test browser does
diff --git a/lib/lp/answers/stories/question-edit.txt b/lib/lp/answers/stories/question-edit.txt
index 831bc04..545c128 100644
--- a/lib/lp/answers/stories/question-edit.txt
+++ b/lib/lp/answers/stories/question-edit.txt
@@ -39,7 +39,7 @@ And viewing that page should show the updated information.
 
     >>> soup = find_main_content(test_browser.contents)
     >>> print(soup.find('div', 'report').renderContents().strip())
-    <p>Hi! I&#x27;m trying to learn about SVG but I can&#x27;t get it to
+    <p>Hi! I'm trying to learn about SVG but I can't get it to
     work at all in firefox. Maybe there is a plugin? Help! Thanks.
     Mark</p>
     >>> print(soup.find('h1').renderContents())
diff --git a/lib/lp/answers/stories/question-message.txt b/lib/lp/answers/stories/question-message.txt
index a2a6c0f..d101e3d 100644
--- a/lib/lp/answers/stories/question-message.txt
+++ b/lib/lp/answers/stories/question-message.txt
@@ -76,7 +76,7 @@ class.
 
     >>> print(text.findAll('p')[-1])
     <p><span class="foldable">--...
-    &lt;email address hidden&gt;<br />
+    &lt;email address hidden&gt;<br/>
     Witty signatures rock!
     </span></p>
 
@@ -88,9 +88,9 @@ tag of 'foldable' class, citation lines are always displayed. Again
 we can continue with the anonymous user to see the markup.
 
     >>> print(text.findAll('p')[-2])
-    <p>Somebody said sometime ago:<br />
+    <p>Somebody said sometime ago:<br/>
     <span class="foldable-quoted">
-    &gt; 1. Remove the letters  c, j, q, x, w<br />
-    &gt;    from the English Language.<br />
+    &gt; 1. Remove the letters  c, j, q, x, w<br/>
+    &gt;    from the English Language.<br/>
     &gt; 2. Remove the penny from US currency.
     </span></p>
diff --git a/lib/lp/answers/stories/question-obfuscation.txt b/lib/lp/answers/stories/question-obfuscation.txt
index 59c04d9..927268a 100644
--- a/lib/lp/answers/stories/question-obfuscation.txt
+++ b/lib/lp/answers/stories/question-obfuscation.txt
@@ -38,7 +38,7 @@ in the question's description.
     >>> description = find_main_content(user_browser.contents).p
     >>> description.renderContents()
     'I am not able to open my email client if i click on a
-     <a ... href="mailto:user@xxxxxxxxxx";>mailto:...user@xxxxxxxxxxxx</a>
+     <a href="mailto:user@xxxxxxxxxx"; ...>mailto:...user@xxxxxxxxxxxx</a>
      link ...'
 
 No Privileges Person can see email addresses in the FAQ's
diff --git a/lib/lp/answers/stories/question-reject-and-change-status.txt b/lib/lp/answers/stories/question-reject-and-change-status.txt
index 4f87f04..3a47d02 100644
--- a/lib/lp/answers/stories/question-reject-and-change-status.txt
+++ b/lib/lp/answers/stories/question-reject-and-change-status.txt
@@ -67,7 +67,7 @@ and the rejection message is added to the question board.
 
     >>> content = find_main_content(admin_browser.contents)
     >>> print(content.findAll('div', 'boardCommentBody')[-1].renderContents())
-    <p>Rejecting because it&#x27;s a duplicate of <a...>bug #1</a>.</p>
+    <p>Rejecting because it's a duplicate of <a...>bug #1</a>.</p>
 
 The call to help with this problem is not displayed.
 
@@ -126,7 +126,7 @@ status change:
     >>> for error in find_tags_by_class(admin_browser.contents, 'message'):
     ...     print(error.renderContents())
     There are 2 errors.
-    You didn&#x27;t change the status.
+    You didn't change the status.
     You must provide an explanation message.
 
 To correct the mistake of the previous example, the administrator would
@@ -153,5 +153,5 @@ and the explanation message is added to the question discussion:
 
     >>> content = find_main_content(admin_browser.contents)
     >>> print(content.findAll('div', 'boardCommentBody')[-1].renderContents())
-    <p>Setting status back to &#x27;Open&#x27;. Questions similar to a
+    <p>Setting status back to 'Open'. Questions similar to a
     bug report should be linked to it, not rejected.</p>
diff --git a/lib/lp/answers/stories/question-workflow.txt b/lib/lp/answers/stories/question-workflow.txt
index 8b4bae1..6fd675d 100755
--- a/lib/lp/answers/stories/question-workflow.txt
+++ b/lib/lp/answers/stories/question-workflow.txt
@@ -141,7 +141,7 @@ discussion.
     Status: Open ...
 
     >>> print_last_comment(owner_browser.contents)
-    The following SVG doesn&#x27;t display properly:
+    The following SVG doesn't display properly:
     http://www.w3.org/2001/08/rdfweb/rdfweb-chaals-and-dan.svg
 
 
@@ -180,8 +180,8 @@ When the owner comes back on the question page, they will now see a new
     ...     'http://launchpad.test/firefox/+question/2')
     >>> soup = find_main_content(owner_browser.contents)
     >>> soup.findAll('div', 'boardComment')[-1].find('input', type='submit')
-    <input type="submit" name="field.actions.confirm"
-     value="This Solved My Problem" />
+    <input name="field.actions.confirm" type="submit"
+     value="This Solved My Problem"/>
 
 (Note although we have three comments on the question, that's the only
 one that has this button. Only answers have this button.)
@@ -216,7 +216,7 @@ The confirmed answer is also highlighted.
     >>> soup = find_main_content(owner_browser.contents)
     >>> bestAnswer = soup.findAll('div', 'boardComment')[-2]
     >>> print(bestAnswer.find('img'))
-    <img src="/@@/favourite-yes" ... title="Marked as best answer" />
+    <img ... src="/@@/favourite-yes" ... title="Marked as best answer"/>
 
     >>> print(soup.find(
     ...     'div', 'boardCommentBody highlighted').renderContents())
@@ -279,8 +279,7 @@ its status back to 'Open'.
     >>> print_last_comment(owner_browser.contents)
     Actually, there are still SVGs that do not display correctly.
     For example, the following
-    http://people.w3.org/maxf/ChessGML/immortal.svg doesn&#x27;t
-    display correctly.
+    http://people.w3.org/maxf/ChessGML/immortal.svg doesn't display correctly.
 
 This also removes the highlighting from the previous answer and sets the
 answerer back to None.
@@ -350,7 +349,7 @@ The answer's message is also highlighted as the best answer.
     >>> soup = find_main_content(owner_browser.contents)
     >>> bestAnswer = soup.find('img', {'title' : 'Marked as best answer'})
     >>> print(bestAnswer)
-    <img src="/@@/favourite-yes" ... title="Marked as best answer" />
+    <img ... src="/@@/favourite-yes" ... title="Marked as best answer"/>
 
     >>> answerer = bestAnswer.parent.find('a')
     >>> print(extract_text(answerer))
diff --git a/lib/lp/answers/stories/this-is-a-faq.txt b/lib/lp/answers/stories/this-is-a-faq.txt
index 3bb9edb..3627a74 100644
--- a/lib/lp/answers/stories/this-is-a-faq.txt
+++ b/lib/lp/answers/stories/this-is-a-faq.txt
@@ -173,7 +173,7 @@ with an error message.
     http://answers.launchpad.test/firefox/+question/2/+linkfaq
     >>> print_feedback_messages(user_browser.contents)
     There is 1 error.
-    You didn&#x27;t modify the linked FAQ.
+    You didn't modify the linked FAQ.
 
 To remove the FAQ, the user selects the 'No existing...' option and
 submit the form again.
@@ -186,7 +186,7 @@ The new message was added to the question:
     >>> print(extract_text(find_tags_by_class(
     ...     user_browser.contents, 'boardCommentBody')[-1]).encode(
     ...     'ascii', 'backslashreplace'))
-    Sorry, this document doesn&#x27;t really answer your question.
+    Sorry, this document doesn't really answer your question.
 
 The link was also removed from the details portlet:
 
diff --git a/lib/lp/app/browser/doc/base-layout.txt b/lib/lp/app/browser/doc/base-layout.txt
index 888c7d0..eb530d5 100644
--- a/lib/lp/app/browser/doc/base-layout.txt
+++ b/lib/lp/app/browser/doc/base-layout.txt
@@ -145,7 +145,7 @@ Page Footers
     None http://launchpad.test/
     Take the tour http://launchpad.test/+tour
     Read the guide https://help.launchpad.net/
-    Canonical&nbsp;Ltd. http://canonical.com/
+    Canonical Ltd. http://canonical.com/
     Terms of use http://launchpad.test/legal
     Data privacy https://www.ubuntu.com/legal/dataprivacy
     Contact Launchpad Support /feedback
@@ -172,7 +172,7 @@ attribute.
     ...     visibility=PersonVisibility.PRIVATE)
     >>> view = MainOnlyView(team, request)
     >>> body = find_tag_by_id(view.render(), 'document')
-    >>> print body['class']
+    >>> print ' '.join(body['class'])
     tab-overview
         main_only
         private
@@ -184,7 +184,7 @@ When the context is public, the 'public' class is in the class attribute.
     >>> team = factory.makeTeam(owner=user, name='a-public-team')
     >>> view = MainOnlyView(team, request)
     >>> body = find_tag_by_id(view.render(), 'document')
-    >>> print body['class']
+    >>> print ' '.join(body['class'])
     tab-overview main_only public yui3-skin-sam
 
 
@@ -197,7 +197,7 @@ Notifications are displayed between the breadcrumbs and the page content.
     >>> view = MainOnlyView(user, request)
     >>> body_tag = find_tag_by_id(view.render(), 'maincontent')
     >>> print str(body_tag)
-    <div id="maincontent" ...
+    <div ... id="maincontent">
       ...
       <div id="request-notifications">
         <div class="informational message">I cannot do that Dave.</div>
diff --git a/lib/lp/app/browser/doc/launchpadform-view.txt b/lib/lp/app/browser/doc/launchpadform-view.txt
index 46b554d..fc7994a 100644
--- a/lib/lp/app/browser/doc/launchpadform-view.txt
+++ b/lib/lp/app/browser/doc/launchpadform-view.txt
@@ -39,6 +39,6 @@ can be used for subordinate field indentation, for example.
     <div class="field subordinate">
     <label for="field.nickname">Nickname:</label>
     <div>
-    <input class="textType" id="field.nickname" name="field.nickname" ... />
+    <input class="textType" id="field.nickname" name="field.nickname" .../>
     </div>
     </div>
diff --git a/lib/lp/app/browser/tests/test_stringformatter.py b/lib/lp/app/browser/tests/test_stringformatter.py
index b98a3db..6b32101 100644
--- a/lib/lp/app/browser/tests/test_stringformatter.py
+++ b/lib/lp/app/browser/tests/test_stringformatter.py
@@ -517,18 +517,18 @@ class TestDiffFormatter(TestCase):
             [tag.renderContents() for tag in line_numbers])
         text = find_tags_by_class(html, 'text')
         self.assertEqual(
-            ['diff-file text',
-             'diff-header text',
-             'diff-header text',
-             'diff-chunk text',
-             'text',
-             'diff-removed text',
-             'diff-added text',
-             'diff-removed text',
-             'diff-added text',
-             'diff-comment text',
-             'diff-comment text'],
-            [str(tag['class']) for tag in text])
+            [['diff-file', 'text'],
+             ['diff-header', 'text'],
+             ['diff-header', 'text'],
+             ['diff-chunk', 'text'],
+             ['text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text'],
+             ['diff-comment', 'text'],
+             ['diff-comment', 'text']],
+            [tag['class'] for tag in text])
 
     def test_cssClasses_git(self):
         # Git diffs look slightly different, so check that they also end up
@@ -552,17 +552,17 @@ class TestDiffFormatter(TestCase):
             [tag.renderContents() for tag in line_numbers])
         text = find_tags_by_class(html, 'text')
         self.assertEqual(
-            ['diff-file text',
-             'diff-file text',
-             'diff-header text',
-             'diff-header text',
-             'diff-chunk text',
-             'text',
-             'diff-removed text',
-             'diff-added text',
-             'diff-removed text',
-             'diff-added text'],
-            [str(tag['class']) for tag in text])
+            [['diff-file', 'text'],
+             ['diff-file', 'text'],
+             ['diff-header', 'text'],
+             ['diff-header', 'text'],
+             ['diff-chunk', 'text'],
+             ['text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text']],
+            [tag['class'] for tag in text])
 
 
 class TestSideBySideDiffFormatter(TestCase):
@@ -626,23 +626,23 @@ class TestSideBySideDiffFormatter(TestCase):
             [tag.renderContents() for tag in ss_line_numbers])
         text = find_tags_by_class(html, 'text')
         self.assertEqual(
-            ['diff-file text',
-             'diff-header text',
-             'diff-header text',
-             'diff-chunk text',
-             'text',
-             'text',
-             'diff-removed text',
-             'diff-added text',
-             'diff-removed text',
-             'diff-added text',
-             'text',
-             'text',
-             'diff-removed text',
-             'diff-added text',
-             'diff-comment text',
-             'diff-comment text'],
-            [str(tag['class']) for tag in text])
+            [['diff-file', 'text'],
+             ['diff-header', 'text'],
+             ['diff-header', 'text'],
+             ['diff-chunk', 'text'],
+             ['text'],
+             ['text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text'],
+             ['text'],
+             ['text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text'],
+             ['diff-comment', 'text'],
+             ['diff-comment', 'text']],
+            [tag['class'] for tag in text])
 
     def test_cssClasses_git(self):
         # Git diffs look slightly different, so check that they also end up
@@ -673,22 +673,22 @@ class TestSideBySideDiffFormatter(TestCase):
             [tag.renderContents() for tag in ss_line_numbers])
         text = find_tags_by_class(html, 'text')
         self.assertEqual(
-            ['diff-file text',
-             'diff-file text',
-             'diff-header text',
-             'diff-header text',
-             'diff-chunk text',
-             'text',
-             'text',
-             'diff-removed text',
-             'diff-added text',
-             'diff-removed text',
-             'diff-added text',
-             'text',
-             'text',
-             'diff-removed text',
-             'diff-added text'],
-            [str(tag['class']) for tag in text])
+            [['diff-file', 'text'],
+             ['diff-file', 'text'],
+             ['diff-header', 'text'],
+             ['diff-header', 'text'],
+             ['diff-chunk', 'text'],
+             ['text'],
+             ['text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text'],
+             ['text'],
+             ['text'],
+             ['diff-removed', 'text'],
+             ['diff-added', 'text']],
+            [tag['class'] for tag in text])
 
 
 class TestOOPSFormatter(TestCase):
diff --git a/lib/lp/app/stories/basics/copyright.txt b/lib/lp/app/stories/basics/copyright.txt
index c435fc2..f10c8d7 100644
--- a/lib/lp/app/stories/basics/copyright.txt
+++ b/lib/lp/app/stories/basics/copyright.txt
@@ -5,11 +5,11 @@ The tour pages.
   >>> browser.open('http://launchpad.test/')
   >>> browser.getLink('Take the tour').click()
   >>> print extract_text(find_tag_by_id(browser.contents, 'footer-navigation'))
-  Next...&copy; 2004-2020 Canonical Ltd...
+  Next...© 2004-2020 Canonical Ltd...
 
 The main template.
 
   >>> browser.open('http://launchpad.test')
   >>> print extract_text(find_tag_by_id(browser.contents, 'footer'))
-  &copy; 2004-2020 Canonical Ltd.
+  © 2004-2020 Canonical Ltd.
   ...
diff --git a/lib/lp/app/stories/basics/demo-and-lpnet.txt b/lib/lp/app/stories/basics/demo-and-lpnet.txt
index 85431ed..c367ed9 100644
--- a/lib/lp/app/stories/basics/demo-and-lpnet.txt
+++ b/lib/lp/app/stories/basics/demo-and-lpnet.txt
@@ -44,7 +44,7 @@ For a 3-0 page:
     <style...url(/@@/demo)...</style>
     ...
     >>> print extract_text(find_tag_by_id(browser.contents, 'lp-version'))
-    &bull; r... devmode demo site (Get the code!)
+    • r... devmode demo site (Get the code!)
 
     >>> print extract_text(find_tags_by_class(
     ...     browser.contents, 'sitemessage')[0])
@@ -62,6 +62,6 @@ First for a 3-0 page:
 
     >>> browser.open('http://launchpad.test/ubuntu')
     >>> print extract_text(find_tag_by_id(browser.contents, 'lp-version'))
-    &bull; r... devmode (Get the code!)
+    • r... devmode (Get the code!)
     >>> len(find_tags_by_class(browser.contents, 'sitemessage'))
     0
diff --git a/lib/lp/app/stories/form/xx-form-layout.txt b/lib/lp/app/stories/form/xx-form-layout.txt
index bea8e66..fa3b8ec 100644
--- a/lib/lp/app/stories/form/xx-form-layout.txt
+++ b/lib/lp/app/stories/form/xx-form-layout.txt
@@ -20,7 +20,7 @@ The new team form contains an example of a normal text widget.
     <div>
     <label for="field.name">Name:</label>
     <div>
-    <input ... name="field.name" ... />
+    <input ... name="field.name" .../>
     </div>
     <p class="formHelp">....</p>
     </div>
@@ -38,7 +38,7 @@ If the text field is optional, then that is noted after the widget.
     <label for="field.defaultmembershipperiod">Subscription period:</label>
     <span class="fieldRequired">(Optional)</span>
     <div>
-    <input ... id="field.defaultmembershipperiod" ... />
+    <input ... id="field.defaultmembershipperiod" .../>
     </div>
     <p class="formHelp">...</p>
     </div>
@@ -59,7 +59,7 @@ Checkboxes have their label to the right. Let's look at one example.
     <tr>
       <td colspan="2">
        <div>
-          <input ... name="field.project_reviewed" type="checkbox" ...  />
+          <input ... name="field.project_reviewed" type="checkbox" .../>
           <label for="field.project_reviewed">Project reviewed</label>...
       </td>
     </tr>
diff --git a/lib/lp/app/stories/launchpad-root/front-pages.txt b/lib/lp/app/stories/launchpad-root/front-pages.txt
index bb91c29..68a4251 100644
--- a/lib/lp/app/stories/launchpad-root/front-pages.txt
+++ b/lib/lp/app/stories/launchpad-root/front-pages.txt
@@ -47,7 +47,8 @@ The front page also lists the recent blog posts published on the Launchpad
 blog:
 
     >>> print extract_text(
-    ...     find_tag_by_id(browser.contents, 'homepage-blogposts'))
+    ...     find_tag_by_id(browser.contents, 'homepage-blogposts'),
+    ...     formatter='html')
     Recent Launchpad blog posts
     Launchpad EPIC 2010 photo
     &ndash; 16 Jul 2010
@@ -100,4 +101,4 @@ and guide:
 
     >>> print extract_text(
     ...     find_tags_by_class(user_browser.contents, 'lp-arcana')[0])
-    &bull; Take the tour &bull; Read the guide
+    • Take the tour • Read the guide
diff --git a/lib/lp/app/stories/launchpad-search/site-search.txt b/lib/lp/app/stories/launchpad-search/site-search.txt
index d1e625d..b43a2e6 100644
--- a/lib/lp/app/stories/launchpad-search/site-search.txt
+++ b/lib/lp/app/stories/launchpad-search/site-search.txt
@@ -92,7 +92,7 @@ navigation states that the page is showing 1 through 20 of 25 total results.
     Pages matching "bug" in Launchpad
 
     >>> print_search_results()
-    1 &rarr; 20 of 25 pages matching "bug"...
+    1 → 20 of 25 pages matching "bug"...
     Launchpad Bugs...
     Bugs in Ubuntu...
     Bugs related to Sample Person...
@@ -139,7 +139,7 @@ matching ...".
     Launchpad Developers (launchpad)
     Launchpad developers
     Created on 2005-10-13...
-    1 &rarr; 20 of 25 other pages matching "launchpad"...
+    1 → 20 of 25 other pages matching "launchpad"...
     Launchpad Bugs...
 
 
@@ -241,8 +241,8 @@ search.
     >>> print extract_text(
     ...     find_main_content(anon_browser.contents), skip_tags=[])
     Pages matching "fnord" in Launchpad
-    &lt;!-- setFocusByName('field.text'); // --&gt;
-    Your search for &ldquo;fnord&rdquo; did not return any results.
+    <!-- setFocusByName('field.text'); // -->
+    Your search for “fnord” did not return any results.
 
 
 Searches when there is no page service
diff --git a/lib/lp/blueprints/browser/tests/test_specification.py b/lib/lp/blueprints/browser/tests/test_specification.py
index 913381d..a3be9e9 100644
--- a/lib/lp/blueprints/browser/tests/test_specification.py
+++ b/lib/lp/blueprints/browser/tests/test_specification.py
@@ -174,7 +174,7 @@ class TestSpecificationView(BrowserTestCase):
             spec, name='+index', principal=spec.owner,
             rootsite='blueprints')
         li = find_tag_by_id(view.render(), 'spec-url')
-        self.assertEqual('nofollow', li.a['rel'])
+        self.assertEqual(['nofollow'], li.a['rel'])
         self.assertEqual(spec.specurl, li.a['href'])
 
     def test_registration_date_displayed(self):
diff --git a/lib/lp/blueprints/browser/tests/test_specificationtarget.py b/lib/lp/blueprints/browser/tests/test_specificationtarget.py
index 6c5a61a..aca15b6 100644
--- a/lib/lp/blueprints/browser/tests/test_specificationtarget.py
+++ b/lib/lp/blueprints/browser/tests/test_specificationtarget.py
@@ -6,6 +6,7 @@ from __future__ import absolute_import, print_function, unicode_literals
 __metaclass__ = type
 
 from fixtures import FakeLogger
+import six
 from zope.component import getUtility
 from zope.security.proxy import removeSecurityProxy
 
@@ -142,9 +143,9 @@ class TestAssignments(TestCaseWithFactory):
         view = create_initialized_view(product, name='+assignments',
             query_string="batch=1")
         content = view.render()
-        self.assertEqual('next',
+        self.assertEqual(['next'],
             find_tag_by_id(content, 'upper-batch-nav-batchnav-next')['class'])
-        self.assertEqual('next',
+        self.assertEqual(['next'],
             find_tag_by_id(content, 'lower-batch-nav-batchnav-next')['class'])
 
 
@@ -309,7 +310,7 @@ class SpecificationSetViewTestCase(TestCaseWithFactory):
         target_widget = view.widgets['scope'].target_widget
         self.assertIsNot(
             None, content.find(True, id=target_widget.show_widget_id))
-        text = str(content)
+        text = six.text_type(content)
         picker_vocab = 'DistributionOrProductOrProjectGroup'
         self.assertIn(picker_vocab, text)
         focus_script = "setFocusByName('field.search_text')"
diff --git a/lib/lp/blueprints/stories/blueprints/xx-milestones.txt b/lib/lp/blueprints/stories/blueprints/xx-milestones.txt
index 36cbe2d..98d68bd 100644
--- a/lib/lp/blueprints/stories/blueprints/xx-milestones.txt
+++ b/lib/lp/blueprints/stories/blueprints/xx-milestones.txt
@@ -38,7 +38,7 @@ We expect to be redirected to the spec home page.
 And on that page, we expect to see that the spec is targeted to the 1.0
 milestone.
 
-  >>> find_main_content(admin_browser.contents)
+  >>> print(find_main_content(admin_browser.contents))
   <...Milestone target:...
   <.../firefox/+milestone/1.0...
 
diff --git a/lib/lp/blueprints/stories/sprints/xx-sprints.txt b/lib/lp/blueprints/stories/sprints/xx-sprints.txt
index d3451f7..57bc6b3 100644
--- a/lib/lp/blueprints/stories/sprints/xx-sprints.txt
+++ b/lib/lp/blueprints/stories/sprints/xx-sprints.txt
@@ -102,7 +102,7 @@ a error message.
     ...     print tag.renderContents()
     There is 1 error.
     <BLANKLINE>
-    This event can&#x27;t start after it ends
+    This event can't start after it ends
 
 Also, the date is now presented in a canonicalised format, with time in
 minutes rather than second-level accuracy:
@@ -215,7 +215,7 @@ should receive a nice error message.
     >>> for tag in find_tags_by_class(browser.contents, 'message'):
     ...     print tag.renderContents()
     There is 1 error.
-    This event can&#x27;t start after it ends
+    This event can't start after it ends
 
 We fix the dates and change the address, we expect to be redirected to the
 sprint home page.
diff --git a/lib/lp/blueprints/stories/standalone/xx-batching.txt b/lib/lp/blueprints/stories/standalone/xx-batching.txt
index ab75339..6603874 100644
--- a/lib/lp/blueprints/stories/standalone/xx-batching.txt
+++ b/lib/lp/blueprints/stories/standalone/xx-batching.txt
@@ -59,7 +59,7 @@ blueprint is listed:
   >>> browser.open("http://blueprints.launchpad.test/big-project";)
   >>> print extract_text(first_tag_by_class(browser.contents, 
   ...                                      'batch-navigation-index'))
-  1...&rarr...1...of...1 result
+  1...→...1...of...1 result
   
 Let's add some more blueprints:
 
@@ -76,18 +76,18 @@ the blueprints are listed:
   >>> browser.open("http://blueprints.launchpad.test/big-project";)
   >>> print extract_text(first_tag_by_class(browser.contents, 
   ...                                      'batch-navigation-index'))
-  1...&rarr...5...of...20 results
+  1...→...5...of...20 results
 
 We can go to the next batch of blueprints by following the 'Next' link:
     
   >>> browser.getLink('Next').click()
   >>> print extract_text(first_tag_by_class(browser.contents, 
   ...                                      'batch-navigation-index'))
-  6...&rarr...10...of...20 results
+  6...→...10...of...20 results
 
 Following the 'Last' link takes us to the last batch of blueprints:
 
   >>> browser.getLink('Last').click()
   >>> print extract_text(first_tag_by_class(browser.contents, 
   ...                                      'batch-navigation-index'))
-  16...&rarr...20...of...20 results
+  16...→...20...of...20 results
diff --git a/lib/lp/blueprints/stories/standalone/xx-personviews.txt b/lib/lp/blueprints/stories/standalone/xx-personviews.txt
index 8e6360c..81b41e0 100644
--- a/lib/lp/blueprints/stories/standalone/xx-personviews.txt
+++ b/lib/lp/blueprints/stories/standalone/xx-personviews.txt
@@ -22,10 +22,10 @@ that the user is supposed to approve.
     >>> browser.url
     '.../~name16/+specs?role=approver'
     >>> soup = find_main_content(browser.contents)
-    >>> soup('p', 'informational message')
-    [<p class="informational message">
+    >>> print(soup.find('p', 'informational message'))
+    <p class="informational message">
     No feature specifications match your criteria.
-    </p>]
+    </p>
 
 The 'Assignee' link displays a page showing the specifications assigned
 to the person.
@@ -34,10 +34,10 @@ to the person.
     >>> browser.url
     '.../~name16/+specs?role=assignee'
     >>> soup = find_main_content(browser.contents)
-    >>> soup('p', 'informational message')
-    [<p class="informational message">
+    >>> print(soup.find('p', 'informational message'))
+    <p class="informational message">
     No feature specifications match your criteria.
-    </p>]
+    </p>
 
 The 'Subscriber' link displays a page showing the specifications to
 which the person subscribed.
@@ -55,10 +55,10 @@ person drafted.
     >>> browser.url
     '.../~name16/+specs?role=drafter'
     >>> soup = find_main_content(browser.contents)
-    >>> soup('p', 'informational message')
-    [<p class="informational message">
+    >>> print(soup.find('p', 'informational message'))
+    <p class="informational message">
     No feature specifications match your criteria.
-    </p>]
+    </p>
 
 The 'Workload' link displays a page showing the specifications that are
 in the workload of that person.
diff --git a/lib/lp/blueprints/stories/standalone/xx-retargeting.txt b/lib/lp/blueprints/stories/standalone/xx-retargeting.txt
index 0435907..526bb74 100644
--- a/lib/lp/blueprints/stories/standalone/xx-retargeting.txt
+++ b/lib/lp/blueprints/stories/standalone/xx-retargeting.txt
@@ -62,4 +62,4 @@ We stay on the same page and get an error message printed out:
   ...     print tag.renderContents()
   There is 1 error.
   <BLANKLINE>
-  There is no project with the name &#x27;foo bar&#x27;. Please check that name and try again.
+  There is no project with the name 'foo bar'. Please check that name and try again.
diff --git a/lib/lp/bugs/browser/tests/bugtask-search-views.txt b/lib/lp/bugs/browser/tests/bugtask-search-views.txt
index 537c680..1f30808 100644
--- a/lib/lp/bugs/browser/tests/bugtask-search-views.txt
+++ b/lib/lp/bugs/browser/tests/bugtask-search-views.txt
@@ -149,7 +149,7 @@ message.
     ...     debian, '+bugs', form_values)
 
     >>> distro_advanced_search_listingview.getFieldError('bug_reporter')
-    u'There&#x27;s no person with the name or email address &#x27;invalid-reporter&#x27;.'
+    u"There's no person with the name or email address 'invalid-reporter'."
 
 The same if we try with an invalid assignee.
 
@@ -163,7 +163,7 @@ The same if we try with an invalid assignee.
     ...     debian, '+bugs', form_values)
 
     >>> distro_advanced_search_listingview.getFieldError('assignee')
-    u'There&#x27;s no person with the name or email address &#x27;invalid-assignee&#x27;.'
+    u"There's no person with the name or email address 'invalid-assignee'."
 
 Searching by component is possible, as long as the context has defined a
 .currentseries.
diff --git a/lib/lp/bugs/browser/tests/test_bug_views.py b/lib/lp/bugs/browser/tests/test_bug_views.py
index ccde797..628a599 100644
--- a/lib/lp/bugs/browser/tests/test_bug_views.py
+++ b/lib/lp/bugs/browser/tests/test_bug_views.py
@@ -142,12 +142,10 @@ class TestAlsoAffectsLinks(BrowserTestCase):
         browser = self.getUserBrowser(url, user=owner)
         also_affects = find_tag_by_id(
             browser.contents, 'also-affects-product')
-        self.assertIn(
-            'private-disallow', also_affects['class'].split(' '))
+        self.assertIn('private-disallow', also_affects['class'])
         also_affects = find_tag_by_id(
             browser.contents, 'also-affects-package')
-        self.assertIn(
-            'private-disallow', also_affects['class'].split(' '))
+        self.assertIn('private-disallow', also_affects['class'])
 
     def test_also_affects_links_distro_bug(self):
         # We expect that only the Also Affects Project link is disallowed.
@@ -165,12 +163,10 @@ class TestAlsoAffectsLinks(BrowserTestCase):
         browser = self.getUserBrowser(url, user=owner)
         also_affects = find_tag_by_id(
             browser.contents, 'also-affects-product')
-        self.assertIn(
-            'private-disallow', also_affects['class'].split(' '))
+        self.assertIn('private-disallow', also_affects['class'])
         also_affects = find_tag_by_id(
             browser.contents, 'also-affects-package')
-        self.assertNotIn(
-            'private-disallow', also_affects['class'].split(' '))
+        self.assertNotIn('private-disallow', also_affects['class'])
 
 
 class TestEmailObfuscated(BrowserTestCase):
diff --git a/lib/lp/bugs/browser/tests/test_bugtarget_filebug.py b/lib/lp/bugs/browser/tests/test_bugtarget_filebug.py
index 3c7946e..d96ddd4 100644
--- a/lib/lp/bugs/browser/tests/test_bugtarget_filebug.py
+++ b/lib/lp/bugs/browser/tests/test_bugtarget_filebug.py
@@ -270,7 +270,7 @@ class TestBugTargetFileBugConfirmationMessage(TestCaseWithFactory):
         filebug_form_container = filebug_form.findParents(
             id='filebug-form-container')[0]
         class_attrs = [item.strip()
-                       for item in filebug_form_container['class'].split(" ")]
+                       for item in filebug_form_container['class']]
         self.assertTrue('hidden' in class_attrs)
 
     def test_bug_filing_view_with_dupe_search_disabled(self):
diff --git a/lib/lp/bugs/browser/tests/test_bugtask.py b/lib/lp/bugs/browser/tests/test_bugtask.py
index f56d32f..3648a95 100644
--- a/lib/lp/bugs/browser/tests/test_bugtask.py
+++ b/lib/lp/bugs/browser/tests/test_bugtask.py
@@ -13,6 +13,7 @@ import urllib
 from lazr.restful.interfaces import IJSONRequestCache
 from pytz import UTC
 import simplejson
+import six
 import soupmatchers
 from testscenarios import (
     load_tests_apply_scenarios,
@@ -2414,7 +2415,7 @@ class TestBugTaskExpirableListingView(BrowserTestCase):
         title = bug.title
         content = self.getMainContent(
             bug.default_bugtask.target, "+expirable-bugs")
-        self.assertIn(title, str(content))
+        self.assertIn(title, six.text_type(content))
 
 
 class TestBugListingBatchNavigator(TestCaseWithFactory):
diff --git a/lib/lp/bugs/stories/bug-tags/xx-searching-for-tags.txt b/lib/lp/bugs/stories/bug-tags/xx-searching-for-tags.txt
index 38790d8..f28c4a4 100644
--- a/lib/lp/bugs/stories/bug-tags/xx-searching-for-tags.txt
+++ b/lib/lp/bugs/stories/bug-tags/xx-searching-for-tags.txt
@@ -33,10 +33,9 @@ If an invalid tag name is entered, an error message will be displayed.
 
     >>> for tag in find_tags_by_class(anon_browser.contents, 'message'):
     ...     print(tag.renderContents())
-    &#x27;!!invalid!!&#x27; isn&#x27;t a valid tag name. Tags must start
-    with a letter or number and be lowercase. The characters
-    &quot;+&quot;, &quot;-&quot; and &quot;.&quot; are also allowed
-    after the first character.
+    '!!invalid!!' isn't a valid tag name. Tags must start with a letter
+    or number and be lowercase. The characters "+", "-" and "." are also
+    allowed after the first character.
 
 
 == Cross-Site Scripting, or XSS ==
@@ -61,14 +60,14 @@ Indeed, the markup is valid and correctly escaped:
     >>> print(find_tag_by_id(anon_browser.contents, 'field.tag').prettify())
     <input class="textType" id="field.tag"
            name="field.tag" size="20" type="text"
-           value='&lt;script&gt;alert("cheezburger");&lt;/script&gt;' />
+           value='&lt;script&gt;alert("cheezburger");&lt;/script&gt;'/>
 
 The error message is also valid and correctly escaped:
 
     >>> for tag in find_tags_by_class(anon_browser.contents, 'message'):
     ...     print(tag.prettify())
     <div class="message">
-    &#x27;&lt;script&gt;alert(&quot;cheezburger&quot;);&lt;/script&gt;&#x27; isn&#x27;t ...
+    '&lt;script&gt;alert("cheezburger");&lt;/script&gt;' isn't ...
     </div>
 
 The script we tried to inject is not present, unescaped, anywhere in
diff --git a/lib/lp/bugs/stories/bug-tags/xx-tags-on-bug-page.txt b/lib/lp/bugs/stories/bug-tags/xx-tags-on-bug-page.txt
index add372e..fbc98a7 100644
--- a/lib/lp/bugs/stories/bug-tags/xx-tags-on-bug-page.txt
+++ b/lib/lp/bugs/stories/bug-tags/xx-tags-on-bug-page.txt
@@ -21,10 +21,9 @@ If we enter an invalid tag name, we'll get an error.
     >>> for tag in find_tags_by_class(user_browser.contents, 'message'):
     ...     print(tag.renderContents())
     There is 1 error.
-    &#x27;!!invalid!!&#x27; isn&#x27;t a valid tag name. Tags must start
-    with a letter or number and be lowercase. The characters
-    &quot;+&quot;, &quot;-&quot; and &quot;.&quot; are also allowed
-    after the first character.
+    '!!invalid!!' isn't a valid tag name. Tags must start with a letter
+    or number and be lowercase. The characters "+", "-" and "." are also
+    allowed after the first character.
 
 Let's specify two valid tags.
 
diff --git a/lib/lp/bugs/stories/bugattachments/xx-attachments-to-bug-report.txt b/lib/lp/bugs/stories/bugattachments/xx-attachments-to-bug-report.txt
index e76508b..eaf7ac8 100644
--- a/lib/lp/bugs/stories/bugattachments/xx-attachments-to-bug-report.txt
+++ b/lib/lp/bugs/stories/bugattachments/xx-attachments-to-bug-report.txt
@@ -53,7 +53,7 @@ displayed as the bug report).
     >>> link = browser.getLink('sample data')
     >>> print(link.url)
     http://bugs.launchpad.test/jokosher/+bug/11/+attachment/.../+files/test.txt
-    >>> print(comment_0.find('a', text='Edit').parent)
+    >>> print(comment_0.find('a', text='Edit'))
     <a class="sprite edit action-icon"
        href="/jokosher/+bug/11/+attachment/...">Edit</a>
 
diff --git a/lib/lp/bugs/stories/bugs/xx-bug-activity.txt b/lib/lp/bugs/stories/bugs/xx-bug-activity.txt
index 5313ceb..d2e3dc1 100644
--- a/lib/lp/bugs/stories/bugs/xx-bug-activity.txt
+++ b/lib/lp/bugs/stories/bugs/xx-bug-activity.txt
@@ -54,7 +54,7 @@ page.
     ...     """Print all the comments on the page."""
     ...     comment_divs = find_tags_by_class(page, 'boardComment')
     ...     for div_tag in comment_divs[subset]:
-    ...         print(extract_text(div_tag).replace("&#8594;", "=>"))
+    ...         print(extract_text(div_tag))
     ...         print('-' * 8)
 
     >>> user_browser.open(
@@ -68,12 +68,12 @@ page.
     on 2007-12-18
     Changed in thunderbird:
     status:
-    Unknown => New
+    Unknown → New
     --------
     No Privileges Person (no-priv)
     ...
     #7
-    Here&#x27;s a comment for testing, like.
+    Here's a comment for testing, like.
     ...
     --------
 
@@ -186,13 +186,13 @@ We'll add a milestone to Redfish to demonstrate this.
     ...
     Changed in redfish:
     assignee:
-    nobody => Foo Bar (name16)
+    nobody → Foo Bar (name16)
     importance:
-    Undecided => High
+    Undecided → High
     milestone:
-    none => foo
+    none → foo
     status:
-    New => Confirmed
+    New → Confirmed
     --------
 
 If a change is made to a bug task which is targeted to a distro source
@@ -208,7 +208,7 @@ package, the name of the package and the distro will be displayed.
     ... ago
     Changed in mozilla-firefox (Ubuntu):
     status:
-    New => Confirmed
+    New → Confirmed
     --------
 
 If a change has a comment associated with it it will be displayed in the
@@ -230,11 +230,11 @@ bundled with that comment in the UI.
     Lookit, a change!
     Changed in mozilla-firefox (Ubuntu):
     status:
-    New => Confirmed
+    New → Confirmed
     importance:
-    Medium => Low
+    Medium → Low
     status:
-    Confirmed => New
+    Confirmed → New
     Hide
     --------
 
@@ -254,7 +254,7 @@ If a target of a bug task is changed the old and new value will be shown.
     #2
     ...
     affects:
-    mozilla-firefox (Ubuntu) => linux-source-2.6.15 (Ubuntu)
+    mozilla-firefox (Ubuntu) → linux-source-2.6.15 (Ubuntu)
     Hide
     --------
 
@@ -284,7 +284,7 @@ Changes to information_type are shown.
     Foo Bar (name16)
     ... ago
     information type:
-    Public => Private
+    Public → Private
     --------
 
     >>> admin_browser.open(
@@ -296,5 +296,5 @@ Changes to information_type are shown.
     Foo Bar (name16)
     ... ago
     information type:
-    Private Security => Private
+    Private Security → Private
     --------
diff --git a/lib/lp/bugs/stories/bugs/xx-bug-affects-me-too.txt b/lib/lp/bugs/stories/bugs/xx-bug-affects-me-too.txt
index 3d792b9..e1c6458 100644
--- a/lib/lp/bugs/stories/bugs/xx-bug-affects-me-too.txt
+++ b/lib/lp/bugs/stories/bugs/xx-bug-affects-me-too.txt
@@ -94,7 +94,7 @@ dynamic (Javascript enabled) scenarios.
 
    >>> def class_filter(css_class):
    ...     def test(node):
-   ...         return css_class in node.get('class', '').split()
+   ...         return css_class in node.get('class', [])
    ...     return test
 
    >>> static_content = find_tag_by_id(
@@ -114,10 +114,10 @@ dynamic (Javascript enabled) scenarios.
 The dynamic content is hidden by the presence of the "hidden" CSS
 class.
 
-   >>> print(static_content.get('class'))
+   >>> print(' '.join(static_content.get('class')))
    static
 
-   >>> print(dynamic_content.get('class'))
+   >>> print(' '.join(dynamic_content.get('class')))
    dynamic hidden
 
 It is the responsibilty of Javascript running in the page to unhide
diff --git a/lib/lp/bugs/stories/bugs/xx-bug-comments-truncated.txt b/lib/lp/bugs/stories/bugs/xx-bug-comments-truncated.txt
index 97d75d8..f421e1a 100644
--- a/lib/lp/bugs/stories/bugs/xx-bug-comments-truncated.txt
+++ b/lib/lp/bugs/stories/bugs/xx-bug-comments-truncated.txt
@@ -65,11 +65,11 @@ The whole comment is visible on this page:
     <div class="boardCommentBody">
     <div class="comment-text" itemprop="commentText"><p>This
       would be a real killer feature. If there is already code
-      to make it possible, why aren&#x27;t there tons of press announcements
+      to make it possible, why aren't there tons of press announcements
       about the secuirty possibilities. Imagine - no more embarrassing
       emails for Mr Gates... everything they delete would actually
-      disappear! I&#x27;m sure Redmond will switch over as soon as they hear
-      about this. It&#x27;s not a bug, it&#x27;s a feature!</p></div>
+      disappear! I'm sure Redmond will switch over as soon as they hear
+      about this. It's not a bug, it's a feature!</p></div>
     </div>
 
     >>> config_data  = config.pop('max_comment_data')
@@ -157,7 +157,7 @@ includes the hooks for the style and script.
 
     >>> print(text.findAll('p')[-2])
     <p><span class="foldable">--...
-    &lt;email address hidden&gt;<br />
+    &lt;email address hidden&gt;<br/>
     Witty signatures rock!
     </span></p>
 
@@ -170,10 +170,10 @@ always displayed. Again we can continue with the anonymous user to
 see the markup.
 
     >>> print(text.findAll('p')[-3])
-    <p>Somebody said sometime ago:<br />
+    <p>Somebody said sometime ago:<br/>
     <span class="foldable-quoted">
-    &gt; 1. Remove the letters  c, j, q, x, w<br />
-    &gt;    from the English Language.<br />
+    &gt; 1. Remove the letters  c, j, q, x, w<br/>
+    &gt;    from the English Language.<br/>
     &gt; 2. Remove the penny from US currency.
     </span></p>
 
@@ -182,17 +182,17 @@ starts with '-----BEGIN PGP'.  There are two kinds of PGP blocks,
 the notice that the message is signed, and the signature.
 
     >>> print(text.findAll('p')[0])
-    <p><span class="foldable">-----BEGIN PGP SIGNED MESSAGE-----<br />
+    <p><span class="foldable">-----BEGIN PGP SIGNED MESSAGE-----<br/>
     Hash: SHA1
     </span></p>
 
     >>> print(text.findAll('p')[-1])
-    <p><span class="foldable">-----BEGIN PGP SIGNATURE-----<br />
-    Version: GnuPG v1.4.1 (GNU/Linux)<br />
-    Comment: Using GnuPG with Thunderbird<br />
-    <br />
-    iD8DBQFED60Y0F+<wbr />nu1YWqI0RAqrNAJ<wbr />...
-    T2PIWy0CUJsX8RX<wbr />St/M51WE=<br />
-    =J2S5<br />
+    <p><span class="foldable">-----BEGIN PGP SIGNATURE-----<br/>
+    Version: GnuPG v1.4.1 (GNU/Linux)<br/>
+    Comment: Using GnuPG with Thunderbird<br/>
+    <br/>
+    iD8DBQFED60Y0F+<wbr/>nu1YWqI0RAqrNAJ<wbr/>...
+    T2PIWy0CUJsX8RX<wbr/>St/M51WE=<br/>
+    =J2S5<br/>
     -----END PGP SIGNATURE-----
     </span></p>
diff --git a/lib/lp/bugs/stories/bugs/xx-bug-create-question.txt b/lib/lp/bugs/stories/bugs/xx-bug-create-question.txt
index fa21327..ad46675 100644
--- a/lib/lp/bugs/stories/bugs/xx-bug-create-question.txt
+++ b/lib/lp/bugs/stories/bugs/xx-bug-create-question.txt
@@ -39,7 +39,7 @@ Privileges Person chooses to make the bug into a question. There is a
 field for a comment. They decide to create the question using the
 'Convert this bug to a question' button.
 
-    >>> find_main_content(user_browser.contents).p
+    >>> print(find_main_content(user_browser.contents).p)
     <p>... the bug's status is set to Invalid. The new question
     will be linked to the bug. ...
 
@@ -53,7 +53,7 @@ informational message stating that a question was created from the bug.
     'Bug #10 ... : Bugs : linux-source-2.6.15 package : Ubuntu'
 
     >>> content = find_main_content(user_browser.contents)
-    >>> content.find(id="bug-is-question")
+    >>> print(content.find(id="bug-is-question"))
     <p...This bug report was converted into a question:
      question #...: <a ...>another test bug</a>. </p>
 
@@ -132,7 +132,7 @@ question.
     >>> print(user_browser.title)
     Convert this bug to a question...
 
-    >>> find_main_content(user_browser.contents).p
+    >>> print(find_main_content(user_browser.contents).p)
     <p>
     This bug cannot be converted into a question.
     Mozilla Thunderbird does not use Launchpad to track bugs.
@@ -161,7 +161,7 @@ a bookmark, they see that they cannot create the question again.
     >>> user_browser.title
     'Convert this bug to a question...
 
-    >>> find_main_content(user_browser.contents).p
+    >>> print(find_main_content(user_browser.contents).p)
     <p>
     This bug cannot be converted into a question.
     A question was already created from this bug. ...
@@ -222,7 +222,7 @@ Privileges Person chooses to reactivate the bug. There is an optional
 field for a comment. No other input is needed. No Privileges Person uses
 the 'Convert back to a bug' button.
 
-    >>> find_main_content(user_browser.contents).p
+    >>> print(find_main_content(user_browser.contents).p)
     <p>... Reactivate this bug report by removing the question created
     from the bug. ...
 
@@ -287,7 +287,7 @@ remove.
     >>> print(user_browser.title)
     Bug #12 - Convert this...
 
-    >>> find_main_content(user_browser.contents).p
+    >>> print(find_main_content(user_browser.contents).p)
     <p>
     The bug was not converted to a question. There is nothing to change. ...
 
diff --git a/lib/lp/bugs/stories/bugs/xx-bug-heat-on-bug-page.txt b/lib/lp/bugs/stories/bugs/xx-bug-heat-on-bug-page.txt
index 505c0b2..d42f14c 100644
--- a/lib/lp/bugs/stories/bugs/xx-bug-heat-on-bug-page.txt
+++ b/lib/lp/bugs/stories/bugs/xx-bug-heat-on-bug-page.txt
@@ -5,4 +5,4 @@ Bug heat appears on the bug index page:
     >>> anon_browser.open('http://bugs.launchpad.test/firefox/+bug/1')
     >>> content = find_main_content(anon_browser.contents)
     >>> print(content.find('a', href='/+help-bugs/bug-heat.html'))
-    <a href="/+help-bugs/bug-heat.html" target="help" class="sprite flame">0</a>
+    <a class="sprite flame" href="/+help-bugs/bug-heat.html" target="help">0</a>
diff --git a/lib/lp/bugs/stories/bugs/xx-bug-hidden-comments.txt b/lib/lp/bugs/stories/bugs/xx-bug-hidden-comments.txt
index b3313c0..b656eba 100644
--- a/lib/lp/bugs/stories/bugs/xx-bug-hidden-comments.txt
+++ b/lib/lp/bugs/stories/bugs/xx-bug-hidden-comments.txt
@@ -76,7 +76,7 @@ For admin users, the message is still visible in the bug page.
 Admin users will see the hidden message highlighted with an
 'adminHiddenComment' style.
 
-    >>> print(last_comment.parent['class'])
+    >>> print(' '.join(last_comment.parent['class']))
     boardComment adminHiddenComment
 
 Admin users can also reach the message via direct link, and it is
@@ -93,7 +93,7 @@ highlighted with the 'adminHiddenComment style there too.
     >>> main_content = find_main_content(admin_browser.contents)
     >>> last_comment = main_content('div', 'boardCommentBody')[-1]
     >>> last_comment_text = extract_text(last_comment.div)
-    >>> print(last_comment.parent['class'])
+    >>> print(' '.join(last_comment.parent['class']))
     boardComment adminHiddenComment
 
 Also for the owner of comment the message is still visible in the bug page.
@@ -110,7 +110,7 @@ Also for the owner of comment the message is still visible in the bug page.
 Owner of the comment will see the hidden message highlighted with an
 'adminHiddenComment' style.
 
-    >>> print(last_comment.parent['class'])
+    >>> print(' '.join(last_comment.parent['class']))
     boardComment adminHiddenComment
 
 Owner of the comment can also reach the message via direct link, and it is
@@ -127,5 +127,5 @@ highlighted with the 'adminHiddenComment style there too.
     >>> main_content = find_main_content(user_browser.contents)
     >>> last_comment = main_content('div', 'boardCommentBody')[-1]
     >>> last_comment_text = extract_text(last_comment.div)
-    >>> print(last_comment.parent['class'])
+    >>> print(' '.join(last_comment.parent['class']))
     boardComment adminHiddenComment
diff --git a/lib/lp/bugs/stories/bugs/xx-bug-index-lots-of-comments.txt b/lib/lp/bugs/stories/bugs/xx-bug-index-lots-of-comments.txt
index 47c5150..930294f 100644
--- a/lib/lp/bugs/stories/bugs/xx-bug-index-lots-of-comments.txt
+++ b/lib/lp/bugs/stories/bugs/xx-bug-index-lots-of-comments.txt
@@ -47,7 +47,7 @@ reply to the wrong message.
 
     >>> print(find_tag_by_id(
     ...     user_browser.contents, 'add-comment-form-container'))
-    <div id="add-comment-form-container" class="hidden">...
+    <div class="hidden" id="add-comment-form-container">...
     <div id="add-comment-form">...</div>...
     </div>
 
diff --git a/lib/lp/bugs/stories/bugs/xx-front-page-search.txt b/lib/lp/bugs/stories/bugs/xx-front-page-search.txt
index 418465c..612f0c2 100644
--- a/lib/lp/bugs/stories/bugs/xx-front-page-search.txt
+++ b/lib/lp/bugs/stories/bugs/xx-front-page-search.txt
@@ -103,7 +103,7 @@ in the Launchpad.
     'test bug'
     >>> for message in find_tags_by_class(anon_browser.contents, 'message'):
     ...     print(message.renderContents())
-    There is no project named &#x27;invalid&#x27; registered in Launchpad
+    There is no project named 'invalid' registered in Launchpad
 
 If the user doesn't know what name to write, they can use the 'Choose'
 link if the browser supports javascript. The test browser does not
diff --git a/lib/lp/bugs/stories/bugs/xx-incomplete-bugs.txt b/lib/lp/bugs/stories/bugs/xx-incomplete-bugs.txt
index 7cf10f5..7d1aade 100644
--- a/lib/lp/bugs/stories/bugs/xx-incomplete-bugs.txt
+++ b/lib/lp/bugs/stories/bugs/xx-incomplete-bugs.txt
@@ -26,7 +26,7 @@ was supplied (through comments) are 'Incomplete (without response)'.
     ...     ['INCOMPLETE_WITHOUT_RESPONSE'])
     >>> user_browser.getControl('Search', index=1).click()
     >>> find_tag_by_id(user_browser.contents, 'bugs-table-listing').findChild('a')
-    <a href="http://bugs.launchpad.test/jokosher/+bug/11"; class="bugtitle">...</a>
+    <a class="bugtitle" href="http://bugs.launchpad.test/jokosher/+bug/11";>...</a>
 
 Bugs that have been marked incomplete and for which new information was
 supplied are 'Incomplete (with response)'.
@@ -63,7 +63,7 @@ They try again to find that bug using the advanced search form.
     ...     ['INCOMPLETE_WITH_RESPONSE'])
     >>> user_browser.getControl('Search', index=1).click()
     >>> find_tag_by_id(user_browser.contents, 'bugs-table-listing').findChild('a')
-    <a href="http://bugs.launchpad.test/jokosher/+bug/11"; class="bugtitle">...</a>
+    <a class="bugtitle" href="http://bugs.launchpad.test/jokosher/+bug/11";>...</a>
 
 The bug is there, since they supplied new information in a comment. No
 Privileges Person makes sure that it no longer is in the list of
@@ -74,7 +74,7 @@ incomplete bugs without response.
     >>> user_browser.getControl(name='field.status:list').value = (
     ...     ['INCOMPLETE_WITH_RESPONSE'])
     >>> user_browser.getControl('Search', index=1).click()
-    >>> ('<a href="http://bugs.launchpad.test/jokosher/+bug/11"; class="bugtitle">' in
+    >>> ('<a class="bugtitle" href="http://bugs.launchpad.test/jokosher/+bug/11";>' in
     ...     find_tag_by_id(user_browser.contents, 'bugs-table-listing'))
     False
 
@@ -261,13 +261,16 @@ Since no new comments have been added after we changed the status to
 Incomplete, we can now find that bug searching for Incomplete (without
 response) bugs.
 
+    >>> import six
+
     >>> user_browser.open(
     ...     'http://bugs.launchpad.test/jokosher/+bugs?advanced=1')
     >>> user_browser.getControl(name='field.status:list').value = (
     ...     ['INCOMPLETE_WITHOUT_RESPONSE'])
     >>> user_browser.getControl('Search', index=1).click()
-    >>> ('<a href="http://bugs.launchpad.test/jokosher/+bug/11"; class="bugtitle">' in
-    ...     str(find_tag_by_id(user_browser.contents, 'bugs-table-listing')))
+    >>> ('<a class="bugtitle" href="http://bugs.launchpad.test/jokosher/+bug/11";>' in
+    ...     six.text_type(
+    ...         find_tag_by_id(user_browser.contents, 'bugs-table-listing')))
     True
 
 A default search turns that bug up as well.
@@ -276,6 +279,7 @@ A default search turns that bug up as well.
     >>> user_browser.getControl('Search', index=0).click()
     >>> print(user_browser.url)
     http://bugs.launchpad.test/jokosher/+bugs?...&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE...
-    >>> ('<a href="http://bugs.launchpad.test/jokosher/+bug/11"; class="bugtitle">' in
-    ...     str(find_tag_by_id(user_browser.contents, 'bugs-table-listing')))
+    >>> ('<a class="bugtitle" href="http://bugs.launchpad.test/jokosher/+bug/11";>' in
+    ...     six.text_type(
+    ...         find_tag_by_id(user_browser.contents, 'bugs-table-listing')))
     True
diff --git a/lib/lp/bugs/stories/bugs/xx-numbered-comments.txt b/lib/lp/bugs/stories/bugs/xx-numbered-comments.txt
index 3aa0ff4..60eba31 100644
--- a/lib/lp/bugs/stories/bugs/xx-numbered-comments.txt
+++ b/lib/lp/bugs/stories/bugs/xx-numbered-comments.txt
@@ -12,7 +12,7 @@ easily, the numbers are displayed in the comment header.
     >>> for comment in comments:
     ...     number_node = comment.find(None, 'bug-comment-index')
     ...     person_node = comment.find(
-    ...         lambda node: 'person' in node.get('class', ''))
+    ...         lambda node: 'person' in ' '.join(node.get('class', [])))
     ...     comment_node = comment.find(None, 'comment-text')
     ...     print("%s: %s\n  %s" % (
     ...         extract_text(number_node),
@@ -21,7 +21,7 @@ easily, the numbers are displayed in the comment header.
     #1: Valentina Commissari (tsukimi)
       The solution to this is to make Jokosher use autoa
     #2: Diogo Matsubara (matsubara)
-      I&#x27;m not sure that autoaudiosink is in fact th
+      I'm not sure that autoaudiosink is in fact the bes
     #3: Karl Tilbury (karl)
       Unfortunately, the lead developer of autoaudiosink
     #4: Daniel Henrique Debonzi (debonzi)
diff --git a/lib/lp/bugs/stories/bugtask-management/xx-bugtask-edit-forms.txt b/lib/lp/bugs/stories/bugtask-management/xx-bugtask-edit-forms.txt
index 20a2981..8ba4163 100644
--- a/lib/lp/bugs/stories/bugtask-management/xx-bugtask-edit-forms.txt
+++ b/lib/lp/bugs/stories/bugtask-management/xx-bugtask-edit-forms.txt
@@ -48,7 +48,7 @@ respective bug task.
     Target
     Distribution
     ...
-    Project (Find&hellip;)
+    Project (Find…)
     Status  Importance  Milestone
     New...  Low...      (nothing selected)...
     Assigned to... Mark Shuttleworth (mark)
diff --git a/lib/lp/bugs/stories/bugtask-management/xx-change-assignee.txt b/lib/lp/bugs/stories/bugtask-management/xx-change-assignee.txt
index 325d595..2e3990f 100644
--- a/lib/lp/bugs/stories/bugtask-management/xx-change-assignee.txt
+++ b/lib/lp/bugs/stories/bugtask-management/xx-change-assignee.txt
@@ -153,5 +153,5 @@ But if they try to set other persons or teams, they get an error message.
     >>> user_browser.getControl("Save Changes", index=0).click()
     >>> print_errors(user_browser.contents)
     There is 1 error in the data you entered. Please fix it and try again.
-    (Find&hellip;)
+    (Find…)
     Constraint not satisfied
diff --git a/lib/lp/bugs/stories/bugtask-searches/xx-advanced-people-filters.txt b/lib/lp/bugs/stories/bugtask-searches/xx-advanced-people-filters.txt
index f285d7c..ef280d3 100644
--- a/lib/lp/bugs/stories/bugtask-searches/xx-advanced-people-filters.txt
+++ b/lib/lp/bugs/stories/bugtask-searches/xx-advanced-people-filters.txt
@@ -57,7 +57,7 @@ If we enter an invalid assignee, we'll get a nice error message.
     ...     'invalid-assignee')
     >>> anon_browser.getControl('Search', index=0).click()
     >>> print_feedback_messages(anon_browser.contents)
-    There&#x27;s no person with the name or email address &#x27;invalid-assignee&#x27;.
+    There's no person with the name or email address 'invalid-assignee'.
 
     >>> anon_browser.open(
     ...     'http://bugs.launchpad.test/~name12/+reportedbugs?advanced=1')
@@ -65,7 +65,7 @@ If we enter an invalid assignee, we'll get a nice error message.
     ...     'invalid-assignee')
     >>> anon_browser.getControl('Search', index=0).click()
     >>> print_feedback_messages(anon_browser.contents)
-    There&#x27;s no person with the name or email address &#x27;invalid-assignee&#x27;.
+    There's no person with the name or email address 'invalid-assignee'.
 
 
 Searching by reporter
@@ -94,15 +94,14 @@ and invalid searches don't OOPS:
     ...     'invalid-reporter')
     >>> anon_browser.getControl('Search', index=0).click()
     >>> print_feedback_messages(anon_browser.contents)
-    There&#x27;s no person with the name or email address &#x27;invalid-reporter&#x27;.
+    There's no person with the name or email address 'invalid-reporter'.
 
     >>> anon_browser.open('http://bugs.launchpad.test/~name12/+assignedbugs')
     >>> anon_browser.getControl(name='field.bug_reporter').value = (
     ...     'invalid-reporter')
     >>> anon_browser.getControl('Search', index=0).click()
     >>> print_feedback_messages(anon_browser.contents)
-    There&#x27;s no person with the name or email address
-    &#x27;invalid-reporter&#x27;.
+    There's no person with the name or email address 'invalid-reporter'.
 
 
 Searching for a bug commenter's bugs
@@ -121,7 +120,7 @@ displayed.
     >>> anon_browser.getControl('Search', index=0).click()
     >>> for message in find_tags_by_class(anon_browser.contents, 'message'):
     ...     print(message.renderContents())
-    There&#x27;s no person with the name or email address &#x27;non-existent&#x27;.
+    There's no person with the name or email address 'non-existent'.
 
 Entering an existing person shows all bugs that person has commented on
 or made metadata changes to.
@@ -183,7 +182,7 @@ displayed:
     >>> anon_browser.getControl('Search', index=0).click()
     >>> for message in find_tags_by_class(anon_browser.contents, 'message'):
     ...     print(message.renderContents())
-    There&#x27;s no person with the name or email address &#x27;non-existent&#x27;.
+    There's no person with the name or email address 'non-existent'.
 
 Entering an existing person shows all bugs for packages or products that
 the person is subscribed to. To demonstrate, we'll begin with a user who
diff --git a/lib/lp/bugs/stories/bugtask-searches/xx-listing-basics.txt b/lib/lp/bugs/stories/bugtask-searches/xx-listing-basics.txt
index c4e68ad..38dacdb 100644
--- a/lib/lp/bugs/stories/bugtask-searches/xx-listing-basics.txt
+++ b/lib/lp/bugs/stories/bugtask-searches/xx-listing-basics.txt
@@ -226,8 +226,8 @@ relevant listings:
     ...     'http://bugs.launchpad.test/debian/sarge/+source/mozilla-firefox')
     >>> milestone = find_tags_by_class(browser.contents, 'sprite milestone')
     >>> print(milestone[0])
-    <a href="http://launchpad.test/debian/+milestone/3.1"; alt="milestone 3.1"
-    title="Linked to milestone 3.1" class="sprite milestone"></a>
+    <a alt="milestone 3.1" class="sprite milestone"
+    href="http://launchpad.test/debian/+milestone/3.1"; title="Linked to milestone 3.1"></a>
 
 
 Patches also appear as badges in bug listings.
diff --git a/lib/lp/bugs/stories/bugtracker/bugtrackers-index.txt b/lib/lp/bugs/stories/bugtracker/bugtrackers-index.txt
index 19b25f9..74bc2e2 100644
--- a/lib/lp/bugs/stories/bugtracker/bugtrackers-index.txt
+++ b/lib/lp/bugs/stories/bugtracker/bugtrackers-index.txt
@@ -28,21 +28,21 @@ The page presents a table with all bugtrackers currently registered:
     Debian Bug tracker
     http://bugs.debian.org
       --> http://bugs.debian.org
-    &mdash;
+    —
     Debbugs
     5
     ------------------------
     Email bugtracker
     mailto:bugs@xxxxxxxxxxx
       --> mailto:bugs@xxxxxxxxxxx
-    &mdash;
+    —
     Email Address
     0
     ------------------------
-    T&#x27;other Gnome GBugGTracker
+    T'other Gnome GBugGTracker
     http://bugzilla.gnome.org/
       --> http://bugzilla.gnome.org/
-    &mdash;
+    —
     Bugzilla
     0
     ------------------------
@@ -65,7 +65,7 @@ auto-created ones - so the title is also obfuscated.
     &lt;email address hidden&gt; bug tracker
     mailto:&lt;email address hidden&gt;
       --> None
-    &mdash;
+    —
     Email Address
     0
     ------------------------
@@ -77,7 +77,7 @@ The watch counts match the number of bugs listed, of course:
     >>> nav = find_tags_by_class(user_browser.contents,
     ...     'batch-navigation-index')
     >>> print(extract_text(nav[0]))
-    1 &rarr; 5 of 5 results
+    1 → 5 of 5 results
 
 The listing also displays projects and projects linked to bug trackers.
 Let's link a pair to debbugs:
@@ -123,7 +123,7 @@ linked to bugtrackers are also linked.
     Debian Bug tracker
     http://bugs.debian.org
       --> http://bugs.debian.org
-    a52dec, Derby, iso-codes &hellip;
+    a52dec, Derby, iso-codes …
     Debbugs
     5
     ------------------------
diff --git a/lib/lp/bugs/stories/bugtracker/xx-bugtracker.txt b/lib/lp/bugs/stories/bugtracker/xx-bugtracker.txt
index 915c240..a3d38d8 100644
--- a/lib/lp/bugs/stories/bugtracker/xx-bugtracker.txt
+++ b/lib/lp/bugs/stories/bugtracker/xx-bugtracker.txt
@@ -80,7 +80,7 @@ is informed about it.
     ...     print(extract_text(message))
     There is 1 error.
     http://bugzilla.mozilla.org/ is already registered in Launchpad
-    as &quot;The Mozilla.org Bug Tracker&quot; (mozilla.org).
+    as "The Mozilla.org Bug Tracker" (mozilla.org).
 
 The same happens if the requested URL is aliased to another bug tracker.
 Aliases can be edited once a bug tracker has been added, but for now
@@ -107,7 +107,7 @@ we'll dig directly to the database.
     ...     print(extract_text(message))
     There is 1 error.
     http://alias.example.com/ is already registered in Launchpad
-    as &quot;GnomeGBug GTracker&quot; (gnome-bugzilla).
+    as "GnomeGBug GTracker" (gnome-bugzilla).
 
 After successfully registering the bug tracker, the user is redirected
 to the bug tracker page.
@@ -222,7 +222,7 @@ tracker uses.
     >>> print_feedback_messages(user_browser.contents)
     There is 1 error.
     http://bugzilla.mozilla.org/ is already registered in Launchpad
-    as &quot;The Mozilla.org Bug Tracker&quot; (mozilla.org).
+    as "The Mozilla.org Bug Tracker" (mozilla.org).
 
 If the user inadvertently enters an invalid URL, they are shown an
 informative error message explaining why it is invalid.
@@ -233,7 +233,7 @@ informative error message explaining why it is invalid.
 
     >>> print_feedback_messages(user_browser.contents)
     There is 1 error.
-    &quot;what? my wife does this stuff&quot; is not a valid URI
+    "what? my wife does this stuff" is not a valid URI
 
     >>> user_browser.getControl('Location', index=0).value = (
     ...     b'http://ξνεr.been.fishing?')
@@ -325,7 +325,7 @@ bugtracker.
     >>> print_feedback_messages(user_browser.contents)
     There is 1 error.
     http://bugzilla.mozilla.org/ is already registered in Launchpad
-    as &quot;The Mozilla.org Bug Tracker&quot; (mozilla.org).
+    as "The Mozilla.org Bug Tracker" (mozilla.org).
 
 Multiple aliases can be entered by separating URLs with whitespace.
 
@@ -357,7 +357,7 @@ shown informative error messages.
     >>> print_feedback_messages(user_browser.contents)
     There is 1 error.
     URIs must consist of ASCII characters
-    &quot;been&quot; is not a valid URI
+    "been" is not a valid URI
 
 
 Deleting a bug tracker
@@ -387,7 +387,7 @@ deletion yet:
     'http://bugs.launchpad.test/bugs/bugtrackers'
 
     >>> print_feedback_messages(user_browser.contents)
-    Freddy&#x27;s Bugs has been deleted.
+    Freddy's Bugs has been deleted.
 
 Bug trackers can be deleted by anyone, subject to a few restrictions:
 
@@ -678,14 +678,14 @@ Then we can see how logged-in users and anonymous users see the page:
     >>> print_watches(user_browser)
     #12:
     Copy, Cut and Delete operations should work on selections
-      --> &mdash;: mailto:bugs@xxxxxxxxxxx
+      --> —: mailto:bugs@xxxxxxxxxxx
 
     >>> anon_browser.open(
     ...     'http://launchpad.test/bugs/bugtrackers/email')
     >>> print_watches(anon_browser)
     #12:
     Copy, Cut and Delete operations should work on selections
-      --> &mdash;: None
+      --> —: None
 
 Info portlet
 ------------
diff --git a/lib/lp/bugs/stories/bugwatches/xx-bugwatch-comments.txt b/lib/lp/bugs/stories/bugwatches/xx-bugwatch-comments.txt
index 972be1f..b1e9221 100644
--- a/lib/lp/bugs/stories/bugwatches/xx-bugwatch-comments.txt
+++ b/lib/lp/bugs/stories/bugwatches/xx-bugwatch-comments.txt
@@ -18,7 +18,7 @@ These comments can be displayed like any other comments on a bug. Bug
     <p>Hi!...Usually CD-ROMs are handled in /etc/fstab, so this might
     not even be a...pmount bug...
     ----------------------------------------
-    <p>I&#x27;ll be happy to add the info you request to the bug report
+    <p>I'll be happy to add the info you request to the bug report
     if it will...
     ----------------------------------------
 
@@ -43,6 +43,6 @@ The ordinary user can't see these comments:
     <p>Hi!...Usually CD-ROMs are handled in /etc/fstab, so this might
     not even be a...pmount bug...
     ----------------------------------------
-    <p>I&#x27;ll be happy to add the info you request to the bug report
+    <p>I'll be happy to add the info you request to the bug report
     if it will...
     ----------------------------------------
diff --git a/lib/lp/bugs/stories/bugwatches/xx-edit-bugwatch.txt b/lib/lp/bugs/stories/bugwatches/xx-edit-bugwatch.txt
index fa54876..6e9e46e 100644
--- a/lib/lp/bugs/stories/bugwatches/xx-edit-bugwatch.txt
+++ b/lib/lp/bugs/stories/bugwatches/xx-edit-bugwatch.txt
@@ -37,7 +37,7 @@ Likewise, stupid URLs are rejected with a polite error message.
     >>> for message in find_tags_by_class(admin_browser.contents, 'message'):
     ...     print(extract_text(message))
     There is 1 error.
-    &quot;GELLER&quot; is not a valid URI
+    "GELLER" is not a valid URI
 
 
 BugWatch details
diff --git a/lib/lp/bugs/stories/cve/cve-linking.txt b/lib/lp/bugs/stories/cve/cve-linking.txt
index 95974dc..a2bc6f0 100644
--- a/lib/lp/bugs/stories/cve/cve-linking.txt
+++ b/lib/lp/bugs/stories/cve/cve-linking.txt
@@ -37,7 +37,7 @@ title is linked
     Bug #5:...Firefox install instructions should be complete ...
 
     >>> print(content.a)
-    <a href=".../bugs/5" class="sprite bug">Bug #5: ...
+    <a class="sprite bug" href=".../bugs/5">Bug #5: ...
 
 It is also possible to link a bug using its nickname. For example, bug
 #2 has 'blackhole' as its nickname:
diff --git a/lib/lp/bugs/stories/cve/cve-pages.txt b/lib/lp/bugs/stories/cve/cve-pages.txt
index 86d09b9..75819c2 100644
--- a/lib/lp/bugs/stories/cve/cve-pages.txt
+++ b/lib/lp/bugs/stories/cve/cve-pages.txt
@@ -82,7 +82,7 @@ The CVE page links to the related bugs.
     >>> for tag in find_tags_by_class(
     ...     anon_browser.contents, 'menu-link-linkbug'):
     ...     print(tag)
-    <a href="+linkbug" class="menu-link-linkbug sprite add">Link to bug</a>
+    <a class="menu-link-linkbug sprite add" href="+linkbug">Link to bug</a>
     >>> 'Candidate' in anon_browser.contents
     True
     >>> '20050826 Multiple PHP' in anon_browser.contents
diff --git a/lib/lp/bugs/stories/cve/xx-cve-link-xss.txt b/lib/lp/bugs/stories/cve/xx-cve-link-xss.txt
index 799a8b0..96b75a0 100644
--- a/lib/lp/bugs/stories/cve/xx-cve-link-xss.txt
+++ b/lib/lp/bugs/stories/cve/xx-cve-link-xss.txt
@@ -21,7 +21,7 @@ Indeed, the markup is valid and correctly escaped:
     ...     user_browser.contents, 'field.sequence').prettify())
     <input class="textType" id="field.sequence"
            name="field.sequence" size="20" type="text"
-           value='&lt;script&gt;alert("cheezburger");&lt;/script&gt;' />
+           value='&lt;script&gt;alert("cheezburger");&lt;/script&gt;'/>
 
 The error message is also valid and correctly escaped:
 
@@ -32,7 +32,7 @@ The error message is also valid and correctly escaped:
     </p>
     <BLANKLINE>
     <div class="message">
-    &lt;script&gt;alert(&quot;cheezburger&quot;);&lt;/script&gt;
+    &lt;script&gt;alert("cheezburger");&lt;/script&gt;
       is not a valid CVE number
     </div>
 
diff --git a/lib/lp/bugs/stories/guided-filebug/xx-bug-reporting-guidelines.txt b/lib/lp/bugs/stories/guided-filebug/xx-bug-reporting-guidelines.txt
index 03484d4..96782f6 100644
--- a/lib/lp/bugs/stories/guided-filebug/xx-bug-reporting-guidelines.txt
+++ b/lib/lp/bugs/stories/guided-filebug/xx-bug-reporting-guidelines.txt
@@ -80,31 +80,31 @@ description.
     Ubuntu
       <http://bugs.launchpad.test/ubuntu/+filebug>
     Ubuntu bug reporting guidelines:
-    The version of Ubuntu you&#x27;re using.
+    The version of Ubuntu you're using.
     See http://example.com for more details.
     Thank you for filing a bug for https://launchpad.test/ubuntu
     *
     Mozilla
       <http://.../firefox/+filebug?field.title=It+doesn%27t+work&field.tags=>
     Mozilla Firefox bug reporting guidelines:
-    The version of Firefox you&#x27;re using.
+    The version of Firefox you're using.
     See http://example.com for more details.
     Thank you for filing a bug for https://launchpad.test/firefox
     *
     Firefox
       <http://bugs.launchpad.test/firefox/+filebug>
     Mozilla Firefox bug reporting guidelines:
-    The version of Firefox you&#x27;re using.
+    The version of Firefox you're using.
     See http://example.com for more details.
     Thank you for filing a bug for https://launchpad.test/firefox
     *
     alsa-utils in Ubuntu
       <http://bugs.launchpad.test/ubuntu/+source/alsa-utils/+filebug>
     alsa-utils (Ubuntu) bug reporting guidelines:
-    The version of alsa-utils in Ubuntu you&#x27;re using.
+    The version of alsa-utils in Ubuntu you're using.
     See http://example.com for more details.
     Ubuntu bug reporting guidelines:
-    The version of Ubuntu you&#x27;re using.
+    The version of Ubuntu you're using.
     See http://example.com for more details.
     Thank you for filing a bug for
     https://launchpad.test/ubuntu/+source/alsa-utils
@@ -129,7 +129,7 @@ which the guidelines are taken from the respective distribution.
     >>> print(extract_text(find_tag_by_id(
     ...     user_browser.contents, 'bug-reporting-guidelines')))
     Ubuntu bug reporting guidelines:
-    The version of Ubuntu you&#x27;re using.
+    The version of Ubuntu you're using.
     See http://example.com for more details.
 
 Any URLS in the guidelines will be linkified, with the target attribute
@@ -167,7 +167,7 @@ shown.
     >>> print(extract_text(find_tag_by_id(
     ...     user_browser.contents, 'bug-reporting-guidelines')))
     Ubuntu bug reporting guidelines:
-    The version of Ubuntu you&#x27;re using.
+    The version of Ubuntu you're using.
     See http://example.com for more details.
 
 Changing the package to alsa-utils does not make the alsa-utils
@@ -180,7 +180,7 @@ guidelines appear.
     >>> print(extract_text(find_tag_by_id(
     ...     user_browser.contents, 'bug-reporting-guidelines')))
     Ubuntu bug reporting guidelines:
-    The version of Ubuntu you&#x27;re using.
+    The version of Ubuntu you're using.
     See http://example.com for more details.
 
 XXX: allenap 2008-11-14 bug=297743: These limitations have been filed
diff --git a/lib/lp/bugs/stories/guided-filebug/xx-bug-reporting-tools.txt b/lib/lp/bugs/stories/guided-filebug/xx-bug-reporting-tools.txt
index 6d3d2a1..7d94d6f 100644
--- a/lib/lp/bugs/stories/guided-filebug/xx-bug-reporting-tools.txt
+++ b/lib/lp/bugs/stories/guided-filebug/xx-bug-reporting-tools.txt
@@ -30,7 +30,7 @@ to give the data to the +filebug page.
 
     >>> for message in find_tags_by_class(anon_browser.contents, 'message'):
     ...     print(message.renderContents())
-    Your ticket is &quot;...&quot;
+    Your ticket is "..."
 
 To avoid having the tool from parsing the HTML page, the token is
 returned as a X-Launchpad-Blob-Token header in the response as well:
diff --git a/lib/lp/bugs/stories/guided-filebug/xx-displaying-similar-bugs.txt b/lib/lp/bugs/stories/guided-filebug/xx-displaying-similar-bugs.txt
index 4b30307..bd092de 100644
--- a/lib/lp/bugs/stories/guided-filebug/xx-displaying-similar-bugs.txt
+++ b/lib/lp/bugs/stories/guided-filebug/xx-displaying-similar-bugs.txt
@@ -11,10 +11,10 @@ duplicates.
     ...     bugs_list = find_tags_by_class(content, 'similar-bug')
     ...     for node in bugs_list:
     ...         label = node.findAll('label')[0]
-    ...         label_class = label['class']
+    ...         label_class = ' '.join(label['class'])
     ...         text_lines = [line.strip() for line in
     ...                       extract_text(node).splitlines()]
-    ...         summary = ' '.join(text_lines[:2]).replace('&#8203;','')
+    ...         summary = ' '.join(text_lines[:2]).replace(u'\u200B', u'')
     ...         status = ' '.join(text_lines[2:])
     ...         # All this trouble is worth it when you see ndiff output
     ...         # from a failing test, and it *makes sense* :)
diff --git a/lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt b/lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt
index 5214136..04ec80b 100644
--- a/lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt
+++ b/lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt
@@ -37,7 +37,7 @@ was attached.
 
     >>> print_feedback_messages(user_browser.contents)
     Thank you for your bug report.
-    The file &quot;example.txt&quot; was attached to the bug report.
+    The file "example.txt" was attached to the bug report.
 
 No Privileges Person can see the attachment in the attachments portlet.
 
diff --git a/lib/lp/bugs/stories/guided-filebug/xx-product-guided-filebug.txt b/lib/lp/bugs/stories/guided-filebug/xx-product-guided-filebug.txt
index baa4028..2ef98dc 100644
--- a/lib/lp/bugs/stories/guided-filebug/xx-product-guided-filebug.txt
+++ b/lib/lp/bugs/stories/guided-filebug/xx-product-guided-filebug.txt
@@ -22,7 +22,8 @@ If no title is entered, the user is asked to supply one.
     >>> for message in top_portlet.findAll(attrs={'class': 'error message'}):
     ...     print(message.renderContents())
     There is 1 error.
-    >>> for message in top_portlet.findAll(attrs={'class': 'message'}):
+    >>> for message in top_portlet.findAll(
+    ...         lambda node: node.attrs.get('class') == ['message']):
     ...     print(message.renderContents())
     Required input is missing.
 
diff --git a/lib/lp/bugs/stories/guided-filebug/xx-project-guided-filebug.txt b/lib/lp/bugs/stories/guided-filebug/xx-project-guided-filebug.txt
index c2aec1d..0c8760d 100644
--- a/lib/lp/bugs/stories/guided-filebug/xx-project-guided-filebug.txt
+++ b/lib/lp/bugs/stories/guided-filebug/xx-project-guided-filebug.txt
@@ -28,8 +28,8 @@ product's +filebug page to search for duplicates.
     'http://bugs...?field.title=Evolution+crashes&field.tags='
     >>> print(find_main_content(user_browser.contents).renderContents())
     <...
-    <input type="submit" id="field.actions.search"
-    name="field.actions.search" value="Continue" class="button" /> ...
+    <input class="button" id="field.actions.search"
+    name="field.actions.search" type="submit" value="Continue"/> ...
 
 Entering a description and submitting the bug takes the user to the bug
 page.
diff --git a/lib/lp/bugs/stories/structural-subscriptions/xx-bug-subscriptions.txt b/lib/lp/bugs/stories/structural-subscriptions/xx-bug-subscriptions.txt
index 9be1212..0c064d2 100644
--- a/lib/lp/bugs/stories/structural-subscriptions/xx-bug-subscriptions.txt
+++ b/lib/lp/bugs/stories/structural-subscriptions/xx-bug-subscriptions.txt
@@ -106,7 +106,7 @@ No Privileges Person is now subscribed...
     >>> for message in find_tags_by_class(browser.contents, 'message'):
     ...     print(message.renderContents())
     No Privileges Person will now receive an email each time someone reports
-    or changes a public bug in &quot;mozilla-firefox in Ubuntu&quot;.
+    or changes a public bug in "mozilla-firefox in Ubuntu".
 
     >>> browser.open(
     ... 'http://bugs.launchpad.test/ubuntu/+source/mozilla-firefox/+subscribe')
@@ -129,7 +129,7 @@ No Privileges Person is now subscribed...
     >>> print(find_tags_by_class(
     ...    browser.contents, 'informational message')[0].contents[0])
     No Privileges Person will no longer automatically receive email about
-    public bugs in &quot;mozilla-firefox in Ubuntu&quot;.
+    public bugs in "mozilla-firefox in Ubuntu".
     >>> browser.open(
     ... 'http://bugs.launchpad.test/ubuntu/+source/mozilla-firefox/+subscribe')
     >>> print(extract_text(find_portlet(browser.contents, 'Subscribers')))
diff --git a/lib/lp/bugs/tests/bug.py b/lib/lp/bugs/tests/bug.py
index 0e17bee..dfaff9b 100644
--- a/lib/lp/bugs/tests/bug.py
+++ b/lib/lp/bugs/tests/bug.py
@@ -92,7 +92,7 @@ def print_bug_affects_table(content, highlighted_only=False):
             # Don't print the bugtask edit form.
             continue
         # Strip zero-width white-spaces.
-        print extract_text(tr).replace('&#8203;', '')
+        print extract_text(tr).replace(u'\u200B', u'')
 
 
 def print_remote_bugtasks(content):
@@ -102,7 +102,7 @@ def print_remote_bugtasks(content):
     """
     affects_table = find_tags_by_class(content, 'listing')[0]
     for span in affects_table.findAll('span'):
-        for key, value in span.attrs:
+        for key, value in span.attrs.items():
             if 'bug-remote' in value:
                 target = extract_text(span.findAllPrevious('td')[-2])
                 print target, extract_text(span.findNext('a'))
@@ -120,7 +120,7 @@ def print_bugs_list(content, list_id):
         None, {'class': 'similar-bug'})
     for node in bugs_list:
         # Also strip zero-width spaces out.
-        print extract_text(node).replace('&#8203;', '')
+        print extract_text(node).replace(u'\u200B', u'')
 
 
 def print_bugtasks(text, show_heat=None):
@@ -341,4 +341,4 @@ def print_bug_tag_anchors(anchors):
     for anchor in anchors:
         href = anchor['href']
         if href != '+edit' and '/+help-bugs/tag-help.html' not in href:
-            print anchor['class'], anchor.contents[0]
+            print ' '.join(anchor['class']), anchor.contents[0]
diff --git a/lib/lp/buildmaster/stories/xx-builder-page.txt b/lib/lp/buildmaster/stories/xx-builder-page.txt
index 2eafcbd..c27de04 100644
--- a/lib/lp/buildmaster/stories/xx-builder-page.txt
+++ b/lib/lp/buildmaster/stories/xx-builder-page.txt
@@ -203,7 +203,7 @@ He can see now, in the details portlet that the builder is in manual-mode.
 And a relevant notification is displayed after the mode toggle.
 
     >>> print_feedback_messages(cprov_browser.contents)
-    The builder &quot;Bob The Builder&quot; was updated successfully.
+    The builder "Bob The Builder" was updated successfully.
 
 Via the 'edit' form Celso can also modify the 'builderok',
 'failure_notes', 'virtualized' and 'virtual machine' fields. All the
@@ -222,7 +222,7 @@ Changing the details via the Change details page also generates a
 notification.
 
     >>> print_feedback_messages(cprov_browser.contents)
-    The builder &quot;Bob The Builder&quot; was updated successfully.
+    The builder "Bob The Builder" was updated successfully.
 
 
 Marking a builder as inactive
diff --git a/lib/lp/code/browser/tests/test_branchmergeproposal.py b/lib/lp/code/browser/tests/test_branchmergeproposal.py
index 177b396..8180bc0 100644
--- a/lib/lp/code/browser/tests/test_branchmergeproposal.py
+++ b/lib/lp/code/browser/tests/test_branchmergeproposal.py
@@ -193,7 +193,7 @@ class TestBranchMergeProposalMergedViewMixin:
             self.arbitrary_revisions[2])
         browser.getControl('Mark as Merged').click()
         self.assertEqual(
-            ["The proposal&#x27;s merged revision has been updated."],
+            ["The proposal's merged revision has been updated."],
             get_feedback_messages(browser.contents))
         self.assertIn(
             'Status:\nMerged\nMerged at revision:\n%s' % (
diff --git a/lib/lp/code/browser/tests/test_sourcepackagerecipe.py b/lib/lp/code/browser/tests/test_sourcepackagerecipe.py
index c96ac22..97c7a1a 100644
--- a/lib/lp/code/browser/tests/test_sourcepackagerecipe.py
+++ b/lib/lp/code/browser/tests/test_sourcepackagerecipe.py
@@ -52,7 +52,6 @@ from lp.services.beautifulsoup import BeautifulSoup
 from lp.services.database.constants import UTC_NOW
 from lp.services.propertycache import clear_property_cache
 from lp.services.webapp import canonical_url
-from lp.services.webapp.escaping import html_escape
 from lp.services.webapp.interfaces import ILaunchpadRoot
 from lp.services.webapp.servers import LaunchpadTestRequest
 from lp.testing import (
@@ -506,7 +505,7 @@ class TestSourcePackageRecipeAddViewMixin:
         browser.getControl('Create Recipe').click()
         self.assertEqual(
             get_feedback_messages(browser.contents)[1],
-            html_escape('The recipe instruction "run" is not permitted here.'))
+            'The recipe instruction "run" is not permitted here.')
 
     def createRecipe(self, recipe_text, branch=None):
         if branch is None:
@@ -730,7 +729,7 @@ class TestSourcePackageRecipeAddViewMixin:
         browser.getControl('Create Recipe').click()
         self.assertEqual(
             get_feedback_messages(browser.contents)[1],
-            html_escape("You already have a PPA for Ubuntu named 'foo'."))
+            "You already have a PPA for Ubuntu named 'foo'.")
 
     def test_create_new_ppa_missing_name(self):
         # If a new PPA is being created, and the user has not specified a
@@ -990,7 +989,7 @@ class TestSourcePackageRecipeEditViewMixin:
 
         self.assertEqual(
             get_feedback_messages(browser.contents)[1],
-            html_escape('The recipe instruction "run" is not permitted here.'))
+            'The recipe instruction "run" is not permitted here.')
 
     def test_edit_recipe_format_too_new(self):
         # If the recipe's format version is too new, we should notify the
diff --git a/lib/lp/code/stories/branches/xx-bazaar-home.txt b/lib/lp/code/stories/branches/xx-bazaar-home.txt
index 5b90cca..f2a5868 100644
--- a/lib/lp/code/stories/branches/xx-bazaar-home.txt
+++ b/lib/lp/code/stories/branches/xx-bazaar-home.txt
@@ -32,7 +32,7 @@ with a link to the complete listing.
     >>> preview = find_tag_by_id(browser.contents, 'project-cloud-preview')
     >>> print(extract_text(preview))
     Most active projects in the last month
-    see all projects&#8230;
+    see all projects…
 
     >>> print(preview.findAll('a')[-1]['href'])
     /projects
@@ -93,8 +93,8 @@ registered is the ordering, the registered date is also shown.
     Last Modified
     Last Commit
     >>> links = find_tag_by_id(browser.contents, 'branch-batch-links')
-    >>> links.renderContents()
-    '...1...&rarr;...6...of...28...results...'
+    >>> print(links.decode_contents())
+    <...1...→...6...of...28...results...
 
 
 Recently changed
diff --git a/lib/lp/code/stories/branches/xx-branch-index.txt b/lib/lp/code/stories/branches/xx-branch-index.txt
index d745bec..9801d52 100644
--- a/lib/lp/code/stories/branches/xx-branch-index.txt
+++ b/lib/lp/code/stories/branches/xx-branch-index.txt
@@ -235,11 +235,10 @@ it has been mirrored:
 The branch description should not be shown if there is none.
 
     >>> def get_branch_description(browser):
-    ...     return extract_text(find_tag_by_id(
-    ...         browser.contents, 'branch-description'))
+    ...     tag = find_tag_by_id(browser.contents, 'branch-description')
+    ...     return extract_text(tag) if tag is not None else None
     >>> print(get_branch_description(browser))
-    Traceback (most recent call last):
-    TypeError: expected string or buffer
+    None
 
 Branches that have never been mirrored don't have a 'Last mirrored'
 field.
diff --git a/lib/lp/code/stories/branches/xx-branch-listings.txt b/lib/lp/code/stories/branches/xx-branch-listings.txt
index 431aa01..fed83f7 100644
--- a/lib/lp/code/stories/branches/xx-branch-listings.txt
+++ b/lib/lp/code/stories/branches/xx-branch-listings.txt
@@ -25,7 +25,7 @@ branches in the listings.
     >>> links = find_tag_by_id(browser.contents, 'branch-batch-links')
     >>> print(links.renderContents())
     <BLANKLINE>
-    ...1...&rarr;...6...of 10 results...
+    ...1...→...6...of 10 results...
 
     >>> table = find_tag_by_id(browser.contents, 'branchtable')
     >>> for row in table.thead.findAll('tr'):
@@ -52,7 +52,7 @@ and are really just branch metadata without the revisions behind them.
     >>> links = find_tag_by_id(browser.contents, 'branch-batch-links')
     >>> print(links.renderContents())
     <BLANKLINE>
-    ...7...&rarr;...10...of 10 results...
+    ...7...→...10...of 10 results...
 
     >>> table = find_tag_by_id(browser.contents, 'branchtable')
     >>> for row in table.tbody.findAll('tr'):
@@ -114,7 +114,7 @@ Now all types of branches should be shown.
     >>> links = find_tag_by_id(browser.contents, 'branch-batch-links')
     >>> print(links.renderContents())
     <BLANKLINE>
-    ...1...&rarr;...6...of 12 results...
+    ...1...→...6...of 12 results...
 
     >>> table = find_tag_by_id(browser.contents, 'branchtable')
     >>> for row in table.tbody.findAll('tr'):
@@ -130,7 +130,7 @@ Now all types of branches should be shown.
     >>> links = find_tag_by_id(browser.contents, 'branch-batch-links')
     >>> print(links.renderContents())
     <BLANKLINE>
-    ...7...&rarr;...12...of 12 results...
+    ...7...→...12...of 12 results...
 
     >>> table = find_tag_by_id(browser.contents, 'branchtable')
     >>> for row in table.tbody.findAll('tr'):
@@ -213,15 +213,17 @@ We display badges for associated bugs.
     >>> def branchSummary(browser):
     ...     table = find_tag_by_id(browser.contents, 'branchtable')
     ...     for row in table.tbody.findAll('tr'):
-    ...         if row.getText().startswith('A development focus branch'):
+    ...         if row.get_text(strip=True).startswith(
+    ...                 'A development focus branch'):
     ...             continue
     ...         cells = row.findAll('td')
     ...         first_cell = cells[0]
     ...         anchors = first_cell.findAll('a')
     ...         print(anchors[0].get('href'))
     ...         # Badges in the next cell
-    ...         for img in cells[1].findAll('img'):
-    ...             print(img['title'])
+    ...         if len(cells) > 1:
+    ...             for img in cells[1].findAll('img'):
+    ...                 print(img['title'])
 
     >>> browser.open(
     ...     'http://code.launchpad.test/firefox/+branches'
diff --git a/lib/lp/code/stories/branches/xx-branch-tag-cloud.txt b/lib/lp/code/stories/branches/xx-branch-tag-cloud.txt
index cfd404d..ff953cf 100644
--- a/lib/lp/code/stories/branches/xx-branch-tag-cloud.txt
+++ b/lib/lp/code/stories/branches/xx-branch-tag-cloud.txt
@@ -25,6 +25,6 @@ the link is shown.
 
     >>> tags = find_tag_by_id(anon_browser.contents, 'project-tags')
     >>> for anchor in tags.findAll('a'):
-    ...     print(anchor.renderContents(), anchor['class'])
+    ...     print(anchor.renderContents(), ' '.join(anchor['class']))
     linux cloud-size-largest cloud-medium
     wibble cloud-size-smallest cloud-dark
diff --git a/lib/lp/code/stories/branches/xx-branchmergeproposal-listings.txt b/lib/lp/code/stories/branches/xx-branchmergeproposal-listings.txt
index 8df8d36..6182e73 100644
--- a/lib/lp/code/stories/branches/xx-branchmergeproposal-listings.txt
+++ b/lib/lp/code/stories/branches/xx-branchmergeproposal-listings.txt
@@ -78,7 +78,7 @@ summary.
     >>> print_tag_with_id(browser.contents, "proposals")
     Approved reviews ready to land
     Branch Merge Proposal   Requested By          Lines Activity
-    lp://dev/~bob/fooix/approved &rArr; lp://dev/fooix
+    lp://dev/~bob/fooix/approved ⇒ lp://dev/fooix
             Bob                                         None
     Reviews requested or in progress
     Branch Merge Proposal   Requested By          Lines Activity
diff --git a/lib/lp/code/stories/branches/xx-branchmergeproposals.txt b/lib/lp/code/stories/branches/xx-branchmergeproposals.txt
index 2c9c76e..8c1eb2e 100644
--- a/lib/lp/code/stories/branches/xx-branchmergeproposals.txt
+++ b/lib/lp/code/stories/branches/xx-branchmergeproposals.txt
@@ -383,7 +383,7 @@ branch widget is shown.
     >>> nopriv_browser.getLink('Propose for merging').click()
     >>> for widget in get_target_branch_widgets(nopriv_browser):
     ...     print(widget)
-    <input type="text" ...
+    <input ... type="text" ...
 
 Test validation of errors...
 
diff --git a/lib/lp/code/stories/branches/xx-bug-branch-links.txt b/lib/lp/code/stories/branches/xx-bug-branch-links.txt
index 12b3cb0..401fff0 100644
--- a/lib/lp/code/stories/branches/xx-bug-branch-links.txt
+++ b/lib/lp/code/stories/branches/xx-bug-branch-links.txt
@@ -157,5 +157,4 @@ Deleting a branch with linked bugs
 
     >>> print(find_tag_by_id(admin_browser.contents, 'deletion-items'))
     <ul ...
-    <a href...>Bug #...: bug-title...</a>...
-
+    <a ...href...>Bug #...: bug-title...</a>...
diff --git a/lib/lp/code/stories/branches/xx-person-portlet-teambranches.txt b/lib/lp/code/stories/branches/xx-person-portlet-teambranches.txt
index e37bcef..b9886ed 100644
--- a/lib/lp/code/stories/branches/xx-person-portlet-teambranches.txt
+++ b/lib/lp/code/stories/branches/xx-person-portlet-teambranches.txt
@@ -36,6 +36,6 @@ there.
     Branches owned by
     >>> print(tb.li)
     <li>
-      <img src="http://.../vikings.png"; width="14" height="14" />
+      <img height="14" src="http://.../vikings.png"; width="14"/>
       <a href="/~vikings">Vikings</a>
     </li>
diff --git a/lib/lp/code/stories/codeimport/xx-codeimport-results.txt b/lib/lp/code/stories/codeimport/xx-codeimport-results.txt
index 7e762f1..039ee60 100644
--- a/lib/lp/code/stories/codeimport/xx-codeimport-results.txt
+++ b/lib/lp/code/stories/codeimport/xx-codeimport-results.txt
@@ -30,7 +30,8 @@ file stored with the result.
 
     >>> browser.open(branch_url_1)
     >>> import_results = find_tag_by_id(browser.contents, 'import-results')
-    >>> print(extract_text(import_results).replace('&mdash;', '--'))
+    >>> print(extract_text(
+    ...     import_results, formatter='html').replace('&mdash;', '--'))
     Import started on 2007-12-07 on odin and finished on 2007-12-07
       taking 7 hours -- see the log
     Import started on 2007-12-06 on odin and finished on 2007-12-06
@@ -54,19 +55,19 @@ is the text of the failure or success type.
     >>> # are declared in the enumeration.
     >>> for img in import_results.findAll('img'):
     ...     print(img)
-    <img src="/@@/no" title="Unsupported feature" />
-    <img src="/@@/no" title="Foreign branch invalid" />
-    <img src="/@@/no" title="Internal Failure" />
-    <img src="/@@/no" title="Failure" />
-    <img src="/@@/yes-gray" title="Partial Success" />
-    <img src="/@@/yes" title="Success with no changes" />
-    <img src="/@@/yes" title="Success" />
+    <img src="/@@/no" title="Unsupported feature"/>
+    <img src="/@@/no" title="Foreign branch invalid"/>
+    <img src="/@@/no" title="Internal Failure"/>
+    <img src="/@@/no" title="Failure"/>
+    <img src="/@@/yes-gray" title="Partial Success"/>
+    <img src="/@@/yes" title="Success with no changes"/>
+    <img src="/@@/yes" title="Success"/>
 
     >>> browser.open(branch_url_2)
     >>> import_results = find_tag_by_id(browser.contents, 'import-results')
     >>> for img in import_results.findAll('img'):
     ...     print(img)
-    <img src="/@@/no" title="Job killed" />
-    <img src="/@@/no" title="Job reclaimed" />
-    <img src="/@@/no" title="Broken remote branch" />
-    <img src="/@@/no" title="Forbidden URL" />
+    <img src="/@@/no" title="Job killed"/>
+    <img src="/@@/no" title="Job reclaimed"/>
+    <img src="/@@/no" title="Broken remote branch"/>
+    <img src="/@@/no" title="Forbidden URL"/>
diff --git a/lib/lp/registry/browser/tests/nameblacklist-views.txt b/lib/lp/registry/browser/tests/nameblacklist-views.txt
index ed5d62d..0a5bc44 100644
--- a/lib/lp/registry/browser/tests/nameblacklist-views.txt
+++ b/lib/lp/registry/browser/tests/nameblacklist-views.txt
@@ -25,7 +25,8 @@ person names can be seen on the /+nameblacklist page.
     >>> ignored = login_person(registry_expert)
     >>> view = create_initialized_view(name_blacklist_set, '+index',
     ...                                principal=registry_expert)
-    >>> print extract_text(find_tag_by_id(view.render(), 'blacklist'))
+    >>> print extract_text(
+    ...     find_tag_by_id(view.render(), 'blacklist'), formatter='html')
     Regular Expression                   Admin    Comment
     ^admin Edit blacklist expression     &mdash;
     blacklist Edit blacklist expression  &mdash;  For testing purposes
diff --git a/lib/lp/registry/browser/tests/pillar-views.txt b/lib/lp/registry/browser/tests/pillar-views.txt
index 8add88b..603a3a0 100644
--- a/lib/lp/registry/browser/tests/pillar-views.txt
+++ b/lib/lp/registry/browser/tests/pillar-views.txt
@@ -133,7 +133,7 @@ The progress bar is shown as a green bar.
     >>> rendered = view.render()
     >>> print find_tag_by_id(rendered, 'progressbar')
     <div id="progressbar" ...>
-    <img src="/@@/green-bar" ... width: 25%.../>
+    <img ...src="/@@/green-bar" ... width: 25%.../>
     ...
 
 Each application is displayed (except for blueprints) with an
diff --git a/lib/lp/registry/browser/tests/productrelease-views.txt b/lib/lp/registry/browser/tests/productrelease-views.txt
index b826b36..60455fc 100644
--- a/lib/lp/registry/browser/tests/productrelease-views.txt
+++ b/lib/lp/registry/browser/tests/productrelease-views.txt
@@ -132,8 +132,8 @@ show a formoverlay that updates the milestone_for_release field.
             Y.on('domready', function () {
                 var select_menu = get_by_id('field.milestone_for_release');
                 var create_milestone_link = Y.Node.create(
-                    '&lt;a href="+addmilestone" id="create-milestone-link" ' +
-                    'class="add js-action sprite"&gt;Create milestone&lt;/a&gt;'); ...
+                    '<a href="+addmilestone" id="create-milestone-link" ' +
+                    'class="add js-action sprite">Create milestone</a>'); ...
 
 
 Editing a a product release
diff --git a/lib/lp/registry/browser/tests/productseries-views.txt b/lib/lp/registry/browser/tests/productseries-views.txt
index b400936..3aa6138 100644
--- a/lib/lp/registry/browser/tests/productseries-views.txt
+++ b/lib/lp/registry/browser/tests/productseries-views.txt
@@ -99,7 +99,7 @@ the class table is 'listing hidden'.
     listing hidden
 
     >>> table = find_tag_by_id(view.render(), 'series-simple')
-    >>> print table['class']
+    >>> print ' '.join(table['class'])
     listing hidden
 
 When the product series has milestones, the class is just 'listing'.
diff --git a/lib/lp/registry/browser/tests/sourcepackage-views.txt b/lib/lp/registry/browser/tests/sourcepackage-views.txt
index 40151ed..d157ee5 100644
--- a/lib/lp/registry/browser/tests/sourcepackage-views.txt
+++ b/lib/lp/registry/browser/tests/sourcepackage-views.txt
@@ -184,7 +184,7 @@ bonkers project, the portlet will display that information.
     Bug supervisor: no
     Bug tracker: no
     Branch: no
-    There are no registered releases for the Bonkers &rArr; crazy.
+    There are no registered releases for the Bonkers ⇒ crazy.
 
 A new source project that is not linked to an upstream will result in
 the portlet showing the suggested project.
@@ -201,8 +201,8 @@ the portlet showing the suggested project.
 
     >>> content = extract_text(find_tag_by_id(view.render(), 'no-upstreams'))
     >>> print content
-    Launchpad doesn&#8217;t know which project and series this
-    package belongs to. ...
+    Launchpad doesn’t know which project and series this package belongs to.
+    ...
     Is the following project the upstream for this source package?
     Registered upstream project:
     Lernid
@@ -231,8 +231,8 @@ item is reserved for the "Choose another upstream project" option.
 
     >>> content = extract_text(find_tag_by_id(view.render(), 'no-upstreams'))
     >>> print content
-    Launchpad doesn&#8217;t know which project and series this
-    package belongs to. ...
+    Launchpad doesn’t know which project and series this package belongs to.
+    ...
     Is one of these projects the upstream for this source package?
     Registered upstream project:
     Lernid...
@@ -380,7 +380,7 @@ deleted.
 
     >>> view = create_initialized_view(package, name='+portlet-associations')
     >>> print extract_text(find_tag_by_id(view.render(), 'no-upstreams'))
-    Launchpad doesn&#8217;t know which project ...
+    Launchpad doesn’t know which project ...
     There are no projects registered in Launchpad that are a potential
     match for this source package. Can you help us find one?
     Registered upstream project:
diff --git a/lib/lp/registry/browser/tests/test_peoplemerge.py b/lib/lp/registry/browser/tests/test_peoplemerge.py
index 724917e..a2ea2b6 100644
--- a/lib/lp/registry/browser/tests/test_peoplemerge.py
+++ b/lib/lp/registry/browser/tests/test_peoplemerge.py
@@ -73,8 +73,8 @@ class TestRequestPeopleMergeMultipleEmails(RequestPeopleMergeMixin):
         explanation = find_tag_by_id(browser.contents, 'explanation')
         self.assertThat(
             extract_text(explanation), DocTestMatches(
-                "The account..."
-                "has more than one registered email address..."))
+                u"The account..."
+                u"has more than one registered email address..."))
         email_select_control = browser.getControl(name='selected')
         for ctrl in email_select_control.controls:
             ctrl.selected = True
diff --git a/lib/lp/registry/browser/tests/test_team.py b/lib/lp/registry/browser/tests/test_team.py
index 7eb0de9..2056ecc 100644
--- a/lib/lp/registry/browser/tests/test_team.py
+++ b/lib/lp/registry/browser/tests/test_team.py
@@ -7,6 +7,7 @@ import contextlib
 
 from lazr.restful.interfaces import IJSONRequestCache
 import simplejson
+import soupmatchers
 import transaction
 from zope.component import getUtility
 from zope.security.proxy import removeSecurityProxy
@@ -924,7 +925,7 @@ class TestPersonIndexVisibilityView(TestCaseWithFactory):
             self.assertEqual(view.super_teams, list(team.super_teams))
             superteams = find_tag_by_id(html, 'subteam-of')
         self.assertFalse('&lt;hidden&gt;' in superteams)
-        self.assertEqual(
-            '<a href="/~private-team" class="sprite team private">'
-            'Private Team</a>',
-            str(superteams.findNext('a')))
+        self.assertThat(superteams, soupmatchers.Tag(
+            'private team link', 'a',
+            attrs={'href': '/~private-team', 'class': 'sprite team private'},
+            text='Private Team'))
diff --git a/lib/lp/registry/doc/product-widgets.txt b/lib/lp/registry/doc/product-widgets.txt
index b289107..f5fb0e6 100644
--- a/lib/lp/registry/doc/product-widgets.txt
+++ b/lib/lp/registry/doc/product-widgets.txt
@@ -42,8 +42,8 @@ Firefox has not yet selected a bug tracker.
     >>> print firefox.projectgroup.bugtracker
     None
 
-    >>> from BeautifulSoup import Tag
-    >>> from lp.services.beautifulsoup import BeautifulSoup
+    >>> from bs4.element import Tag
+    >>> from lp.services.beautifulsoup import BeautifulSoup4 as BeautifulSoup
     >>> from lp.testing.pages import extract_text
     >>> def print_items(html):
     ...     soup = BeautifulSoup(html)
@@ -288,7 +288,7 @@ presented ordered to appear in a 3 column list.
 
 
     >>> print extract_text(find_tag_by_id(html, 'special'))
-    I don&#x27;t know yet
+    I don't know yet
     Other/Proprietary
     Other/Open Source
 
@@ -373,7 +373,7 @@ But not all of them.
     >>> print_checked_items(license_widget(), links=True)
     [ ] Apache Licence ... <http://www.opensource.org/licenses/apache2.0.php>
     ...
-    [ ] I don&#x27;t know yet
+    [ ] I don't know yet
     [ ] Other/Proprietary
     [ ] Other/Open Source
 
diff --git a/lib/lp/registry/stories/announcements/xx-announcements.txt b/lib/lp/registry/stories/announcements/xx-announcements.txt
index addfc4a..3a8008d 100644
--- a/lib/lp/registry/stories/announcements/xx-announcements.txt
+++ b/lib/lp/registry/stories/announcements/xx-announcements.txt
@@ -612,7 +612,7 @@ not be able to move it.
     Move announcement : Kubuntu announcement headline : GuadaLinex
     >>> kamion_browser.getControl('For').value = 'kubuntu'
     >>> kamion_browser.getControl('Retarget').click()
-    >>> "don&#x27;t have permission" in extract_text(
+    >>> "don't have permission" in extract_text(
     ...     find_main_content(kamion_browser.contents))
     True
     >>> print kamion_browser.title
diff --git a/lib/lp/registry/stories/distribution/xx-distribution-overview.txt b/lib/lp/registry/stories/distribution/xx-distribution-overview.txt
index 3918c76..cc5ddcf 100644
--- a/lib/lp/registry/stories/distribution/xx-distribution-overview.txt
+++ b/lib/lp/registry/stories/distribution/xx-distribution-overview.txt
@@ -30,8 +30,8 @@ Some distributions have listings of major versions, for example Debian:
     >>> anon_browser.open('http://launchpad.test/debian')
     >>> print extract_text(find_tag_by_id(anon_browser.contents, 'sandm'))
     Active series and milestones
-    3.1 &#8220;Sarge&#8221; series - frozen
-    3.0 &#8220;Woody&#8221; series - current
+    3.1 “Sarge” series - frozen
+    3.0 “Woody” series - current
         Milestones: 3.1 and 3.1-rc1
     All series
     All milestones
@@ -88,16 +88,16 @@ along with a link to list all of them.
     ...     find_tag_by_id(anon_browser.contents, 'derivatives'))
     Latest derivatives
     9.9.9
-    &#8220;Hoary Mock&#8221; series
+    “Hoary Mock” series
     (from Warty)
     8.06
-    &#8220;Krunch&#8221; series
+    “Krunch” series
     (from Hoary)
     6.6.6
-    &#8220;Breezy Badger Autotest&#8221; series
+    “Breezy Badger Autotest” series
     (from Warty)
     2005
-    &#8220;Guada2005&#8221; series
+    “Guada2005” series
     (from Hoary)
     All derivatives
 
diff --git a/lib/lp/registry/stories/distributionmirror/xx-distributionmirror-prober-logs.txt b/lib/lp/registry/stories/distributionmirror/xx-distributionmirror-prober-logs.txt
index 2ec43df..b87badf 100644
--- a/lib/lp/registry/stories/distributionmirror/xx-distributionmirror-prober-logs.txt
+++ b/lib/lp/registry/stories/distributionmirror/xx-distributionmirror-prober-logs.txt
@@ -14,8 +14,8 @@ that mirror's distribution.
 
     >>> navigation = find_tags_by_class(
     ...     browser.contents, 'batch-navigation-index')[0]
-    >>> extract_text(navigation.renderContents())
-    u'1...&rarr;...1...of...1...result'
+    >>> print(extract_text(navigation.decode_contents()))
+    1...→...1...of...1...result
 
 A random logged in user won't have the rights to see that page.
 
diff --git a/lib/lp/registry/stories/distributionmirror/xx-reassign-distributionmirror.txt b/lib/lp/registry/stories/distributionmirror/xx-reassign-distributionmirror.txt
index 74b9184..53940e3 100644
--- a/lib/lp/registry/stories/distributionmirror/xx-reassign-distributionmirror.txt
+++ b/lib/lp/registry/stories/distributionmirror/xx-reassign-distributionmirror.txt
@@ -39,8 +39,7 @@ First we'll enter an unexistent name.
     >>> for tag in find_tags_by_class(browser.contents, 'message'):
     ...     tag.renderContents()
     'There is 1 error.'
-    'There&#x27;s no person/team named &#x27;unexistent-name&#x27; in
-    Launchpad.'
+    "There's no person/team named 'unexistent-name' in Launchpad."
 
 We also try to use the name of an unvalidated account, which can't be used as
 the owner of something.
@@ -55,7 +54,7 @@ the owner of something.
     >>> for tag in find_tags_by_class(browser.contents, 'message'):
     ...     tag.renderContents()
     'There is 1 error.'
-    'The person/team named &#x27;matsubara&#x27; is not a valid owner for ...'
+    "The person/team named 'matsubara' is not a valid owner for ..."
 
 Now we try to create a team using a name that is already taken.
 
@@ -67,8 +66,7 @@ Now we try to create a team using a name that is already taken.
     >>> for tag in find_tags_by_class(browser.contents, 'message'):
     ...     tag.renderContents()
     'There is 1 error.'
-    'There&#x27;s already a person/team with the name &#x27;name16&#x27;
-    in Launchpad...'
+    "There's already a person/team with the name 'name16' in Launchpad..."
 
 Okay, let's do it properly now and reassign it to an existing (and validated)
 account.
diff --git a/lib/lp/registry/stories/milestone/object-milestones.txt b/lib/lp/registry/stories/milestone/object-milestones.txt
index 8a3e624..eb0efe0 100644
--- a/lib/lp/registry/stories/milestone/object-milestones.txt
+++ b/lib/lp/registry/stories/milestone/object-milestones.txt
@@ -363,7 +363,7 @@ Each bugtask has one or more badges.
 
     >>> print bug_table.findAll('tr')[1]
     <tr>...Test Bug 1...<a...alt="milestone test-milestone"...
-      class="sprite milestone">...
+      class="sprite milestone"...>...
 
 
 Bugs targeted to development focus series
diff --git a/lib/lp/registry/stories/packaging/xx-sourcepackage-packaging.txt b/lib/lp/registry/stories/packaging/xx-sourcepackage-packaging.txt
index 7665740..7f33abc 100644
--- a/lib/lp/registry/stories/packaging/xx-sourcepackage-packaging.txt
+++ b/lib/lp/registry/stories/packaging/xx-sourcepackage-packaging.txt
@@ -128,6 +128,6 @@ to the source package page and an informational message will be displayed.
     Linked Bonkers project to bonkers source package.
     >>> print extract_text(
     ...     find_tag_by_id(user_browser.contents, 'upstreams'))
-    Bonkers &rArr; trunk
+    Bonkers ⇒ trunk
     Change upstream link
     Remove upstream link...
diff --git a/lib/lp/registry/stories/person/xx-people-search.txt b/lib/lp/registry/stories/person/xx-people-search.txt
index 0fd77d0..702092b 100644
--- a/lib/lp/registry/stories/person/xx-people-search.txt
+++ b/lib/lp/registry/stories/person/xx-people-search.txt
@@ -20,7 +20,7 @@ should just be the one person named "Foo Bar" found.
 
 The listing is sortable.
 
-    >>> print listing['class']
+    >>> print ' '.join(listing['class'])
     listing sortable
 
 Search for all people and teams like "launchpad" the users sees three
@@ -29,7 +29,7 @@ columns of people and teams..
     >>> browser.getControl(name='name').value = 'launchpad'
     >>> browser.getControl('Search').click()
     >>> listing = find_tag_by_id(browser.contents, 'people-results')
-    >>> print extract_text(listing)
+    >>> print extract_text(listing, formatter='html')
     Name                         Launchpad ID              Karma
     Julian Edwards               launchpad-julian-edwards  0
     Launchpad Administrators     admins                    &mdash;
diff --git a/lib/lp/registry/stories/person/xx-person-subscriptions.txt b/lib/lp/registry/stories/person/xx-person-subscriptions.txt
index 654c529..669d940 100644
--- a/lib/lp/registry/stories/person/xx-person-subscriptions.txt
+++ b/lib/lp/registry/stories/person/xx-person-subscriptions.txt
@@ -295,7 +295,7 @@ displayed immediately after the subscription.
     >>> show_nigels_subscriptions()
     Bug mail for Nigel about Scofflaw is filtered; it will be sent
     only if it matches the following filter:
-    &#8220;First&#8221; allows mail through when:
+    “First” allows mail through when:
     the bug is tagged with foo
     (edit)
 
@@ -310,9 +310,9 @@ message.
     >>> show_nigels_subscriptions()
     Bug mail for Nigel about Scofflaw is filtered; it will be sent
     only if it matches one or more of the following filters:
-    &#8220;First&#8221; allows mail through when:
+    “First” allows mail through when:
     the bug is tagged with foo
     (edit)
-    &#8220;Second&#8221; allows mail through when:
+    “Second” allows mail through when:
     the bug is tagged with bar
     (edit)
diff --git a/lib/lp/registry/stories/product/xx-launchpad-project-search.txt b/lib/lp/registry/stories/product/xx-launchpad-project-search.txt
index 3c7ad8f..9e1b3b0 100644
--- a/lib/lp/registry/stories/product/xx-launchpad-project-search.txt
+++ b/lib/lp/registry/stories/product/xx-launchpad-project-search.txt
@@ -50,7 +50,7 @@ The search results contain projects, project-groups, and distributions.
     ...     search_results = find_tag_by_id(
     ...         abrowser.contents, 'search-results')
     ...     for tr in search_results.tbody('tr'):
-    ...         print tr.td.a['class'], extract_text(tr.td.a)
+    ...         print ' '.join(tr.td.a['class']), extract_text(tr.td.a)
     >>> print_search_results(anon_browser)
     sprite distribution Ubuntu
     sprite distribution ubuntutest
diff --git a/lib/lp/registry/stories/product/xx-product-files.txt b/lib/lp/registry/stories/product/xx-product-files.txt
index 4d72d4d..403e23e 100644
--- a/lib/lp/registry/stories/product/xx-product-files.txt
+++ b/lib/lp/registry/stories/product/xx-product-files.txt
@@ -238,7 +238,7 @@ Uploading file signatures is optional, so we'll just try it this once.
     ...     name="field.contenttype").displayValue = ["Installer file"]
     >>> firefox_owner.getControl("Upload").click()
     >>> print_feedback_messages(firefox_owner.contents)
-    Your file &#x27;foo.txt&#x27; has been uploaded.
+    Your file 'foo.txt' has been uploaded.
 
 A file can be uploaded without a GPG signature.
 
@@ -252,7 +252,7 @@ A file can be uploaded without a GPG signature.
     ...     name="field.contenttype").displayValue = ["Installer file"]
     >>> firefox_owner.getControl("Upload").click()
     >>> print_feedback_messages(firefox_owner.contents)
-    Your file &#x27;bar.txt&#x27; has been uploaded.
+    Your file 'bar.txt' has been uploaded.
 
 The uploaded file is also displayed on the project's downloads page for any
 user to see.
@@ -348,7 +348,7 @@ Now let's successfully upload two more files.
     ...     name="field.contenttype").displayValue = ["Installer file"]
     >>> firefox_owner.getControl("Upload").click()
     >>> print_feedback_messages(firefox_owner.contents)
-    Your file &#x27;foo2.txt&#x27; has been uploaded.
+    Your file 'foo2.txt' has been uploaded.
 
     >>> firefox_owner.open('http://launchpad.test/firefox/1.0/1.0.0')
     >>> firefox_owner.getLink('Add download file').click()
@@ -360,7 +360,7 @@ Now let's successfully upload two more files.
     ...     name="field.contenttype").displayValue = ["Installer file"]
     >>> firefox_owner.getControl("Upload").click()
     >>> print_feedback_messages(firefox_owner.contents)
-    Your file &#x27;foo3.txt&#x27; has been uploaded.
+    Your file 'foo3.txt' has been uploaded.
 
 Add a file to a different release on the same project.
 
@@ -374,7 +374,7 @@ Add a file to a different release on the same project.
     ...     name="field.contenttype").displayValue = ["README File"]
     >>> firefox_owner.getControl("Upload").click()
     >>> print_feedback_messages(firefox_owner.contents)
-    Your file &#x27;foo09.txt&#x27; has been uploaded.
+    Your file 'foo09.txt' has been uploaded.
 
 Examine all of the available files for download for firefox now.  They
 are listed within series in reverse chronological order, except
diff --git a/lib/lp/registry/stories/product/xx-product-index.txt b/lib/lp/registry/stories/product/xx-product-index.txt
index 0b5b75b..bd16600 100644
--- a/lib/lp/registry/stories/product/xx-product-index.txt
+++ b/lib/lp/registry/stories/product/xx-product-index.txt
@@ -103,7 +103,7 @@ Any user can see that the project's licence has not been reviewed.
     >>> user_browser.open('http://launchpad.test/thunderbird')
     >>> print extract_text(
     ...     find_tag_by_id(user_browser.contents, 'license-status'))
-    This project&rsquo;s licence has not been reviewed.
+    This project’s licence has not been reviewed.
 
 Changing the state to reviewed but not approved results in the project
 being shown as proprietary.
@@ -130,7 +130,7 @@ direct the owner to purchase a subscription.
     >>> logout()
     >>> owner_browser.open('http://launchpad.test/firefox')
     >>> print find_tag_by_id(owner_browser.contents, 'license-status')
-    <...This project&rsquo;s licence is proprietary...
+    <...This project’s licence is proprietary...
 
     >>> print find_tag_by_id(owner_browser.contents,
     ...     'portlet-requires-subscription')
diff --git a/lib/lp/registry/stories/productrelease/xx-productrelease-view.txt b/lib/lp/registry/stories/productrelease/xx-productrelease-view.txt
index bd594ac..8c41ffd 100644
--- a/lib/lp/registry/stories/productrelease/xx-productrelease-view.txt
+++ b/lib/lp/registry/stories/productrelease/xx-productrelease-view.txt
@@ -26,8 +26,8 @@ release. Each file is linked.
     firefox_0.9.2.orig.tar.gz (md5)              -
 
     >>> print table.a
-    <a title="firefox_0.9.2.orig.tar.gz (9.5 MiB)"
-       href=".../firefox/trunk/0.9.2/+download/firefox_0.9.2.orig.tar.gz">...
+    <a href=".../firefox/trunk/0.9.2/+download/firefox_0.9.2.orig.tar.gz"
+       title="firefox_0.9.2.orig.tar.gz (9.5 MiB)">...
 
 There is an link about how to verify downloaded files.
 
diff --git a/lib/lp/registry/stories/productseries/xx-productseries-index.txt b/lib/lp/registry/stories/productseries/xx-productseries-index.txt
index 42b630d..bbd88b8 100644
--- a/lib/lp/registry/stories/productseries/xx-productseries-index.txt
+++ b/lib/lp/registry/stories/productseries/xx-productseries-index.txt
@@ -23,7 +23,7 @@ The product series overview page summarises the series.
     Download RDF metadata
 
     >>> print extract_text(find_tag_by_id(content, 'description'))
-    The &quot;trunk&quot; series represents the primary line of
+    The "trunk" series represents the primary line of
     development rather than a stable release branch. This is sometimes
     also called MAIN or HEAD.
 
diff --git a/lib/lp/registry/stories/productseries/xx-productseries-series.txt b/lib/lp/registry/stories/productseries/xx-productseries-series.txt
index e789fcf..fe0b3ab 100644
--- a/lib/lp/registry/stories/productseries/xx-productseries-series.txt
+++ b/lib/lp/registry/stories/productseries/xx-productseries-series.txt
@@ -32,13 +32,12 @@ listed.
     Latest milestones: 1.0    Latest releases: 0.9.2, 0.9.1, 0.9
     Bugs targeted: None
     Blueprints targeted: 1 Unknown
-    The &quot;trunk&quot; series represents the primary line of
-    development rather ...
+    The "trunk" series represents the primary line of development rather ...
 
 Any user can see that the trunk series is the focus of development and that
 it is highlighted.
 
-    >>> print series_trunk['class']
+    >>> print ' '.join(series_trunk['class'])
     highlight series
 
 The 1.0 series is not the focus of development, it is active, so it is not
@@ -52,7 +51,7 @@ highlighted.
     Blueprints targeted: None
     The 1.0 branch of the Mozilla web browser. Currently, this is the ...
 
-    >>> print series_1_0['class']
+    >>> print ' '.join(series_1_0['class'])
     series
 
 Any user can see that obsolete series are lowlight. Obsolete series do not
@@ -64,5 +63,5 @@ show bug status counts because it is expensive to retrieve the information.
     Blueprints targeted: None
     Use true GTK UI.
 
-    >>> print series_xxx['class']
+    >>> print ' '.join(series_xxx['class'])
     lowlight series
diff --git a/lib/lp/registry/stories/project/xx-project-add.txt b/lib/lp/registry/stories/project/xx-project-add.txt
index a2d79e3..e0bcb64 100644
--- a/lib/lp/registry/stories/project/xx-project-add.txt
+++ b/lib/lp/registry/stories/project/xx-project-add.txt
@@ -26,7 +26,7 @@ Add a new project without the http://
   >>> admin_browser.getControl('Add').click()
   >>> print_feedback_messages(admin_browser.contents)
   There is 1 error.
-  &quot;www.kde.org&quot; is not a valid URI
+  "www.kde.org" is not a valid URI
 
 Testing if the validator is working for the name field.
 
diff --git a/lib/lp/registry/stories/team-polls/create-poll-options.txt b/lib/lp/registry/stories/team-polls/create-poll-options.txt
index b541d02..2a9711a 100644
--- a/lib/lp/registry/stories/team-polls/create-poll-options.txt
+++ b/lib/lp/registry/stories/team-polls/create-poll-options.txt
@@ -77,9 +77,9 @@ That's only possible when the poll is not yet open.
     ...     'http://launchpad.test/~ubuntu-team/+poll/director-2004/+newoption')
 
     >>> print_feedback_messages(team_admin_browser.contents)
-    You can&#8217;t add new options because the poll is already closed.
+    You can’t add new options because the poll is already closed.
 
     >>> team_admin_browser.open(
     ...     'http://launchpad.test/~ubuntu-team/+poll/never-closes/+newoption')
     >>> print_feedback_messages(team_admin_browser.contents)
-    You can&#8217;t add new options because the poll is already open.
+    You can’t add new options because the poll is already open.
diff --git a/lib/lp/registry/stories/team-polls/create-polls.txt b/lib/lp/registry/stories/team-polls/create-polls.txt
index a3d16b6..ab9d1f5 100644
--- a/lib/lp/registry/stories/team-polls/create-polls.txt
+++ b/lib/lp/registry/stories/team-polls/create-polls.txt
@@ -162,4 +162,4 @@ the poll opens -- at that point new options cannot be added.
     >>> team_admin_browser.getControl('Continue').click()
     >>> print_feedback_messages(team_admin_browser.contents)
     There is 1 error.
-    A poll cannot open less than 12 hours after it&#x27;s created.
+    A poll cannot open less than 12 hours after it's created.
diff --git a/lib/lp/registry/stories/team-polls/edit-options.txt b/lib/lp/registry/stories/team-polls/edit-options.txt
index 36b4d97..38d2ec4 100644
--- a/lib/lp/registry/stories/team-polls/edit-options.txt
+++ b/lib/lp/registry/stories/team-polls/edit-options.txt
@@ -26,7 +26,7 @@ either:
     >>> browser.open('http://launchpad.test/~ubuntu-team/+poll/never-closes4/'
     ...              '+option/20')
     >>> print_feedback_messages(browser.contents)
-    You can&#8217;t edit any options because the poll is already open.
+    You can’t edit any options because the poll is already open.
 
 Since Jeff is an administrator of ubuntu-team and we have a poll that hasn't
 been opened yet, he should be able to edit its options.
diff --git a/lib/lp/registry/stories/team-polls/vote-poll.txt b/lib/lp/registry/stories/team-polls/vote-poll.txt
index 603561d..561f330 100644
--- a/lib/lp/registry/stories/team-polls/vote-poll.txt
+++ b/lib/lp/registry/stories/team-polls/vote-poll.txt
@@ -111,8 +111,7 @@ even if they guess the URL for the voting page.
     >>> for tag in find_tags_by_class(
     ...     non_member_browser.contents, "informational message"):
     ...     print tag.renderContents()
-    You can&#8217;t vote in this poll because you&#8217;re not a member
-    of Ubuntu Team.
+    You can’t vote in this poll because you’re not a member of Ubuntu Team.
 
 
 == Closed polls ==
diff --git a/lib/lp/registry/stories/team-polls/xx-poll-results.txt b/lib/lp/registry/stories/team-polls/xx-poll-results.txt
index def5e4b..a2a96a6 100644
--- a/lib/lp/registry/stories/team-polls/xx-poll-results.txt
+++ b/lib/lp/registry/stories/team-polls/xx-poll-results.txt
@@ -18,7 +18,7 @@ First we check all polls of 'ubuntu-team'.
   >>> anon_browser.open("http://launchpad.test/~ubuntu-team/+poll/leader-2004";)
   >>> print find_main_content(anon_browser.contents)
   <...
-  ...Who&#x27;s going to be the next leader?...
+  ...Who's going to be the next leader?...
   ...Results...
   ...
               <td>
@@ -46,7 +46,7 @@ First we check all polls of 'ubuntu-team'.
   >>> anon_browser.open("http://launchpad.test/~ubuntu-team/+poll/director-2004";)
   >>> print find_main_content(anon_browser.contents)
   <...
-  ...Who&#x27;s going to be the next director?...
+  ...Who's going to be the next director?...
   ...Results...
   ...
   ...A...
diff --git a/lib/lp/registry/stories/team/xx-team-add-my-teams.txt b/lib/lp/registry/stories/team/xx-team-add-my-teams.txt
index d2f679d..1a99bd5 100644
--- a/lib/lp/registry/stories/team/xx-team-add-my-teams.txt
+++ b/lib/lp/registry/stories/team/xx-team-add-my-teams.txt
@@ -107,7 +107,8 @@ your teams as members.
     LinkNotFoundError
 
     >>> browser.open('http://launchpad.test/~ubuntu-team/+add-my-teams')
-    >>> print extract_text(find_tag_by_id(browser.contents, 'candidates'))
+    >>> print extract_text(
+    ...     find_tag_by_id(browser.contents, 'candidates'), formatter='html')
     This is a restricted team
     New members can not be proposed&mdash;they can only be added by one
     of the team's administrators.
diff --git a/lib/lp/registry/stories/team/xx-team-claim.txt b/lib/lp/registry/stories/team/xx-team-claim.txt
index 339ad2b..e18bb82 100644
--- a/lib/lp/registry/stories/team/xx-team-claim.txt
+++ b/lib/lp/registry/stories/team/xx-team-claim.txt
@@ -47,7 +47,7 @@ turning that profile into a team.
 
     >>> print_feedback_messages(user_browser.contents)
     A confirmation message has been sent to
-    &#x27;doc@xxxxxxxxxxxxxxxx&#x27;. Follow the instructions in that
+    'doc@xxxxxxxxxxxxxxxx'. Follow the instructions in that
     message to finish claiming this team. (If the above address is from
     a mailing list, it may be necessary to talk with one of its admins
     to accept the message from Launchpad so that you can finish the
diff --git a/lib/lp/registry/stories/team/xx-team-contactemail-xss.txt b/lib/lp/registry/stories/team/xx-team-contactemail-xss.txt
index aa92b2c..1b6856d 100644
--- a/lib/lp/registry/stories/team/xx-team-contactemail-xss.txt
+++ b/lib/lp/registry/stories/team/xx-team-contactemail-xss.txt
@@ -22,7 +22,7 @@ The markup is valid and correctly escaped:
     ...     admin_browser.contents, 'field.contact_address').prettify()
     <input class="textType" id="field.contact_address"
            name="field.contact_address" size="20" type="text"
-           value='&lt;script&gt;alert("cheezburger");&lt;/script&gt;' />
+           value='&lt;script&gt;alert("cheezburger");&lt;/script&gt;'/>
 
 The error message is also valid and correctly escaped:
 
@@ -31,7 +31,7 @@ The error message is also valid and correctly escaped:
     <p class="error message">
     ...
     <div class="message">
-    &lt;script&gt;alert(&quot;cheezburger&quot;);&lt;/script&gt; isn&#x27;t...
+    &lt;script&gt;alert("cheezburger");&lt;/script&gt; isn't...
     </div>
 
 The script we tried to inject is not present, unescaped, anywhere in
diff --git a/lib/lp/registry/stories/team/xx-team-home.txt b/lib/lp/registry/stories/team/xx-team-home.txt
index 006b1a4..4be0a91 100644
--- a/lib/lp/registry/stories/team/xx-team-home.txt
+++ b/lib/lp/registry/stories/team/xx-team-home.txt
@@ -76,8 +76,7 @@ The subteam-of portlet is not shown if the team is not a subteam.
     >>> browser.open('http://launchpad.test/~ubuntu-team')
     >>> print extract_text(
     ...     find_tag_by_id(browser.contents, 'subteam-of'))
-    Subteam of
-    &#8220;Ubuntu Team&#8221; is a member of these teams: GuadaMen...
+    Subteam of “Ubuntu Team” is a member of these teams: GuadaMen...
 
     >>> browser.open('http://launchpad.test/~launchpad')
     >>> print find_tag_by_id(browser.contents, 'subteam-of')
@@ -98,15 +97,15 @@ members, the empty lists are hidden using the "hidden" css class:
 
     >>> browser.open('http://launchpad.test/~launchpad')
     >>> tag = find_tag_by_id(browser.contents, 'recently-approved')
-    >>> print tag['class']
+    >>> print ' '.join(tag['class'])
     hidden
 
     >>> tag = find_tag_by_id(browser.contents, 'recently-proposed')
-    >>> print tag['class']
+    >>> print ' '.join(tag['class'])
     hidden
 
     >>> tag = find_tag_by_id(browser.contents, 'recently-invited')
-    >>> print tag['class']
+    >>> print ' '.join(tag['class'])
     hidden
 
 In the above case there's no user logged in, so it doesn't actually show
@@ -171,7 +170,7 @@ will even show the path from the user to the team.
     >>> print extract_text(
     ...     find_tag_by_id(sample_browser.contents, 'your-involvement'))
     You are an indirect member of this team:
-    Sample Person &rarr; Warty Security Team &rarr; Ubuntu Gnome Team...
+    Sample Person → Warty Security Team → Ubuntu Gnome Team...
 
 It is also possible to view the set of mugshots of the people in the
 team.  Notice that the output of mugshots is batched.
@@ -228,5 +227,5 @@ will actually go to the team's administrators.
     Contact this team's admins
 
     >>> content.a
-    <a href="+contactuser"...
+    <a ...href="+contactuser"...
       title="Send an email to this team's admins through Launchpad">...
diff --git a/lib/lp/registry/stories/team/xx-team-membership.txt b/lib/lp/registry/stories/team/xx-team-membership.txt
index f21ee07..4288f8e 100644
--- a/lib/lp/registry/stories/team/xx-team-membership.txt
+++ b/lib/lp/registry/stories/team/xx-team-membership.txt
@@ -223,7 +223,7 @@ Sample Person has both direct and indirect memberships:
     None
 
     >>> print extract_text(
-    ...     find_tag_by_id(content, 'participation'))
+    ...     find_tag_by_id(content, 'participation'), formatter='html')
     Team                  Joined      Role    Via                 Mailing List
     HWDB Team             2009-07-09  Member  &mdash;              &mdash;
     Landscape Developers  2006-07-11  Owner   &mdash;              &mdash;
@@ -254,6 +254,7 @@ list column.
 
     >>> admin_browser.open('http://launchpad.test/~admins/+participation')
     >>> print extract_text(
-    ...     find_tag_by_id(admin_browser.contents, 'participation'))
+    ...     find_tag_by_id(admin_browser.contents, 'participation'),
+    ...     formatter='html')
     Team                  Joined      Role    Via
     Mailing List Experts  2007-10-04  Owner   &mdash;
diff --git a/lib/lp/registry/stories/teammembership/xx-renew-subscription.txt b/lib/lp/registry/stories/teammembership/xx-renew-subscription.txt
index 2d887c5..d624aec 100644
--- a/lib/lp/registry/stories/teammembership/xx-renew-subscription.txt
+++ b/lib/lp/registry/stories/teammembership/xx-renew-subscription.txt
@@ -22,7 +22,7 @@ is approved because he is in the active members table.
     >>> print backslashreplace(browser.title)
     Members : \u201cUbuntu Gnome Team\u201d team
     >>> content = find_tag_by_id(browser.contents, 'activemembers')
-    >>> print extract_text(content)
+    >>> print extract_text(content, formatter='html')
     Name               Member since  Expires  Status
     ...
     Mark Shuttleworth  2005-03-03    &ndash;  Approved
diff --git a/lib/lp/registry/tests/test_product.py b/lib/lp/registry/tests/test_product.py
index c28eb8b..67de16c 100644
--- a/lib/lp/registry/tests/test_product.py
+++ b/lib/lp/registry/tests/test_product.py
@@ -93,7 +93,6 @@ from lp.registry.model.product import (
 from lp.registry.model.productlicense import ProductLicense
 from lp.services.database.interfaces import IStore
 from lp.services.webapp.authorization import check_permission
-from lp.services.webapp.escaping import html_escape
 from lp.testing import (
     celebrity_logged_in,
     login,
@@ -1433,7 +1432,7 @@ class TestProductFiles(TestCase):
         firefox_owner.getControl("Upload").click()
         self.assertEqual(
             get_feedback_messages(firefox_owner.contents),
-            [html_escape(u"Your file 'foo\xa5.txt' has been uploaded.")])
+            [u"Your file 'foo\xa5.txt' has been uploaded."])
         firefox_owner.open('http://launchpad.test/firefox/+download')
         content = find_main_content(firefox_owner.contents)
         rows = content.findAll('tr')
diff --git a/lib/lp/snappy/browser/tests/test_snap.py b/lib/lp/snappy/browser/tests/test_snap.py
index 165fd1f..5470916 100644
--- a/lib/lp/snappy/browser/tests/test_snap.py
+++ b/lib/lp/snappy/browser/tests/test_snap.py
@@ -1655,7 +1655,7 @@ class TestSnapRequestBuildsView(BaseTestSnapView):
             Source archive:
             Primary Archive for Ubuntu Linux
             PPA
-            \(Find&hellip;\)
+            \(Find\u2026\)
             Architectures:
             amd64
             i386
diff --git a/lib/lp/soyuz/doc/package-relationship-pages.txt b/lib/lp/soyuz/doc/package-relationship-pages.txt
index f9a0700..bb1abc3 100644
--- a/lib/lp/soyuz/doc/package-relationship-pages.txt
+++ b/lib/lp/soyuz/doc/package-relationship-pages.txt
@@ -57,14 +57,12 @@ correctly like:
      </li>
   </ul>
 
-  >>> from lp.testing.pages import (
-  ...     parse_relationship_section)
+  >>> from lp.testing.pages import parse_relationship_section
 
   >>> parse_relationship_section(pkg_rel_view())
-  LINK: "foobar (&gt;= 1.0.2)" -> http://whatever/
+  LINK: "foobar (>= 1.0.2)" -> http://whatever/
   TEXT: "test (= 1.0)"
 
 
 Note that no link is rendered for IPackageReleationship where 'url' is
 None.
-
diff --git a/lib/lp/soyuz/stories/ppa/xx-copy-packages.txt b/lib/lp/soyuz/stories/ppa/xx-copy-packages.txt
index d52132e..8159d2d 100644
--- a/lib/lp/soyuz/stories/ppa/xx-copy-packages.txt
+++ b/lib/lp/soyuz/stories/ppa/xx-copy-packages.txt
@@ -63,7 +63,7 @@ PPA in order to be able to copy packages.
     ...
     To be able to copy packages you have to participate in at least one
     PPA. Activate your own PPA or join a team with an active PPA.
-    Create a new PPA or &laquo;back
+    Create a new PPA or «back
 
 James will follow the advice and activate his PPA by clicking in the
 provided link.
diff --git a/lib/lp/soyuz/stories/ppa/xx-delete-packages.txt b/lib/lp/soyuz/stories/ppa/xx-delete-packages.txt
index da6db24..e16bb33 100644
--- a/lib/lp/soyuz/stories/ppa/xx-delete-packages.txt
+++ b/lib/lp/soyuz/stories/ppa/xx-delete-packages.txt
@@ -546,4 +546,4 @@ as it would refer to non-existent links.
     Published on 2006-12-01
     Copied from ubuntu breezy-autotest in Primary Archive for Ubuntu Linux
     ...
-    &laquo;back
+    «back
diff --git a/lib/lp/soyuz/stories/ppa/xx-ppa-workflow.txt b/lib/lp/soyuz/stories/ppa/xx-ppa-workflow.txt
index 95e5e11..c774c59 100644
--- a/lib/lp/soyuz/stories/ppa/xx-ppa-workflow.txt
+++ b/lib/lp/soyuz/stories/ppa/xx-ppa-workflow.txt
@@ -425,7 +425,7 @@ a sources.list entry.  If the field fails validation an error is displayed.
     >>> admin_browser.getControl("Save").click()
     >>> print_feedback_messages(admin_browser.contents)
     There is 1 error.
-    &#x27;deb not_a_url&#x27; is not a complete and valid sources.list entry
+    'deb not_a_url' is not a complete and valid sources.list entry
 
 
 There is a maximum value allowed for `IArchive.authorized_size`, it is
@@ -520,7 +520,7 @@ and the rendered form contains the 'name' field.
 
     >>> print_feedback_messages(browser2.contents)
     There is 1 error.
-    You already have a PPA for Ubuntu named &#x27;boomppa&#x27;.
+    You already have a PPA for Ubuntu named 'boomppa'.
 
     >>> print(browser2.getControl(name="field.name").value)
     boomppa
@@ -605,7 +605,7 @@ If Celso, by mistake, uses the same name of one of his existing PPAs
 
     >>> print_feedback_messages(cprov_browser.contents)
     There is 1 error.
-    You already have a PPA for Ubuntu named &#x27;ppa&#x27;.
+    You already have a PPA for Ubuntu named 'ppa'.
 
 If the PPA is named as the distribution it is targeted for it cannot
 be created, mainly because of the way we publish repositories
@@ -794,7 +794,7 @@ contain a notification message that the deletion is in progress.
     http://launchpad.test/~no-priv
 
     >>> print_feedback_messages(no_priv_browser.contents)
-    Deletion of &#x27;PPA for No Privileges Person&#x27; has been
+    Deletion of 'PPA for No Privileges Person' has been
     requested and the repository will be removed shortly.
 
 The deleted PPA is still available to browse via a link on the profile page
diff --git a/lib/lp/soyuz/stories/ppa/xx-private-ppa-subscription-stories.txt b/lib/lp/soyuz/stories/ppa/xx-private-ppa-subscription-stories.txt
index 48e6d5c..cfd14a5 100644
--- a/lib/lp/soyuz/stories/ppa/xx-private-ppa-subscription-stories.txt
+++ b/lib/lp/soyuz/stories/ppa/xx-private-ppa-subscription-stories.txt
@@ -181,7 +181,7 @@ cancellation is displayed.
     Joe Smith                             Joe is my friend    Edit/Cancel
 
     >>> print_feedback_messages(cprov_browser.contents)
-    You have revoked Launchpad Developers&#x27;s access to PPA
+    You have revoked Launchpad Developers's access to PPA
     named p3a for Celso Providelo.
 
 
@@ -259,8 +259,7 @@ the page is redisplayed with new sources.list entries and a notification.
     >>> print_feedback_messages(joe_browser.contents)
     Launchpad has generated the new password you requested for your
     access to the archive PPA named p3a for Celso Providelo. Please
-    follow the instructions below to update your custom
-    &quot;sources.list&quot;.
+    follow the instructions below to update your custom "sources.list".
 
 
 === Scenario 3: A user activates a team subscription ===
diff --git a/lib/lp/soyuz/stories/soyuz/xx-binarypackagerelease-index.txt b/lib/lp/soyuz/stories/soyuz/xx-binarypackagerelease-index.txt
index c267754..1bd9bdb 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-binarypackagerelease-index.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-binarypackagerelease-index.txt
@@ -84,11 +84,11 @@ sections contain only unsatisfied dependencies, which are rendered as
 text:
 
   >>> print_relation('depends')
-  TEXT: "gcc-3.4 (&gt;= 3.4.1-4sarge1)"
-  TEXT: "gcc-3.4 (&lt;&lt; 3.4.2)"
+  TEXT: "gcc-3.4 (>= 3.4.1-4sarge1)"
+  TEXT: "gcc-3.4 (<< 3.4.2)"
   TEXT: "gcc-3.4-base"
-  TEXT: "libc6 (&gt;= 2.3.2.ds1-4)"
-  TEXT: "libstdc++6-dev (&gt;= 3.4.1-4sarge1)"
+  TEXT: "libc6 (>= 2.3.2.ds1-4)"
+  TEXT: "libstdc++6-dev (>= 3.4.1-4sarge1)"
 
   >>> print_relation('conflicts')
   TEXT: "firefox"
@@ -103,11 +103,11 @@ text:
   TEXT: "gnome-mozilla-browser"
 
   >>> print_relation('recommends')
-  TEXT: "gcc-3.4 (&gt;= 3.4.1-4sarge1)"
-  TEXT: "gcc-3.4 (&lt;&lt; 3.4.2)"
+  TEXT: "gcc-3.4 (>= 3.4.1-4sarge1)"
+  TEXT: "gcc-3.4 (<< 3.4.2)"
   TEXT: "gcc-3.4-base"
-  TEXT: "libc6 (&gt;= 2.3.2.ds1-4)"
-  TEXT: "libstdc++6-dev (&gt;= 3.4.1-4sarge1)"
+  TEXT: "libc6 (>= 2.3.2.ds1-4)"
+  TEXT: "libstdc++6-dev (>= 3.4.1-4sarge1)"
 
 Even when there is no information to present and the package control
 files don't contain the field, we still present the  corresponding
diff --git a/lib/lp/soyuz/stories/soyuz/xx-builds-pages.txt b/lib/lp/soyuz/stories/soyuz/xx-builds-pages.txt
index da2791d..4f062a9 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-builds-pages.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-builds-pages.txt
@@ -134,7 +134,7 @@ initialized' one (fix bug #52704).
     ...     for row in build_rows:
     ...         print(rule)
     ...         print(row.td.img['title'])
-    ...         print(extract_text(row))
+    ...         print(extract_text(row, formatter='html'))
     ...     print(rule)
 
     >>> print_build_rows(anon_browser.contents)
diff --git a/lib/lp/soyuz/stories/soyuz/xx-distribution-archives.txt b/lib/lp/soyuz/stories/soyuz/xx-distribution-archives.txt
index 56447e2..e8940fd 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-distribution-archives.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-distribution-archives.txt
@@ -100,7 +100,7 @@ present:
 
     >>> anon_browser.open("http://launchpad.test/~cprov/+archive";)
     >>> print(find_tag_by_id(anon_browser.contents, 'sources-list-entries'))
-    <pre id="sources-list-entries"...
+    <pre ...id="sources-list-entries"...
     ...
     deb-src ...</pre>
 
@@ -177,7 +177,7 @@ is disabled.
     Copy archive disabled-security-rebuild for No Privileges Person : Ubuntu
 
     >>> main_content = find_main_content(nopriv_browser.contents)
-    >>> print(main_content.h1['class'])
+    >>> print(' '.join(main_content.h1['class']))
     disabled
 
     >>> tag = first_tag_by_class(nopriv_browser.contents, 'warning message')
diff --git a/lib/lp/soyuz/stories/soyuz/xx-distributionsourcepackagerelease-pages.txt b/lib/lp/soyuz/stories/soyuz/xx-distributionsourcepackagerelease-pages.txt
index fee55a5..8c49cb1 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-distributionsourcepackagerelease-pages.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-distributionsourcepackagerelease-pages.txt
@@ -172,7 +172,7 @@ of this source in the context distribution are listed.
     Date  Status     Target     Pocket   Component  Section  Version
     ...   Published  Breezy...  release  main       base     1.0
     ...
-    &laquo;back
+    «back
 
     >>> anon_browser.getLink('back').click()
 
diff --git a/lib/lp/soyuz/stories/soyuz/xx-distroarchseries.txt b/lib/lp/soyuz/stories/soyuz/xx-distroarchseries.txt
index ec99340..8752daa 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-distroarchseries.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-distroarchseries.txt
@@ -52,7 +52,8 @@ package summary.
 
 More details are available by clicking on the binary package name.
     >>> print(extract_text(
-    ...     find_tag_by_id(anon_browser.contents, 'search-results')))
+    ...     find_tag_by_id(anon_browser.contents, 'search-results'),
+    ...     formatter='html'))
     1 &rarr; 3 of 3 results
     First &bull; Previous &bull; Next &bull; Last
     mozilla-firefox: Mozilla Firefox Web Browser
diff --git a/lib/lp/soyuz/stories/soyuz/xx-distroseries-index.txt b/lib/lp/soyuz/stories/soyuz/xx-distroseries-index.txt
index 43c4284..2e9f391 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-distroseries-index.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-distroseries-index.txt
@@ -10,7 +10,8 @@ packages within the distroseries in context:
     >>> anon_browser.url
     'http://launchpad.test/ubuntu/warty/+search?text=a'
     >>> print(extract_text(
-    ...     find_tag_by_id(anon_browser.contents, 'search-results')))
+    ...     find_tag_by_id(anon_browser.contents, 'search-results'),
+    ...     formatter='html'))
     1 &rarr; 3 of 3 results
     First &bull; Previous &bull; Next &bull; Last
     foobar: foobar is bad
diff --git a/lib/lp/soyuz/stories/soyuz/xx-distroseries-sources.txt b/lib/lp/soyuz/stories/soyuz/xx-distroseries-sources.txt
index 8ed6d44..0769ed2 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-distroseries-sources.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-distroseries-sources.txt
@@ -158,11 +158,11 @@ Let's inspect a page with non-empty relationships.
 
   >>> depends_section = find_tag_by_id(browser.contents, 'depends')
   >>> parse_relationship_section(str(depends_section))
-  TEXT: "gcc-3.4 (&gt;= 3.4.1-4sarge1)"
-  TEXT: "gcc-3.4 (&lt;&lt; 3.4.2)"
+  TEXT: "gcc-3.4 (>= 3.4.1-4sarge1)"
+  TEXT: "gcc-3.4 (<< 3.4.2)"
   TEXT: "gcc-3.4-base"
-  TEXT: "libc6 (&gt;= 2.3.2.ds1-4)"
-  TEXT: "libstdc++6-dev (&gt;= 3.4.1-4sarge1)"
+  TEXT: "libc6 (>= 2.3.2.ds1-4)"
+  TEXT: "libstdc++6-dev (>= 3.4.1-4sarge1)"
   LINK: "pmount" -> http://launchpad.test/ubuntu/warty/+package/pmount
 
   >>> dependsindep_section = find_tag_by_id(browser.contents, 'dependsindep')
@@ -170,7 +170,7 @@ Let's inspect a page with non-empty relationships.
   TEXT: "bacula-common (= 1.34.6-2)"
   TEXT: "bacula-director-common (= 1.34.6-2)"
   LINK: "pmount" -> http://launchpad.test/ubuntu/warty/+package/pmount
-  TEXT: "postgresql-client (&gt;= 7.4)"
+  TEXT: "postgresql-client (>= 7.4)"
 
   >>> dependsarch_section = find_tag_by_id(browser.contents, 'dependsarch')
   >>> parse_relationship_section(str(dependsarch_section))
diff --git a/lib/lp/soyuz/stories/soyuz/xx-queue-pages-motu.txt b/lib/lp/soyuz/stories/soyuz/xx-queue-pages-motu.txt
index 6f2f702..6824527 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-queue-pages-motu.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-queue-pages-motu.txt
@@ -54,7 +54,7 @@ not have permission to accept items in "main":
     >>> motu_browser.getControl(name="QUEUE_ID").value = ['4']
     >>> motu_browser.getControl(name="Accept").click()
     >>> print_feedback_messages(motu_browser.contents)
-    FAILED: alsa-utils (You have no rights to accept component(s) &#x27;main&#x27;)
+    FAILED: alsa-utils (You have no rights to accept component(s) 'main')
 
 The same applies to the binary upload "pmount" because its build
 produced a package in main:
@@ -62,7 +62,7 @@ produced a package in main:
     >>> motu_browser.getControl(name="QUEUE_ID").value=['2']
     >>> motu_browser.getControl(name="Accept").click()
     >>> print_feedback_messages(motu_browser.contents)
-    FAILED: pmount (You have no rights to accept component(s) &#x27;main&#x27;)
+    FAILED: pmount (You have no rights to accept component(s) 'main')
 
 Let's change the components on some uploads so that the user has
 permission to manipulate them.
@@ -147,5 +147,5 @@ will be left alone.
     ...     name="component_override").displayValue = ["multiverse"]
     >>> motu_browser.getControl(name="Accept").click()
     >>> print_feedback_messages(motu_browser.contents)
-    FAILED: netapplet (You have no rights to accept component(s) &#x27;main&#x27;)
+    FAILED: netapplet (You have no rights to accept component(s) 'main')
     OK: mozilla-firefox(multiverse/(unchanged)/(unchanged))
diff --git a/lib/lp/soyuz/stories/soyuz/xx-queue-pages.txt b/lib/lp/soyuz/stories/soyuz/xx-queue-pages.txt
index 7417a7b..c5f4741 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-queue-pages.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-queue-pages.txt
@@ -209,7 +209,8 @@ image:
     >>> anon_browser.getControl(name="queue_text").value = ''
     >>> anon_browser.getControl("Update").click()
 
-    >>> print(find_tag_by_id(anon_browser.contents, 'queue-4-icon'))
+    >>> print(find_tag_by_id(anon_browser.contents, 'queue-4-icon').decode(
+    ...     formatter='html'))
     <span class="expander-link" id="queue-4-icon">&nbsp;</span>
 
 The 'filelist' is expanded as one or more table rows, right below the
diff --git a/lib/lp/soyuz/stories/soyuz/xx-sourcepackage-changelog.txt b/lib/lp/soyuz/stories/soyuz/xx-sourcepackage-changelog.txt
index c2a91c3..c4ff5b4 100644
--- a/lib/lp/soyuz/stories/soyuz/xx-sourcepackage-changelog.txt
+++ b/lib/lp/soyuz/stories/soyuz/xx-sourcepackage-changelog.txt
@@ -104,8 +104,8 @@ although the version link will point to the same page we're already on.
     <pre ... id="alsa-utils_1.0.9a-4ubuntu1">alsa-utils (1.0.9a-4ubuntu1) ...
     <BLANKLINE>
     ...
-     LP: <a href="/bugs/10" class="bug-link">#10</a>
-     LP: <a href="/bugs/999" class="bug-link">#999</a>
+     LP: <a class="bug-link" href="/bugs/10">#10</a>
+     LP: <a class="bug-link" href="/bugs/999">#999</a>
      LP: #badid
     ...
 
diff --git a/lib/lp/testing/pages.py b/lib/lp/testing/pages.py
index 1c23980..facbfc1 100644
--- a/lib/lp/testing/pages.py
+++ b/lib/lp/testing/pages.py
@@ -75,8 +75,8 @@ from lp.app.interfaces.launchpad import ILaunchpadCelebrities
 from lp.registry.errors import NameAlreadyTaken
 from lp.registry.interfaces.teammembership import TeamMembershipStatus
 from lp.services.beautifulsoup import (
-    BeautifulSoup,
-    SoupStrainer,
+    BeautifulSoup4 as BeautifulSoup,
+    SoupStrainer4 as SoupStrainer,
     )
 from lp.services.config import config
 from lp.services.encoding import wsgi_native_string
@@ -253,7 +253,7 @@ def find_tag_by_id(content, id):
     else:
         elements_with_id = [
             tag for tag in BeautifulSoup(
-                content, parseOnlyThese=SoupStrainer(id=id))]
+                content, parse_only=SoupStrainer(id=id))]
     if len(elements_with_id) == 0:
         return None
     elif len(elements_with_id) == 1:
@@ -279,7 +279,7 @@ def find_tags_by_class(content, class_, only_first=False):
         classes = set(value.split())
         return match_classes.issubset(classes)
     soup = BeautifulSoup(
-        content, parseOnlyThese=SoupStrainer(attrs={'class': class_matcher}))
+        content, parse_only=SoupStrainer(attrs={'class': class_matcher}))
     if only_first:
         find = BeautifulSoup.find
     else:
@@ -323,7 +323,7 @@ def get_feedback_messages(content):
                        'warning message']
     soup = BeautifulSoup(
         content,
-        parseOnlyThese=SoupStrainer(['div', 'p'], {'class': message_classes}))
+        parse_only=SoupStrainer(['div', 'p'], {'class': message_classes}))
     return [extract_text(tag) for tag in soup]
 
 
@@ -466,6 +466,8 @@ def extract_text(content, extract_image_text=False, skip_tags=None,
             # unicode() call here, but we keep it for consistency and clarity
             # purposes.
             result.append(unicode(node[:]))
+        elif isinstance(node, CData4):
+            result.append(unicode(node))
         elif isinstance(node, NavigableString):
             result.append(unicode(node))
         elif isinstance(node, NavigableString4):
diff --git a/lib/lp/translations/stories/importqueue/xx-translation-import-queue.txt b/lib/lp/translations/stories/importqueue/xx-translation-import-queue.txt
index f22031e..4ba2c4b 100644
--- a/lib/lp/translations/stories/importqueue/xx-translation-import-queue.txt
+++ b/lib/lp/translations/stories/importqueue/xx-translation-import-queue.txt
@@ -128,7 +128,7 @@ the last entry.
   >>> anon_browser.open('http://translations.launchpad.test/+imports')
   >>> nav_index = first_tag_by_class(anon_browser.contents,
   ...     'batch-navigation-index')
-  >>> print(extract_text(nav_index))
+  >>> print(extract_text(nav_index, formatter='html'))
   1 &rarr; 5 of 5 results
   >>> rows = find_tags_by_class(anon_browser.contents, 'import_entry_row')
   >>> print(extract_text(rows[4]))
@@ -367,7 +367,7 @@ decent error message:
   >>> browser.getControl('Upload').click()
   >>> for tag in find_tags_by_class(browser.contents, 'message'):
   ...     print(tag)
-  <div...Your upload was ignored because you didn&#x27;t select a file....
+  <div...Your upload was ignored because you didn't select a file....
   ...Please select a file and try again.</div>...
 
 Let's try now a tarball upload. Should work:
diff --git a/lib/lp/translations/stories/productseries/xx-productseries-translations-bzr-request.txt b/lib/lp/translations/stories/productseries/xx-productseries-translations-bzr-request.txt
index f3f90b9..fd04b4a 100644
--- a/lib/lp/translations/stories/productseries/xx-productseries-translations-bzr-request.txt
+++ b/lib/lp/translations/stories/productseries/xx-productseries-translations-bzr-request.txt
@@ -54,7 +54,7 @@ The request is made by clicking on a button labeled
     >>> request_button = find_tag_by_id(
     ...     browser.contents, 'field.actions.request_import')
     >>> print(request_button)
-    <input type="submit"...value="Request one-time import"...
+    <input ...type="submit"...value="Request one-time import"...
     >>> browser.getControl("Request one-time import").click()
     >>> print(browser.url)
     http://translations.launchpad.test/evolution/trunk
diff --git a/lib/lp/translations/stories/standalone/custom-language-codes.txt b/lib/lp/translations/stories/standalone/custom-language-codes.txt
index ca52502..6132599 100644
--- a/lib/lp/translations/stories/standalone/custom-language-codes.txt
+++ b/lib/lp/translations/stories/standalone/custom-language-codes.txt
@@ -98,10 +98,10 @@ to see there.
 
     >>> owner_browser.getLink("no").click()
     >>> main = find_main_content(owner_browser.contents)
-    >>> print(extract_text(main.renderContents()))
+    >>> print(extract_text(main.decode_contents()))
     Custom language code  ...no... for Foo
     For Foo, uploads with the language code
-    &ldquo;no&rdquo;
+    “no”
     are associated with the language
     Norwegian Nynorsk.
     remove custom language code
diff --git a/lib/lp/translations/stories/standalone/xx-person-editlanguages.txt b/lib/lp/translations/stories/standalone/xx-person-editlanguages.txt
index 9d65a89..9b6e020 100644
--- a/lib/lp/translations/stories/standalone/xx-person-editlanguages.txt
+++ b/lib/lp/translations/stories/standalone/xx-person-editlanguages.txt
@@ -19,7 +19,7 @@ lists his current preferences.
     Your preferred languages
     Catalan
     Spanish
-    &... Change your preferred languages...
+    » Change your preferred languages...
 
 The 'Change your preferred languages' link takes him to his
 +editlanguages page.
@@ -55,15 +55,15 @@ confirming his changes.
 
     >>> for message in find_tags_by_class(browser.contents, 'message'):
     ...     print(message.renderContents())
-    Added Welsh to your preferred languages.<br
-    />Removed Spanish from your preferred languages.
+    Added Welsh to your preferred languages.<br/>Removed Spanish from your
+    preferred languages.
 
     >>> browser.open('http://translations.launchpad.test/')
     >>> print(find_languages_section(browser.contents))
     Your preferred languages
     Catalan
     Welsh
-    &...
+    » ...
 
 Joao, a Brazilian, travels to Liechtenstein for business and views the
 languages page from there.
@@ -161,7 +161,7 @@ preferred languages.
     >>> browser.getControl('Save').click()
     >>> for message in find_tags_by_class(browser.contents, 'message'):
     ...     print(extract_text(message))
-    Added Spanish to Landscape Developers&#x27;s preferred languages.
+    Added Spanish to Landscape Developers's preferred languages.
 
 
 == Admins may set a Person's languages ==
@@ -183,7 +183,7 @@ Person's page to do it themselves.
     >>> admin_browser.getControl('Save').click()
     >>> for message in find_tags_by_class(admin_browser.contents, 'message'):
     ...     print(extract_text(message))
-    Added Esperanto to No Privileges Person&#x27;s preferred languages.
+    Added Esperanto to No Privileges Person's preferred languages.
 
 
 The personal page nags
diff --git a/lib/lp/translations/stories/standalone/xx-pofile-details.txt b/lib/lp/translations/stories/standalone/xx-pofile-details.txt
index f5c97d8..888e2fb 100644
--- a/lib/lp/translations/stories/standalone/xx-pofile-details.txt
+++ b/lib/lp/translations/stories/standalone/xx-pofile-details.txt
@@ -39,7 +39,7 @@ a separate method to pretty-print all the translations.
     ...         for cell in cells:
     ...             type = dict(cell.attrs).get('class')
     ...             if type:
-    ...                 return type
+    ...                 return ' '.join(type)
     ...         return None
     ...
     ...     # Get contents of all the `cells`, and join them with spaces.
diff --git a/lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt b/lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt
index fc97599..070d946 100644
--- a/lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt
+++ b/lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt
@@ -180,7 +180,7 @@ Carlos can see that he is viewing the first page of results in the
 navigation bar between the translation controls and the messages.
 
     >>> navigation = find_tags_by_class(browser.contents, 'results')[0].td
-    >>> print(extract_text(navigation).decode('utf-8'))
+    >>> print(extract_text(navigation))
     1 ... 10  of 15 results ...
 
 Carlos uses the 'Save & Continue' button to see the next page of
diff --git a/lib/lp/translations/stories/standalone/xx-pofile-translate-gettext-error-middle-page.txt b/lib/lp/translations/stories/standalone/xx-pofile-translate-gettext-error-middle-page.txt
index 807d054..574be01 100644
--- a/lib/lp/translations/stories/standalone/xx-pofile-translate-gettext-error-middle-page.txt
+++ b/lib/lp/translations/stories/standalone/xx-pofile-translate-gettext-error-middle-page.txt
@@ -50,8 +50,8 @@ And the error is noted in the page.
     <th colspan="3">
       <strong>Error in Translation:</strong>
     </th>
+    <td></td>
     <td>
-    </td><td>
       <div>
         format specifications in 'msgid' and 'msgstr' for argument 1 are not the same
       </div>
diff --git a/lib/lp/translations/stories/standalone/xx-pofile-translate-lang-direction.txt b/lib/lp/translations/stories/standalone/xx-pofile-translate-lang-direction.txt
index cf3e9e4..0461004 100644
--- a/lib/lp/translations/stories/standalone/xx-pofile-translate-lang-direction.txt
+++ b/lib/lp/translations/stories/standalone/xx-pofile-translate-lang-direction.txt
@@ -48,7 +48,6 @@ direction:
 
   >>> print(find_tag_by_id(
   ...     browser.contents, 'msgset_130_es_suggestion_562_0'))
-  <label style="white-space: normal" dir="ltr"
-    for="msgset_130_es_suggestion_562_0_radiobutton"
-    id="msgset_130_es_suggestion_562_0" lang="es">libreta de
-      direcciones de Evolution</label>
+  <label dir="ltr" for="msgset_130_es_suggestion_562_0_radiobutton"
+    id="msgset_130_es_suggestion_562_0" lang="es"
+    style="white-space: normal">libreta de direcciones de Evolution</label>
diff --git a/lib/lp/translations/stories/standalone/xx-pofile-translate-newlines-check.txt b/lib/lp/translations/stories/standalone/xx-pofile-translate-newlines-check.txt
index f381209..d0f8190 100644
--- a/lib/lp/translations/stories/standalone/xx-pofile-translate-newlines-check.txt
+++ b/lib/lp/translations/stories/standalone/xx-pofile-translate-newlines-check.txt
@@ -12,7 +12,7 @@ https://bugzilla.mozilla.org/show_bug.cgi?id=299009
     ...     """Find and print [tags] in browser.contents. End each with '--'."""
     ...     soup = find_main_content(browser.contents)
     ...     for tag in soup.findAll(attrs={'id': tags}):
-    ...         print("%s\n--\n" % tag.renderContents(encoding=None))
+    ...         print(u"%s\n--\n" % tag.decode_contents())
 
     >>> browser = setupBrowser(auth='Basic carlos@xxxxxxxxxxxxx:test')
     >>> browser.open(
@@ -24,7 +24,7 @@ We can see that the message we are interested in is not translated.
     >>> print_tags(browser,
     ...     ['msgset_149', 'msgset_149_singular', 'msgset_149_es_translation_0'])
     20.
-    <input type="hidden" name="msgset_149" />
+    <input name="msgset_149" type="hidden"/>
     --
     Please select a key size in bits.  The cipher you have chosen...
     --
@@ -94,11 +94,11 @@ And, as we can see, we get the trailing new line char
     >>> print_tags(browser, [
     ...     'msgset_165', 'msgset_165_singular', 'msgset_165_es_translation_0'])
     23.
-    <input type="hidden" name="msgset_165" />
+    <input name="msgset_165" type="hidden"/>
     --
-    <code>%s</code>: option `<code>%s</code>&#x27; is ambiguous...
+    <code>%s</code>: option `<code>%s</code>' is ambiguous...
     --
-    <code>%s</code>: la opcion «<code>%s</code>» es ambigua<img alt="" src="/@@/translation-newline" /><br />
+    <code>%s</code>: la opcion «<code>%s</code>» es ambigua<img alt="" src="/@@/translation-newline"/><br/>
     --
 
 Now, we do the right submit, with one trailing new line...
@@ -124,11 +124,11 @@ And, as we can see, we get the same output, just one trailing newline char.
     >>> print_tags(browser, [
     ...     'msgset_165', 'msgset_165_singular', 'msgset_165_es_translation_0'])
     23.
-    <input type="hidden" name="msgset_165" />
+    <input name="msgset_165" type="hidden"/>
     --
-    <code>%s</code>: option `<code>%s</code>&#x27; is ambiguous...
+    <code>%s</code>: option `<code>%s</code>' is ambiguous...
     --
-    <code>%s</code>: la opcion «<code>%s</code>» es ambigua<img alt="" src="/@@/translation-newline" /><br />
+    <code>%s</code>: la opcion «<code>%s</code>» es ambigua<img alt="" src="/@@/translation-newline"/><br/>
     --
 
 Last check, the user sends two new line chars instead of just one...
@@ -154,9 +154,9 @@ And Launchpad comes to the rescue and stores just one!
     >>> print_tags(browser, [
     ...     'msgset_165', 'msgset_165_singular', 'msgset_165_es_translation_0'])
     23.
-    <input type="hidden" name="msgset_165" />
+    <input name="msgset_165" type="hidden"/>
     --
-    <code>%s</code>: option `<code>%s</code>&#x27; is ambiguous...
+    <code>%s</code>: option `<code>%s</code>' is ambiguous...
     --
-    <code>%s</code>: la opcion «<code>%s</code>» es ambigua<img alt="" src="/@@/translation-newline" /><br />
+    <code>%s</code>: la opcion «<code>%s</code>» es ambigua<img alt="" src="/@@/translation-newline"/><br/>
     --
diff --git a/lib/lp/translations/stories/standalone/xx-pofile-translate-private-issues.txt b/lib/lp/translations/stories/standalone/xx-pofile-translate-private-issues.txt
index e434260..2963894 100644
--- a/lib/lp/translations/stories/standalone/xx-pofile-translate-private-issues.txt
+++ b/lib/lp/translations/stories/standalone/xx-pofile-translate-private-issues.txt
@@ -17,21 +17,21 @@ The GNOME standard string for credits is well handled.
   ...     browser.contents, 'msgset_199_es_translation_0')
   >>> print(translation.renderContents())
   To prevent privacy issues, this translation is not available to anonymous
-  users,<br /> if you want to see it, please, <a href="+login">log in</a>
+  users,<br/> if you want to see it, please, <a href="+login">log in</a>
   first.
 
 And the same for KDE one.
 
   >>> msgid = find_tag_by_id(browser.contents, 'msgset_200_singular')
   >>> print(msgid.renderContents())
-  _: EMAIL OF TRANSLATORS<img alt="" src="/@@/translation-newline" /><br />
+  _: EMAIL OF TRANSLATORS<img alt="" src="/@@/translation-newline"/><br/>
   Your emails
 
   >>> translation = find_tag_by_id(
   ...     browser.contents, 'msgset_200_es_translation_0')
   >>> print(translation.renderContents())
   To prevent privacy issues, this translation is not available to anonymous
-  users,<br /> if you want to see it, please, <a href="+login">log in</a>
+  users,<br/> if you want to see it, please, <a href="+login">log in</a>
   first.
 
 Also, suggestions should not appear.
@@ -63,7 +63,7 @@ And the same for KDE one.
 
   >>> msgid = find_tag_by_id(user_browser.contents, 'msgset_200_singular')
   >>> print(msgid.renderContents())
-  _: EMAIL OF TRANSLATORS<img alt="" src="/@@/translation-newline" /><br />
+  _: EMAIL OF TRANSLATORS<img alt="" src="/@@/translation-newline"/><br/>
   Your emails
 
   >>> translation = find_tag_by_id(
diff --git a/lib/lp/translations/stories/standalone/xx-pofile-translate.txt b/lib/lp/translations/stories/standalone/xx-pofile-translate.txt
index 4a790e7..ccb3921 100644
--- a/lib/lp/translations/stories/standalone/xx-pofile-translate.txt
+++ b/lib/lp/translations/stories/standalone/xx-pofile-translate.txt
@@ -230,9 +230,9 @@ single message.
     >>> browser.open('http://translations.launchpad.test/ubuntu/hoary/'
     ...     '+source/evolution/+pots/evolution-2.2/ab/+translate')
     >>> print_feedback_messages(browser.contents)
-    Launchpad can&#8217;t handle the plural items ...
+    Launchpad can’t handle the plural items ...
 
     >>> browser.open('http://translations.launchpad.test/ubuntu/hoary/'
     ...     '+source/evolution/+pots/evolution-2.2/ab/5/+translate')
     >>> print_feedback_messages(browser.contents)
-    Launchpad can&#8217;t handle the plural items ...
+    Launchpad can’t handle the plural items ...
diff --git a/lib/lp/translations/stories/standalone/xx-potemplate-index.txt b/lib/lp/translations/stories/standalone/xx-potemplate-index.txt
index 6bb3973..8bf48de 100644
--- a/lib/lp/translations/stories/standalone/xx-potemplate-index.txt
+++ b/lib/lp/translations/stories/standalone/xx-potemplate-index.txt
@@ -40,7 +40,7 @@ that represents, when the translation was updated, and by whom.
 
     >>> table = content.findAll('table')[0]
     >>> for row in table.findAll('tr'):
-    ...     print(extract_text(row).encode('us-ascii', 'backslashreplace'))
+    ...     print(extract_text(row, formatter='html'))
     Language        Status  Untranslated Need review Changed Last    Edited By
     Afrikaans               22           ...         ...     &mdash; &mdash;
     Japanese                21           ...         ...     ...     Carlos...
@@ -59,13 +59,13 @@ English translations, but they are not displayed to the user.
     ...     '+source/mozilla/+pots/pkgconf-mozilla')
     >>> table = find_tag_by_id(anon_browser.contents, 'language-chart')
     >>> for row in table.findAll('tr')[0:6]:
-    ...     print(extract_text(row).encode('us-ascii', 'backslashreplace'))
+    ...     print(extract_text(row, formatter='html'))
     Language    Status  Untranslated Need review Changed Last  Edited By
     Afrikaans             9            ...         ...   ...   &mdash;
     Czech                 ...          ...         ...   ...   Miroslav Kure
     Danish                ...          ...         ...   ...   Morten Brix...
     Dutch                 ...          ...         ...   ...   Luk Claes
-    Finnish               ...          ...         ...   ...   P\xf6ll\xe4
+    Finnish               ...          ...         ...   ...   P&ouml;ll&auml;
 
 
 Sharing information
diff --git a/lib/lp/translations/stories/standalone/xx-rosetta-homepage.txt b/lib/lp/translations/stories/standalone/xx-rosetta-homepage.txt
index 6ac2da2..7e940c7 100644
--- a/lib/lp/translations/stories/standalone/xx-rosetta-homepage.txt
+++ b/lib/lp/translations/stories/standalone/xx-rosetta-homepage.txt
@@ -47,7 +47,7 @@ translated projects.
     ...     print(extract_text(project))
     Evolution
     >>> print(extract_text(middle_column.findAll('div')[0]))
-    &raquo; List all translatable projects...
+    » List all translatable projects...
 
 The translation front page list of the user's translatable languages.
 in the right column.
diff --git a/lib/lp/translations/stories/standalone/xx-rosetta-sourcepackage-list.txt b/lib/lp/translations/stories/standalone/xx-rosetta-sourcepackage-list.txt
index 595d78f..e4f442c 100644
--- a/lib/lp/translations/stories/standalone/xx-rosetta-sourcepackage-list.txt
+++ b/lib/lp/translations/stories/standalone/xx-rosetta-sourcepackage-list.txt
@@ -37,7 +37,7 @@ indicate that these languages have never been edited my anyone.
 
     >>> table = content.findAll('table')[0]
     >>> for row in table.findAll('tr'):
-    ...     print(extract_text(row))
+    ...     print(extract_text(row, formatter='html'))
     Language        Status Untranslated Need review Changed Last    Edited By
     Afrikaans              22             ...         ...   &mdash; &mdash;
     Sotho, Southern        22             ...         ...   &mdash; &mdash;
diff --git a/lib/lp/translations/stories/standalone/xx-template-description-escaping.txt b/lib/lp/translations/stories/standalone/xx-template-description-escaping.txt
index 006cffc..b7daddf 100644
--- a/lib/lp/translations/stories/standalone/xx-template-description-escaping.txt
+++ b/lib/lp/translations/stories/standalone/xx-template-description-escaping.txt
@@ -32,7 +32,7 @@ The template's description is linkified, so the URL is clickable.
     >>> len(description)
     1
     >>> print(description[0])
-    See <a ... href="http://example.com/";>...</a> for an example!
+    See <a href="http://example.com/"; ...>...</a> for an example!
 
 The same description also shows up on the +translate page for the
 package.  Again,the URL is clickable.
@@ -41,4 +41,4 @@ package.  Again,the URL is clickable.
     >>> len(description)
     1
     >>> print(description[0])
-    See <a ... href="http://example.com/";>...</a> for an example!
+    See <a href="http://example.com/"; ...>...</a> for an example!
diff --git a/lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt b/lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt
index dda582b..e7fb942 100644
--- a/lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt
+++ b/lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt
@@ -192,11 +192,11 @@ First what we represent in the form when there is no translation:
 
     >>> print(find_tag_by_id(browser.contents, 'msgset_142').renderContents())
     13.
-    <input type="hidden" name="msgset_142" />
+    <input name="msgset_142" type="hidden"/>
 
     >>> print(find_tag_by_id(
     ...     browser.contents, 'msgset_142_singular').renderContents())
-    Migrating `<code>%s</code>&#x27;:
+    Migrating `<code>%s</code>':
 
     >>> print(find_tag_by_id(
     ...     browser.contents, 'msgset_142_es_translation_0').renderContents())
@@ -233,8 +233,8 @@ Let's submit an invalid value for this message #13.
       <th colspan="3">
         <strong>Error in Translation:</strong>
       </th>
+      <td></td>
       <td>
-      </td><td>
         <div>
           format specifications in 'msgid' and 'msgstr' for argument 1 are not
           the same
@@ -246,11 +246,11 @@ The message is still without translation:
 
     >>> print(find_tag_by_id(browser.contents, 'msgset_142').renderContents())
     13.
-    <input type="hidden" name="msgset_142" />
+    <input name="msgset_142" type="hidden"/>
 
     >>> print(find_tag_by_id(
     ...     browser.contents, 'msgset_142_singular').renderContents())
-    Migrating `<code>%s</code>&#x27;:
+    Migrating `<code>%s</code>':
 
     >>> print(find_tag_by_id(
     ...     browser.contents, 'msgset_142_es_translation_0').renderContents())
@@ -280,11 +280,11 @@ Check that the message #13 has the new value we submitted.
 
     >>> print(find_tag_by_id(browser.contents, 'msgset_142').renderContents())
     13.
-    <input type="hidden" name="msgset_142" />
+    <input name="msgset_142" type="hidden"/>
 
     >>> print(find_tag_by_id(
     ...     browser.contents, 'msgset_142_singular').renderContents())
-    Migrating `<code>%s</code>&#x27;:
+    Migrating `<code>%s</code>':
 
     >>> print(find_tag_by_id(
     ...     browser.contents, 'msgset_142_es_translation_0').renderContents())
@@ -428,8 +428,8 @@ We submit it
       <th colspan="3">
         <strong>Error in Translation:</strong>
       </th>
+      <td></td>
       <td>
-      </td><td>
         <div>
           This translation has changed since you last saw it.  To avoid
           accidentally reverting work done by others, we added your
@@ -443,7 +443,7 @@ Also, we should still have previous translation:
     >>> print(find_tag_by_id(
     ...     slow_submission.contents, 'msgset_143').renderContents())
     14.
-    <input type="hidden" name="msgset_143" />
+    <input name="msgset_143" type="hidden"/>
 
     >>> print(find_tag_by_id(
     ...     slow_submission.contents, 'msgset_143_singular').renderContents())
diff --git a/lib/lp/translations/stories/translationgroups/xx-translationgroups.txt b/lib/lp/translations/stories/translationgroups/xx-translationgroups.txt
index ceaba7d..f4b2b55 100644
--- a/lib/lp/translations/stories/translationgroups/xx-translationgroups.txt
+++ b/lib/lp/translations/stories/translationgroups/xx-translationgroups.txt
@@ -804,7 +804,7 @@ added (with some extra html code, but the same content we wanted to add)
 
     >>> tag = find_tag_by_id(browser.contents, 'msgset_148_cy_translation_0')
     >>> print(tag.renderContents())
-    foo<img alt="" src="/@@/translation-newline" /><br />
+    foo<img alt="" src="/@@/translation-newline"/><br/>
     %i%i%i
 
 
@@ -1102,7 +1102,7 @@ an error message back.
 
     >>> for tag in find_tags_by_class(admin_browser.contents, 'error'):
     ...     print(tag.renderContents())
-    Ignored your upload because you didn&#x27;t select a file to upload.
+    Ignored your upload because you didn't select a file to upload.
 
 Uploading files with an unkown file format notifies the user that it
 cannot be handled.
diff --git a/lib/lp/translations/stories/translations/xx-translations.txt b/lib/lp/translations/stories/translations/xx-translations.txt
index 077dd0f..9ffa77c 100644
--- a/lib/lp/translations/stories/translations/xx-translations.txt
+++ b/lib/lp/translations/stories/translations/xx-translations.txt
@@ -14,7 +14,7 @@ has no translations at all.
   >>> print(find_tag_by_id(
   ...     user_browser.contents, 'msgset_150').renderContents())
   21.
-  <input type="hidden" name="msgset_150" />
+  <input name="msgset_150" type="hidden"/>
   >>> print(find_tag_by_id(
   ...     user_browser.contents, 'msgset_150_singular').renderContents())
   Found <code>%i</code> invalid file.
@@ -73,7 +73,7 @@ Also, we can see that the message has no active translation yet:
   >>> print(find_tag_by_id(
   ...     user_browser.contents, 'msgset_150').renderContents())
   21.
-  <input type="hidden" name="msgset_150" />
+  <input name="msgset_150" type="hidden"/>
   >>> print(find_tag_by_id(
   ...     user_browser.contents, 'msgset_150_singular').renderContents())
   Found <code>%i</code> invalid file.