beautifulsoup4-4.14.0/CHANGELOG0000644000000000000000000027516413615410400012724 0ustar00= 4.14.0 (20250927) * This version adds function overloading to the find_* methods to make it easier to write type-safe Python. In most cases you can just assign the result of a find() or find_all() call to the type of object you're expecting to get back: a Tag, a NavigableString, a Sequence[Tag], or a Sequence[NavigableString]. It's very rare that you'll have to do a cast or suppress type-checker warnings like you did in previous versions of Beautiful Soup. (In fact, the only time you should still have to do this is if you pass both 'string' and one of the other arguments into one of the find* methods, e.g. tag.find("a", string="tag contents".) The following code has been verified to pass type checking using mypy, pyright, and the Visual Studio Code IDE. It's available in the source repository as scripts/type_checking_smoke_test.py. --- from typing import Optional, Sequence from bs4 import BeautifulSoup, Tag, NavigableString soup = BeautifulSoup("

", 'html.parser') tag:Optional[Tag] string:Optional[NavigableString] tags:Sequence[Tag] strings:Sequence[NavigableString] tag = soup.find() tag = soup.find(id="a") string = soup.find(string="b") tags = soup() tags = soup(id="a") strings = soup(string="b") tags = soup.find_all() tags = soup.find_all(id="a") strings = soup.find_all(string="b") tag = soup.find_next() tag = soup.find_next(id="a") string = soup.find_next(string="b") tags = soup.find_all_next() tags = soup.find_all_next(id="a") strings = soup.find_all_next(string="b") tag = soup.find_next_sibling() tag = soup.find_next_sibling(id="a") string = soup.find_next_sibling(string="b") tags = soup.find_next_siblings() tags = soup.find_next_siblings(id="a") strings = soup.find_next_siblings(string="b") tag = soup.find_previous() tag = soup.find_previous(id="a") string = soup.find_previous(string="b") tags = soup.find_all_previous() tags = soup.find_all_previous(id="a") strings = soup.find_all_previous(string="b") tag = soup.find_previous_sibling() tag = soup.find_previous_sibling(id="a") string = soup.find_previous_sibling(string="bold") tags = soup.find_previous_siblings() tags = soup.find_previous_siblings(id="a") strings = soup.find_previous_siblings(string="b") tag = soup.find_parent() tag = soup.find_parent(id="a") tags = soup.find_parents() tags = soup.find_parents(id="a") # This code will work, but mypy and pyright will both flag it. tags = soup.find_all("a", string="b") --- * The typing for find_parent() and find_parents() was improved without any overloading. Casts should never be necessary, since those methods only ever return Tag and ResultSet[Tag], respectively. * ResultSet now inherits from Sequence. This should make it easier to incorporate ResultSet objects into your type system without needing to handle ResultSet specially. * Fixed an unhandled exception when creating the string representation of a decomposed element. (The output is not *useful* and you still shouldn't do this, but it won't raise an exception anymore.) [bug=2120300] * The default value for the 'attrs' attribute in find* methods is now None, not the empty dictionary. This should have no visible effect on anything. = 4.13.5 (20250824) * Fixed an unhandled exception when parsing invalid markup that contains the { character when using lxml==6.0.0. [bug=2116306] * Fixed a regression when matching a multi-valued attribute against the empty string. [bug=2115352] * Unit tests and test case data are no longer packaged with the wheel. [bug=2107495] * Fixed a bug that gave the wrong result when parsing the empty bytestring. [bug=2110492] * Brought the Spanish translation of the documentation up to date with 4.13.4. Courtesy of Carlos Romero. * For Python 3.13 and above, disabled tests that verify Beautiful Soup's handling of htmlparser's exceptions when given very bad markup. The bug in htmlparser that caused this behavior has been fixed. Patch courtesy of Stefano Rivera. * Used overloading to improve type hints for prettify(). * Updated the SoupStrainer documentation to clarify that during initial parsing, attribute values are always passed into the SoupStrainer as raw strings. [bug=2111651] * Fixed all type checking errors issued by pyright. (Previously only mypy was used for type checking.) * Improved the type hints for PageElement.replace_with. [bug=2114746] * Improved the type hint for the arguments of the lambda function that can be used to match a tag's attribute. [bug=2110401] * Modified some of the lxml tests to accommodate behavioral changes in libxml2 2.14.3. Specifically: 1. XML declarations and processing instructions in HTML documents are rewritten as comments. Note that this means XHTML documents will now turn into regular HTML documents if run through the 'lxml' parser. The 'xml' parser is unaffected. 2. Out-of-range numeric entities are replaced with REPLACEMENT CHARACTER rather than omitted entirely. [bug=2112242] = 4.13.4 (20250415) * If you pass a function as the first argument to a find* method, the function will only ever be called once per tag, with the Tag object as the argument. Starting in 4.13.0, there were cases where the function would be called with a Tag object and then called again with the name of the tag. [bug=2106435] * Added a passthrough implementation for NavigableString.__getitem__ which gives a more helpful exception if the user tries to treat it as a Tag and access its HTML attributes. * Fixed a bug that caused an exception when unpickling the result of parsing certain invalid markup with lxml as the tree builder. [bug=2103126] * Converted the AUTHORS file to UTF-8 for PEP8 compliance. [bug=2107405] = 4.13.3 (20250204) * Modified the 4.13.2 change slightly to restore backwards compatibility. Specifically, calling a find_* method with no arguments should return the first Tag out of the iterator, not the first PageElement. [bug=2097333] = 4.13.2 (20250204) * Gave ElementFilter the ability to explicitly say that it excludes every item in the parse tree. This is used internally in situations where the provided filters are logically inconsistent or match a value against the null set. Without this, it's not always possible to distinguish between a SoupStrainer that excludes everything and one that excludes nothing. This fixes a bug where calls to find_* methods with no arguments returned None, instead of the first item out of the iterator. [bug=2097333] Things added to the API to support this: - The ElementFilter.includes_everything property - The MatchRule.exclude_everything member - The _known_rules argument to ElementFilter.match. This is an optional argument used internally to indicate that an optimization is safe. = 4.13.1 (20250203) * Updated pyproject.toml to require Python 3.7 or above. [bug=2097263] * Pinned the typing-extensions dependency to a minimum version of 4.0.0. [bug=2097262] * Restored the English documentation to the source distribution. [bug=2097237] * Fixed a regression where HTMLFormatter and XMLFormatter were not propagating the indent parameter to the superconstructor. [bug=2097272] = 4.13.0 (20250202) This release introduces Python type hints to all public classes and methods in Beautiful Soup. The addition of these type hints exposed a large number of very small inconsistencies in the code, which I've fixed, but the result is a larger-than-usual number of deprecations and changes that may break backwards compatibility. Chris Papademetrious deserves a special thanks for his work on this release through its long beta process. NOTE: This release was yanked from PyPI on 20250203, because bug 2097263 made it difficult to install Beautiful Soup on Python 3.6. You can still install this version by explicitly pinning beautifulsoup4==4.13.0, but you really shouldn't need to. # Deprecation notices These things now give DeprecationWarnings when you try to use them, and are scheduled to be removed in Beautiful Soup 4.15.0. * Every deprecated method, attribute and class from the 3.0 and 2.0 major versions of Beautiful Soup. These have been deprecated for a very long time, but they didn't issue DeprecationWarning when you tried to use them. Now they do, and they're all going away soon. This mainly refers to methods and attributes with camelCase names, for example: renderContents, replaceWith, replaceWithChildren, findAll, findAllNext, findAllPrevious, findNext, findNextSibling, findNextSiblings, findParent, findParents, findPrevious, findPreviousSibling, findPreviousSiblings, getText, nextSibling, previousSibling, isSelfClosing, fetchNextSiblings, fetchPreviousSiblings, fetchPrevious, fetchPreviousSiblings, fetchParents, findChild, findChildren, childGenerator, nextGenerator, nextSiblingGenerator, previousGenerator, previousSiblingGenerator, recursiveChildGenerator, and parentGenerator. This also includes the BeautifulStoneSoup class. * The SAXTreeBuilder class, which was never officially supported or tested. * The private class method BeautifulSoup._decode_markup(), which has not been used inside Beautiful Soup for many years. * The first argument to BeautifulSoup.decode has been changed from pretty_print:bool to indent_level:int, to match the signature of Tag.decode. Using a bool will still work but will give you a DeprecationWarning. * SoupStrainer.text and SoupStrainer.string are both deprecated, since a single item can't capture all the possibilities of a SoupStrainer designed to match strings. * SoupStrainer.search_tag(). It was never a documented method, but if you use it, you should start using SoupStrainer.allow_tag_creation() instead. * The soup:BeautifulSoup argument to the TreeBuilderForHtml5lib constructor is now required, not optional. It's unclear why it was optional in the first place, so if you discover you need this, contact me for possible un-deprecation. # Compatibility notices * This version drops support for Python 3.6. The minimum supported major Python version for Beautiful Soup is now Python 3.7. * Deprecation warnings have been added for all deprecated methods and attributes (see above). Going forward, deprecated names will be removed two feature releases or one major release after the deprecation warning is added. * The storage for a tag's attribute values now modifies incoming values to be consistent with the HTML or XML spec. This means that if you set an attribute value to a number, it will be converted to a string immediately, rather than being converted when you output the document. [bug=2065525] More importantly for backwards compatibility, setting an HTML attribute value to True will set the attribute's value to the appropriate string per the HTML spec. Setting an attribute value to False or None will remove the attribute value from the tag altogether, rather than (effectively, as before) setting the value to the string "False" or the string "None". This means that some programs that modify documents will generate different output than they would in earlier versions of Beautiful Soup, but the new documents are more likely to represent the intent behind the modifications. To give a specific example, if you have code that looks something like this: checkbox1['checked'] = True checkbox2['checked'] = False Then a document that used to look like this (with most browsers treating both boxes as checked): Will now look like this (with browsers treating only the first box as checked): You can get the old behavior back by instantiating a TreeBuilder with `attribute_dict_class=dict`, or you can customize how Beautiful Soup treates attribute values by passing in a custom subclass of dict. * If Tag.get_attribute_list() is used to access an attribute that's not set, the return value is now an empty list rather than [None]. * If you pass an empty list as the attribute value when searching the tree, you will now find all tags which have that attribute set to a value in the empty list--that is, you will find nothing. This is consistent with other situations where a list of acceptable values is provided. Previously, an empty list was treated the same as None and False, and you would have found the tags which did not have that attribute set at all. [bug=2045469] * For similar reasons, if you pass in limit=0 to a find() method, you will now get zero results. Previously, you would get all matching results. * When using one of the find() methods or creating a SoupStrainer, if you specify the same attribute value in ``attrs`` and the keyword arguments, you'll end up with two different ways to match that attribute. Previously the value in keyword arguments would override the value in ``attrs``. * All exceptions were moved to the bs4.exceptions module, and all warnings to the bs4._warnings module (named so as not to shadow Python's built-in warnings module). All warnings and exceptions are exported from the bs4 module, which is probably the safest place to import them from in your own code. * As a side effect of this, the string constant BeautifulSoup.NO_PARSER_SPECIFIED_WARNING was moved to GuessedAtParserWarning.MESSAGE. * The 'html5' formatter is now much less aggressive about escaping ampersands, escaping only the ampersands considered "ambiguous" by the HTML5 spec (which is almost none of them). This is the sort of change that might break your unit test suite, but the resulting markup will be much more readable and more HTML5-ish. To quickly get the old behavior back, change code like this: tag.encode(formatter='html5') to this: tag.encode(formatter='html5-4.12') In the future, the 'html5' formatter may be become the default HTML formatter, which will change Beautiful Soup's default output. This will break a lot of test suites so it's not going to happen for a while. [bug=1902431] * Tag.sourceline and Tag.sourcepos now always have a consistent data type: Optional[int]. Previously these values were sometimes an Optional[int], and sometimes they were Optional[Tag], the result of searching for a child tag called or . [bug=2065904] If your code does search for a tag called or , it may stop finding that tag when you upgrade to Beautiful Soup 4.13. If this happens, you'll need to replace code that treats "sourceline" or "sourcepos" as tag names: tag.sourceline with code that explicitly calls the find() method: tag.find("sourceline").name Making the behavior of sourceline and sourcepos consistent has the side effect of fixing a major performance problem when a Tag is copied. With this change, the store_line_numbers argument to the BeautifulSoup constructor becomes much less useful, and its use is now discouraged, thought I'm not deprecating it yet. Please contact me if you have a performance or security rationale for setting store_line_numbers=False. * append(), extend(), insert(), and unwrap() were moved from PageElement to Tag. Those methods manipulate the 'contents' collection, so they would only have ever worked on Tag objects. * The BeautifulSoupHTMLParser constructor now requires a BeautifulSoup object as its first argument. This almost certainly does not affect you, since you probably use HTMLParserTreeBuilder, not BeautifulSoupHTMLParser directly. * The TreeBuilderForHtml5lib methods fragmentClass(), getFragment(), and testSerializer() now raise NotImplementedError. These methods are called only by html5lib's test suite, and Beautiful Soup isn't integrated into that test suite, so this code was long since unused and untested. These methods are _not_ deprecated, since they are methods defined by html5lib. They may one day have real implementations, as part of a future effort to integrate Beautiful Soup into html5lib's test suite. * AttributeValueWithCharsetSubstitution.encode() is renamed to substitute_encoding, to avoid confusion with the much different str.encode() * Using PageElement.replace_with() to replace an element with itself returns the element instead of None. * All TreeBuilder constructors now take the empty_element_tags argument. The sets of tags found in HTMLTreeBuilder.empty_element_tags and HTMLTreeBuilder.block_elements are now in HTMLTreeBuilder.DEFAULT_EMPTY_ELEMENT_TAGS and HTMLTreeBuilder.DEFAULT_BLOCK_ELEMENTS, to avoid confusing them with instance variables. * The unused constant LXMLTreeBuilderForXML.DEFAULT_PARSER_CLASS has been removed. * Some of the arguments in the methods of LXMLTreeBuilderForXML have been renamed for consistency with the names lxml uses for those arguments in the superclass. This won't affect you unless you were calling methods like LXMLTreeBuilderForXML.start() directly. * In particular, the arguments to LXMLTreeBuilderForXML.prepare_markup have been changed to match the arguments to the superclass, TreeBuilder.prepare_markup. Specifically, document_declared_encoding now appears before exclude_encodings, not after. If you were calling this method yourself, I recommend switching to using keyword arguments instead. # New features * The new ElementFilter class encapsulates Beautiful Soup's rules about matching elements and deciding which parts of a document to parse. It's easy to override those rules with subclassing or function composition. The SoupStrainer class, which contains all the matching logic you're familiar with from the find_* methods, is now a subclass of ElementFilter. * The new PageElement.filter() method provides a fully general way of finding elements in a Beautiful Soup parse tree. You can specify a function to iterate over the tree and an ElementFilter to determine what matches. * The new_tag() method now takes a 'string' argument. This allows you to set the string contents of a Tag when creating it. Patch by Chris Papademetrious. [bug=2044599] * Defined a number of new iterators which are the same as existing iterators, but which yield the element itself before beginning to traverse the tree. [bug=2052936] [bug=2067634] - PageElement.self_and_parents - PageElement.self_and_descendants - PageElement.self_and_next_elements - PageElement.self_and_next_siblings - PageElement.self_and_previous_elements - PageElement.self_and_previous_siblings self_and_parents yields the element you call it on and then all of its parents. self_and_next_element yields the element you call it on and then every element parsed afterwards; and so on. * The NavigableString class now has a .string property which returns the string itself. This makes it easier to iterate over a mixed list of Tag and NavigableString objects. [bug=2044794] * Defined a new method, Tag.copy_self(), which creates a copy of a Tag with the same attributes but no contents. [bug=2065120] Note that this method used to be a private method named _clone(). The _clone() method has been removed, so if you were using it, change your code to call copy_self() instead. * The PageElement.append() method now returns the element that was appended; it used to have no return value. [bug=2093025] * The methods PageElement.insert(), PageElement.extend(), PageElement.insert_before(), and PageElement.insert_after() now return a list of the items inserted. These methods used to have no return value. [bug=2093025] * The PageElement.insert() method now takes a variable number of arguments and returns a list of all elements inserted, to match insert_before() and insert_after(). (Even if I hadn't made the variable-argument change, an edge case around inserting one Beautiful Soup object into another means that insert()'s return value needs to be a list.) [bug=2093025] * Defined a new warning class, UnusualUsageWarning, which is a superclass for all of the warnings issued when Beautiful Soup notices something unusual but not guaranteed to be wrong, like markup that looks like a URL (MarkupResemblesLocatorWarning) or XML being run through an HTML parser (XMLParsedAsHTMLWarning). The text of these warnings has been revamped to explain in more detail what is going on, how to check if you've made a mistake, and how to make the warning go away if you are acting deliberately. If these warnings are interfering with your workflow, or simply annoying you, you can filter all of them by filtering UnusualUsageWarning, without worrying about losing the warnings Beautiful Soup issues when there *definitely* is a problem you need to correct. * It's now possible to modify the behavior of the list used to store the values of multi-valued attributes such as HTML 'class', by passing in whatever class you want instantiated (instead of a normal Python list) to the TreeBuilder constructor as attribute_value_list_class. [bug=2052943] # Improvements * decompose() was moved from Tag to its superclass PageElement, since there's no reason it won't also work on NavigableString objects. * Emit an UnusualUsageWarning if the user tries to search for an attribute called _class; they probably mean "class_". [bug=2025089] * The MarkupResemblesLocatorWarning issued when the markup resembles a filename is now issued less often, due to improvements in detecting markup that's unlikely to be a filename. [bug=2052988] * Emit a warning if a document is parsed using a SoupStrainer that's set up to filter everything. In these cases, filtering everything is the most consistent thing to do, but there was no indication that this was happening, so the behavior may have seemed mysterious. * When using one of the find() methods or creating a SoupStrainer, you can pass a list of any accepted object (strings, regular expressions, etc.) for any of the objects. Previously you could only pass in a list of strings. * A SoupStrainer can now filter tag creation based on a tag's namespaced name. Previously only the unqualified name could be used. * Added the correct stacklevel to another instance of the XMLParsedAsHTMLWarning. [bug=2034451] * Improved the wording of the TypeError raised when you pass something other than markup into the BeautifulSoup constructor. [bug=2071530] * Optimized the case where you use Tag.insert() to "insert" a PageElement into its current location. [bug=2077020] * Changes to make tests work whether tests are run under soupsieve 2.6 or an earlier version. Based on a patch by Stefano Rivera. * Removed the strip_cdata argument to lxml's HTMLParser constructor, which never did anything and is deprecated as of lxml 5.3.0. Patch by Stefano Rivera. [bug=2076897] # Bug fixes * Copying a tag with a multi-valued attribute now makes a copy of the list of values, eliminating a bug where both the old and new copy shared the same list. [bug=2067412] * The lxml TreeBuilder, like the other TreeBuilders, now filters a document's initial DOCTYPE if you've set up a SoupStrainer that eliminates it. [bug=2062000] * A lot of things can go wrong if you modify the parse tree while iterating over it, especially if you are removing or replacing elements. Most of those things fall under the category of unexpected behavior (which is why I don't recommend doing this), but there are a few ways that caused unhandled exceptions. The list comprehensions used by Beautiful Soup (e.g. .descendants, which powers the find* methods) should now work correctly in those cases, or at least not raise exceptions. As part of this work, I changed when the list comprehension determines the next element. Previously it was done after the yield statement; now it's done before the yield statement. This lets you remove the yielded element in calling code, or modify it in a way that would break this calculation, without causing an exception. So if your code relies on modifying the tree in a way that 'steers' a list comprehension, rather than using the list comprension to decide which bits of the tree to modify, it will probably stop working at this point. [bug=2091118] * Fixed an error in the lookup table used when converting ISO-Latin-1 to ASCII, which no one should do anyway. * Corrected the markup that's output in the unlikely event that you encode a document to a Python internal encoding (like "palmos") that's not recognized by the HTML or XML standard. * UnicodeDammit.markup is now always a bytestring representing the *original* markup (sans BOM), and UnicodeDammit.unicode_markup is always the converted Unicode equivalent of the original markup. Previously, UnicodeDammit.markup was treated inconsistently and would often end up containing Unicode. UnicodeDammit.markup was not a documented attribute, but if you were using it, you probably want to switch to using .unicode_markup instead. = 4.12.3 (20240117) * The Beautiful Soup documentation now has a Spanish translation, thanks to Carlos Romero. Delong Wang's Chinese translation has been updated to cover Beautiful Soup 4.12.0. * Fixed a regression such that if you set .hidden on a tag, the tag becomes invisible but its contents are still visible. User manipulation of .hidden is not a documented or supported feature, so don't do this, but it wasn't too difficult to keep the old behavior working. * Fixed a case found by Mengyuhan where html.parser giving up on markup would result in an AssertionError instead of a ParserRejectedMarkup exception. * Added the correct stacklevel to instances of the XMLParsedAsHTMLWarning. [bug=2034451] * Corrected the syntax of the license definition in pyproject.toml. Patch by Louis Maddox. [bug=2032848] * Corrected a typo in a test that was causing test failures when run against libxml2 2.12.1. [bug=2045481] = 4.12.2 (20230407) * Fixed an unhandled exception in BeautifulSoup.decode_contents and methods that call it. [bug=2015545] = 4.12.1 (20230405) NOTE: the following things are likely to be dropped in the next feature release of Beautiful Soup: Official support for Python 3.6. Inclusion of unit tests and test data in the wheel file. Two scripts: demonstrate_parser_differences.py and test-all-versions. Changes: * This version of Beautiful Soup replaces setup.py and setup.cfg with pyproject.toml. Beautiful Soup now uses tox as its test backend and hatch to do builds. * The main functional improvement in this version is a nonrecursive technique for regenerating a tree. This technique is used to avoid situations where, in previous versions, doing something to a very deeply nested tree would overflow the Python interpreter stack: 1. Outputting a tree as a string, e.g. with BeautifulSoup.encode() [bug=1471755] 2. Making copies of trees (copy.copy() and copy.deepcopy() from the Python standard library). [bug=1709837] 3. Pickling a BeautifulSoup object. (Note that pickling a Tag object can still cause an overflow.) * Making a copy of a BeautifulSoup object no longer parses the document again, which should improve performance significantly. * When a BeautifulSoup object is unpickled, Beautiful Soup now tries to associate an appropriate TreeBuilder object with it. * Tag.prettify() will now consistently end prettified markup with a newline. * Added unit tests for fuzz test cases created by third parties. Some of these tests are skipped since they point to problems outside of Beautiful Soup, but this change puts them all in one convenient place. * PageElement now implements the known_xml attribute. (This was technically a bug, but it shouldn't be an issue in normal use.) [bug=2007895] * The demonstrate_parser_differences.py script was still written in Python 2. I've converted it to Python 3, but since no one has mentioned this over the years, it's a sign that no one uses this script and it's not serving its purpose. = 4.12.0 (20230320) * Introduced the .css property, which centralizes all access to the Soup Sieve API. This allows Beautiful Soup to give direct access to as much of Soup Sieve that makes sense, without cluttering the BeautifulSoup and Tag classes with a lot of new methods. This does mean one addition to the BeautifulSoup and Tag classes (the .css property itself), so this might be a breaking change if you happen to use Beautiful Soup to parse XML that includes a tag called . In particular, code like this will stop working in 4.12.0: soup.css['id'] Code like this will work just as before: soup.find_one('css')['id'] The Soup Sieve methods supported through the .css property are select(), select_one(), iselect(), closest(), match(), filter(), escape(), and compile(). The BeautifulSoup and Tag classes still support the select() and select_one() methods; they have not been deprecated, but they have been demoted to convenience methods. [bug=2003677] * When the html.parser parser decides it can't parse a document, Beautiful Soup now consistently propagates this fact by raising a ParserRejectedMarkup error. [bug=2007343] * Removed some error checking code from diagnose(), which is redundant with similar (but more Pythonic) code in the BeautifulSoup constructor. [bug=2007344] * Added intersphinx references to the documentation so that other projects have a target to point to when they reference Beautiful Soup classes. [bug=1453370] = 4.11.2 (20230131) * Fixed test failures caused by nondeterministic behavior of UnicodeDammit's character detection, depending on the platform setup. [bug=1973072] * Fixed another crash when overriding multi_valued_attributes and using the html5lib parser. [bug=1948488] * The HTMLFormatter and XMLFormatter constructors no longer return a value. [bug=1992693] * Tag.interesting_string_types is now propagated when a tag is copied. [bug=1990400] * Warnings now do their best to provide an appropriate stacklevel, improving the usefulness of the message. [bug=1978744] * Passing a Tag's .contents into PageElement.extend() now works the same way as passing the Tag itself. * Soup Sieve tests will be skipped if the library is not installed. = 4.11.1 (20220408) This release was done to ensure that the unit tests are packaged along with the released source. There are no functionality changes in this release, but there are a few other packaging changes: * The Japanese and Korean translations of the documentation are included. * The changelog is now packaged as CHANGELOG, and the license file is packaged as LICENSE. NEWS.txt and COPYING.txt are still present, but may be removed in the future. * TODO.txt is no longer packaged, since a TODO is not relevant for released code. = 4.11.0 (20220407) * Ported unit tests to use pytest. * Added special string classes, RubyParenthesisString and RubyTextString, to make it possible to treat ruby text specially in get_text() calls. [bug=1941980] * It's now possible to customize the way output is indented by providing a value for the 'indent' argument to the Formatter constructor. The 'indent' argument works very similarly to the argument of the same name in the Python standard library's json.dump() function. [bug=1955497] * If the charset-normalizer Python module (https://pypi.org/project/charset-normalizer/) is installed, Beautiful Soup will use it to detect the character sets of incoming documents. This is also the module used by newer versions of the Requests library. For the sake of backwards compatibility, chardet and cchardet both take precedence if installed. [bug=1955346] * Added a workaround for an lxml bug (https://bugs.launchpad.net/lxml/+bug/1948551) that causes problems when parsing a Unicode string beginning with BYTE ORDER MARK. [bug=1947768] * Issue a warning when an HTML parser is used to parse a document that looks like XML but not XHTML. [bug=1939121] * Do a better job of keeping track of namespaces as an XML document is parsed, so that CSS selectors that use namespaces will do the right thing more often. [bug=1946243] * Some time ago, the misleadingly named "text" argument to find-type methods was renamed to the more accurate "string." But this supposed "renaming" didn't make it into important places like the method signatures or the docstrings. That's corrected in this version. "text" still works, but will give a DeprecationWarning. [bug=1947038] * Fixed a crash when pickling a BeautifulSoup object that has no tree builder. [bug=1934003] * Fixed a crash when overriding multi_valued_attributes and using the html5lib parser. [bug=1948488] * Standardized the wording of the MarkupResemblesLocatorWarning warnings to omit untrusted input and make the warnings less judgmental about what you ought to be doing. [bug=1955450] * Removed support for the iconv_codec library, which doesn't seem to exist anymore and was never put up on PyPI. (The closest replacement on PyPI, iconv_codecs, is GPL-licensed, so we can't use it--it's also quite old.) = 4.10.0 (20210907) * This is the first release of Beautiful Soup to only support Python 3. I dropped Python 2 support to maintain support for newer versions (58 and up) of setuptools. See: https://github.com/pypa/setuptools/issues/2769 [bug=1942919] * The behavior of methods like .get_text() and .strings now differs depending on the type of tag. The change is visible with HTML tags like

This numeric entity is missing the final semicolon:
, that attribute value was closed by the subsequent tag
a
This document contains (do you see it?)
This document ends with That attribute value was bogus
The doctype is invalid because it contains extra whitespace
That boolean attribute had no value
Here's a nonexistent entity: &#foo; (do you see it?)
This document ends before the entity finishes: >

Paragraphs shouldn't contain block display elements, but this one does:

you see?

Multiple values for the same attribute.
Here's a table
This tag contains nothing but whitespace:

This p tag is cut off by

the end of the blockquote tag
Here's a nested table:
foo
This table contains bare markup
\n
This document contains a surprise doctype
Mixed case tags are folded to lowercase
Tag name contains Unicode characters
Attribute name contains Unicode characters
""" class SoupTest(object): @property def default_builder(self) -> Type[TreeBuilder]: return default_builder def soup(self, markup: _IncomingMarkup, **kwargs: Any) -> BeautifulSoup: """Build a Beautiful Soup object from markup.""" builder = kwargs.pop("builder", self.default_builder) return BeautifulSoup(markup, builder=builder, **kwargs) def document_for(self, markup: str, **kwargs: Any) -> str: """Turn an HTML fragment into a document. The details depend on the builder. """ return self.default_builder(**kwargs).test_fragment_to_document(markup) def assert_soup( self, to_parse: _IncomingMarkup, compare_parsed_to: Optional[str] = None ) -> None: """Parse some markup using Beautiful Soup and verify that the output markup is as expected. """ builder = self.default_builder obj = BeautifulSoup(to_parse, builder=builder) if compare_parsed_to is None: assert isinstance(to_parse, str) compare_parsed_to = to_parse # Verify that the documents come out the same. assert obj.decode() == self.document_for(compare_parsed_to) # Also run some checks on the BeautifulSoup object itself: # Verify that every tag that was opened was eventually closed. # There are no tags in the open tag counter. assert all(v == 0 for v in list(obj.open_tag_counter.values())) # The only tag in the tag stack is the one for the root # document. assert [obj.ROOT_TAG_NAME] == [x.name for x in obj.tagStack] assertSoupEquals = assert_soup def assertConnectedness(self, element: Tag) -> None: """Ensure that next_element and previous_element are properly set for all descendants of the given element. """ earlier = None for e in element.descendants: if earlier: assert e == earlier.next_element assert earlier == e.previous_element earlier = e def linkage_validator( self, el: Tag, _recursive_call: bool = False ) -> Optional[PageElement]: """Ensure proper linkage throughout the document.""" descendant = None # Document element should have no previous element or previous sibling. # It also shouldn't have a next sibling. if el.parent is None: assert ( el.previous_element is None ), "Bad previous_element\nNODE: {}\nPREV: {}\nEXPECTED: {}".format( el, el.previous_element, None ) assert ( el.previous_sibling is None ), "Bad previous_sibling\nNODE: {}\nPREV: {}\nEXPECTED: {}".format( el, el.previous_sibling, None ) assert ( el.next_sibling is None ), "Bad next_sibling\nNODE: {}\nNEXT: {}\nEXPECTED: {}".format( el, el.next_sibling, None ) idx = 0 child = None last_child = None last_idx = len(el.contents) - 1 for child in el.contents: descendant = None # Parent should link next element to their first child # That child should have no previous sibling if idx == 0: if el.parent is not None: assert ( el.next_element is child ), "Bad next_element\nNODE: {}\nNEXT: {}\nEXPECTED: {}".format( el, el.next_element, child ) assert ( child.previous_element is el ), "Bad previous_element\nNODE: {}\nPREV: {}\nEXPECTED: {}".format( child, child.previous_element, el ) assert ( child.previous_sibling is None ), "Bad previous_sibling\nNODE: {}\nPREV {}\nEXPECTED: {}".format( child, child.previous_sibling, None ) # If not the first child, previous index should link as sibling to this index # Previous element should match the last index or the last bubbled up descendant else: assert ( child.previous_sibling is el.contents[idx - 1] ), "Bad previous_sibling\nNODE: {}\nPREV {}\nEXPECTED {}".format( child, child.previous_sibling, el.contents[idx - 1] ) assert ( el.contents[idx - 1].next_sibling is child ), "Bad next_sibling\nNODE: {}\nNEXT {}\nEXPECTED {}".format( el.contents[idx - 1], el.contents[idx - 1].next_sibling, child ) if last_child is not None: assert ( child.previous_element is last_child ), "Bad previous_element\nNODE: {}\nPREV {}\nEXPECTED {}\nCONTENTS {}".format( child, child.previous_element, last_child, child.parent.contents ) assert ( last_child.next_element is child ), "Bad next_element\nNODE: {}\nNEXT {}\nEXPECTED {}".format( last_child, last_child.next_element, child ) if isinstance(child, Tag) and child.contents: descendant = self.linkage_validator(child, True) assert descendant is not None # A bubbled up descendant should have no next siblings assert ( descendant.next_sibling is None ), "Bad next_sibling\nNODE: {}\nNEXT {}\nEXPECTED {}".format( descendant, descendant.next_sibling, None ) # Mark last child as either the bubbled up descendant or the current child if descendant is not None: last_child = descendant else: last_child = child # If last child, there are non next siblings if idx == last_idx: assert ( child.next_sibling is None ), "Bad next_sibling\nNODE: {}\nNEXT {}\nEXPECTED {}".format( child, child.next_sibling, None ) idx += 1 child = descendant if descendant is not None else child if child is None: child = el if not _recursive_call and child is not None: target: Optional[Tag] = el while True: if target is None: assert ( child.next_element is None ), "Bad next_element\nNODE: {}\nNEXT {}\nEXPECTED {}".format( child, child.next_element, None ) break elif target.next_sibling is not None: assert ( child.next_element is target.next_sibling ), "Bad next_element\nNODE: {}\nNEXT {}\nEXPECTED {}".format( child, child.next_element, target.next_sibling ) break target = target.parent # We are done, so nothing to return return None else: # Return the child to the recursive caller return child def assert_selects(self, tags: Iterable[Tag], should_match: Iterable[str]) -> None: """Make sure that the given tags have the correct text. This is used in tests that define a bunch of tags, each containing a single string, and then select certain strings by some mechanism. """ assert [tag.string for tag in tags] == should_match def assert_selects_ids( self, tags: Iterable[Tag], should_match: Iterable[str] ) -> None: """Make sure that the given tags have the correct IDs. This is used in tests that define a bunch of tags, each containing a single string, and then select certain strings by some mechanism. """ assert [tag["id"] for tag in tags] == should_match class TreeBuilderSmokeTest(SoupTest): # Tests that are common to HTML and XML tree builders. @pytest.mark.parametrize( "multi_valued_attributes", [None, {}, dict(b=["class"]), {"*": ["notclass"]}] ) def test_attribute_not_multi_valued(self, multi_valued_attributes): markup = '' soup = self.soup(markup, multi_valued_attributes=multi_valued_attributes) assert soup.a["class"] == "a b c" @pytest.mark.parametrize( "multi_valued_attributes", [dict(a=["class"]), {"*": ["class"]}] ) def test_attribute_multi_valued(self, multi_valued_attributes): markup = '' soup = self.soup(markup, multi_valued_attributes=multi_valued_attributes) assert soup.a["class"] == ["a", "b", "c"] def test_invalid_doctype(self): # We don't have an official opinion on how these are parsed, # but they shouldn't crash any of the parsers. markup = "content" self.soup(markup) markup = "" self.soup(markup) def test_doctype_filtered(self): markup = "\n\n" soup = self.soup(markup, parse_only=SoupStrainer(name="html")) assert not any(isinstance(x, Doctype) for x in soup.descendants) def test_custom_attribute_dict_class(self): class MyAttributeDict(dict): def __setitem__(self, key: str, value: Any): # Ignore the provided value and substitute a # hard-coded one. super().__setitem__(key, "OVERRIDDEN") markup = 'f' builder = self.default_builder(attribute_dict_class=MyAttributeDict) soup = self.soup(markup, builder=builder) tag = soup.a assert isinstance(tag.attrs, MyAttributeDict) assert "OVERRIDDEN" == tag["attr1"] tag["attr3"] = True assert "OVERRIDDEN" == tag["attr3"] expect = 'f' assert expect == tag.decode() def test_custom_attribute_value_list_class(self): class MyCustomAttributeValueList(AttributeValueList): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.append("extra") builder = self.default_builder( multi_valued_attributes={"*": set(["attr2"])}, attribute_value_list_class=MyCustomAttributeValueList, ) markup = 'f' soup = self.soup(markup, builder=builder) tag = soup.a assert tag["attr1"] == "val1" assert tag["attr2"] == ["val2", "extra"] assert isinstance(tag["attr2"], MyCustomAttributeValueList) class HTMLTreeBuilderSmokeTest(TreeBuilderSmokeTest): """A basic test of a treebuilder's competence. Any HTML treebuilder, present or future, should be able to pass these tests. With invalid markup, there's room for interpretation, and different parsers can handle it differently. But with the markup in these tests, there's not much room for interpretation. """ def test_empty_element_tags(self): """Verify that all HTML4 and HTML5 empty element (aka void element) tags are handled correctly. """ for name in [ "area", "base", "br", "col", "embed", "hr", "img", "input", "keygen", "link", "menuitem", "meta", "param", "source", "track", "wbr", "spacer", "frame", ]: soup = self.soup("") new_tag = soup.new_tag(name) assert new_tag.is_empty_element is True self.assert_soup("


", "


") self.assert_soup("


", "


") def test_special_string_containers(self): soup = self.soup("") assert isinstance(soup.style.string, Stylesheet) assert isinstance(soup.script.string, Script) soup = self.soup("") assert isinstance(soup.style.string, Stylesheet) # The contents of the style tag resemble an HTML comment, but # it's not treated as a comment. assert soup.style.string == "" assert isinstance(soup.style.string, Stylesheet) def test_pickle_and_unpickle_identity(self): # Pickling a tree, then unpickling it, yields a tree identical # to the original. tree = self.soup("foo") dumped = pickle.dumps(tree, pickle.HIGHEST_PROTOCOL) loaded = pickle.loads(dumped) assert loaded.__class__ == BeautifulSoup assert loaded.decode() == tree.decode() def test_pickle_and_unpickle_bad_markup(self): markup = """ blabla """ soup = self.soup(markup) pickled = pickle.dumps(soup, pickle.HIGHEST_PROTOCOL) soup = pickle.loads(pickled) assert soup.builder.is_xml is False def assertDoctypeHandled(self, doctype_fragment: str) -> None: """Assert that a given doctype string is handled correctly.""" doctype_str, soup = self._document_with_doctype(doctype_fragment) # Make sure a Doctype object was created. doctype = soup.contents[0] assert doctype.__class__ == Doctype assert doctype == doctype_fragment assert soup.encode("utf8")[: len(doctype_str)] == doctype_str # Make sure that the doctype was correctly associated with the # parse tree and that the rest of the document parsed. assert soup.p is not None assert soup.p.contents[0] == "foo" def _document_with_doctype( self, doctype_fragment: str, doctype_string: str = "DOCTYPE" ) -> Tuple[bytes, BeautifulSoup]: """Generate and parse a document with the given doctype.""" doctype = "" % (doctype_string, doctype_fragment) markup = doctype + "\n

foo

" soup = self.soup(markup) return doctype.encode("utf8"), soup def test_normal_doctypes(self): """Make sure normal, everyday HTML doctypes are handled correctly.""" self.assertDoctypeHandled("html") self.assertDoctypeHandled( 'html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"' ) def test_empty_doctype(self): soup = self.soup("") doctype = soup.contents[0] assert "" == doctype.strip() def test_mixed_case_doctype(self): # A lowercase or mixed-case doctype becomes a Doctype. for doctype_fragment in ("doctype", "DocType"): doctype_str, soup = self._document_with_doctype("html", doctype_fragment) # Make sure a Doctype object was created and that the DOCTYPE # is uppercase. doctype = soup.contents[0] assert doctype.__class__ == Doctype assert doctype == "html" assert soup.encode("utf8")[: len(doctype_str)] == b"" # Make sure that the doctype was correctly associated with the # parse tree and that the rest of the document parsed. assert soup.p.contents[0] == "foo" def test_public_doctype_with_url(self): doctype = 'html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"' self.assertDoctypeHandled(doctype) def test_system_doctype(self): self.assertDoctypeHandled('foo SYSTEM "http://www.example.com/"') def test_namespaced_system_doctype(self): # We can handle a namespaced doctype with a system ID. self.assertDoctypeHandled('xsl:stylesheet SYSTEM "htmlent.dtd"') def test_namespaced_public_doctype(self): # Test a namespaced doctype with a public id. self.assertDoctypeHandled('xsl:stylesheet PUBLIC "htmlent.dtd"') def test_real_xhtml_document(self): """A real XHTML document should come out more or less the same as it went in.""" markup = b""" Hello. Goodbye. """ # libxml 2.14.3 turns the XML declaration into a comment, turning the document # from XHTML to regular HTML. That's fine with this test. markup_as_html = markup.replace(b'', b'') with warnings.catch_warnings(record=True) as w: soup = self.soup(markup) without_minor_differences = soup.encode("utf-8").replace(b"\n", b"") assert (without_minor_differences == markup.replace(b"\n", b"") or without_minor_differences == markup_as_html.replace(b"\n", b"")) # No warning was issued about parsing an XML document as HTML, # because XHTML is both. assert w == [] def test_namespaced_html(self): # When a namespaced XML document is parsed as HTML it should # be treated as HTML with weird tag names. markup = b"""content""" with warnings.catch_warnings(record=True) as w: soup = self.soup(markup) assert 2 == len(soup.find_all("ns1:foo")) # n.b. no "you're parsing XML as HTML" warning was given # because there was no XML declaration. assert [] == w def test_detect_xml_parsed_as_html(self): # A warning is issued when parsing an XML document as HTML, # but basic stuff should still work. markup = b"""string""" with warnings.catch_warnings(record=True) as w: soup = self.soup(markup) assert soup.tag.string == "string" [warning] = w assert isinstance(warning.message, XMLParsedAsHTMLWarning) assert str(warning.message) == XMLParsedAsHTMLWarning.MESSAGE # NOTE: the warning is not issued if the document appears to # be XHTML (tested with test_real_xhtml_document in the # superclass) or if there is no XML declaration (tested with # test_namespaced_html in the superclass). def test_processing_instruction(self): # We test both Unicode and bytestring to verify that # process_markup correctly sets processing_instruction_class # even when the markup is already Unicode and there is no # need to process anything. # # Since HTML doesn't have any processing instructions, it's also # legal to convert the PI into a comment, as libxml2 2.14.3 does. markup = """""" markup_as_comment = "" soup = self.soup(markup) assert soup.decode() in [markup, markup_as_comment] markup = markup.encode("utf8") markup_as_comment = markup_as_comment.encode("utf8") soup = self.soup(markup) assert soup.encode("utf8") in [markup, markup_as_comment] def test_deepcopy(self): """Make sure you can copy the tree builder. This is important because the builder is part of a BeautifulSoup object, and we want to be able to copy that. """ copy.deepcopy(self.default_builder) def test_p_tag_is_never_empty_element(self): """A

tag is never designated as an empty-element tag. Even if the markup shows it as an empty-element tag, it shouldn't be presented that way. """ soup = self.soup("

") assert not soup.p.is_empty_element assert str(soup.p) == "

" def test_unclosed_tags_get_closed(self): """A tag that's not closed by the end of the document should be closed. This applies to all tags except empty-element tags. """ self.assert_soup("

", "

") self.assert_soup("", "") self.assert_soup("
", "
") def test_br_is_always_empty_element_tag(self): """A
tag is designated as an empty-element tag. Some parsers treat

as one
tag, some parsers as two tags, but it should always be an empty-element tag. """ soup = self.soup("

") assert soup.br.is_empty_element assert str(soup.br) == "
" def test_nested_formatting_elements(self): self.assert_soup("") def test_double_head(self): html = """ Ordinary HEAD element test Hello, world! """ soup = self.soup(html) assert "text/javascript" == soup.find("script")["type"] def test_comment(self): # Comments are represented as Comment objects. markup = "

foobaz

" self.assert_soup(markup) soup = self.soup(markup) comment = soup.find(string="foobar") assert comment.__class__ == Comment # The comment is properly integrated into the tree. foo = soup.find(string="foo") assert comment == foo.next_element baz = soup.find(string="baz") assert comment == baz.previous_element def test_preserved_whitespace_in_pre_and_textarea(self): """Whitespace must be preserved in 
 and \n"
        self.assert_soup(pre_markup)
        self.assert_soup(textarea_markup)

        soup = self.soup(pre_markup)
        assert soup.pre.prettify() == pre_markup

        soup = self.soup(textarea_markup)
        assert soup.textarea.prettify() == textarea_markup

        soup = self.soup("")
        assert soup.textarea.prettify() == "\n"

    def test_nested_inline_elements(self):
        """Inline elements can be nested indefinitely."""
        b_tag = "Inside a B tag"
        self.assert_soup(b_tag)

        nested_b_tag = "
A nested tag
"
        self.assert_soup(nested_b_tag)

        double_nested_b_tag = "
A doubly nested tag
"
        self.assert_soup(double_nested_b_tag)

    def test_nested_block_level_elements(self):
        """Block elements can be nested."""
        soup = self.soup("
Foo
")
        blockquote = soup.blockquote
        assert blockquote.p.b.string == "Foo"
        assert blockquote.b.string == "Foo"

    def test_correctly_nested_tables(self):
        """One table can go inside another one."""
        markup = (
            ''
            ""
            ""
        )

        self.assert_soup(
            markup,
            '
Here's another table:"
            ''
            ""
            "
foo
Here\'s another table:'
            '
foo
'
            "
",
        )

        self.assert_soup(
            ""
            ""
            "
Foo
Bar
Baz
"
        )

    def test_multivalued_attribute_with_whitespace(self):
        # Whitespace separating the values of a multi-valued attribute
        # should be ignored.

        markup = '
'
        soup = self.soup(markup)
        assert ["foo", "bar"] == soup.div["class"]

        # If you search by the literal name of the class it's like the whitespace
        # wasn't there.
        assert soup.div == soup.find("div", class_="foo bar")

    def test_deeply_nested_multivalued_attribute(self):
        # html5lib can set the attributes of the same tag many times
        # as it rearranges the tree. This has caused problems with
        # multivalued attributes.
        markup = '
'
        soup = self.soup(markup)
        assert ["css"] == soup.div.div["class"]

    def test_multivalued_attribute_on_html(self):
        # html5lib uses a different API to set the attributes ot the
        #  tag. This has caused problems with multivalued
        # attributes.
        markup = ''
        soup = self.soup(markup)
        assert ["a", "b"] == soup.html["class"]

    def test_angle_brackets_in_attribute_values_are_escaped(self):
        self.assert_soup('', '')

    def test_strings_resembling_character_entity_references(self):
        # "&T" and "&p" look like incomplete character entities, but they are
        # not.
        self.assert_soup(
            "
• AT&T is in the s&p 500
",
            "
\u2022 AT&T is in the s&p 500
",
        )

    def test_apos_entity(self):
        self.assert_soup(
            "
Bob's Bar
",
            "
Bob's Bar
",
        )

    def test_entities_in_foreign_document_encoding(self):
        # “ and ” are invalid numeric entities referencing
        # Windows-1252 characters. - references a character common
        # to Windows-1252 and Unicode, and ☃ references a
        # character only found in Unicode.
        #
        # All of these entities should be converted to Unicode
        # characters.
        markup = "
“Hello” -☃
"
        soup = self.soup(markup)
        assert "“Hello” -☃" == soup.p.string

    def test_entities_in_attributes_converted_to_unicode(self):
        expect = '
'
        self.assert_soup('
', expect)
        self.assert_soup('
', expect)
        self.assert_soup('
', expect)
        self.assert_soup('
', expect)

    def test_entities_in_text_converted_to_unicode(self):
        expect = "
pi\N{LATIN SMALL LETTER N WITH TILDE}ata
"
        self.assert_soup("
piñata
", expect)
        self.assert_soup("
piñata
", expect)
        self.assert_soup("
piñata
", expect)
        self.assert_soup("
piñata
", expect)

    def test_quot_entity_converted_to_quotation_mark(self):
        self.assert_soup(
            "
I said "good day!"
", '
I said "good day!"
'
        )

    def test_out_of_range_entity(self):
        expect = "\N{REPLACEMENT CHARACTER}"
        self.assert_soup("�", expect)
        self.assert_soup("�", expect)
        self.assert_soup("�", expect)

    def test_multipart_strings(self):
        "Mostly to prevent a recurrence of a bug in the html5lib treebuilder."
        soup = self.soup("
\nfoo
")
        assert "p" == soup.h2.string.next_element.name
        assert "p" == soup.p.name
        self.assertConnectedness(soup)

    def test_invalid_html_entity(self):
        # The html.parser treebuilder can't distinguish between an
        # invalid HTML entity with a semicolon and an invalid HTML
        # entity with no semicolon (see its subclass for the tested
        # behavior). But the other treebuilders can.
        markup = "
a &nosuchentity b
"
        soup = self.soup(markup)
        assert "
a &nosuchentity b
" == soup.p.decode()

        markup = "
a &nosuchentity; b
"
        soup = self.soup(markup)
        assert "
a &nosuchentity; b
" == soup.p.decode()

    def test_head_tag_between_head_and_body(self):
        "Prevent recurrence of a bug in the html5lib treebuilder."
        content = """
  
  foo

"""
        soup = self.soup(content)
        assert soup.html.body is not None
        self.assertConnectedness(soup)

    def test_multiple_copies_of_a_tag(self):
        "Prevent recurrence of a bug in the html5lib treebuilder."
        content = """

 
   
  

"""
        soup = self.soup(content)
        self.assertConnectedness(soup.article)

    def test_basic_namespaces(self):
        """Parsers don't need to *understand* namespaces, but at the
        very least they should not choke on namespaces or lose
        data."""

        markup = b'4'
        soup = self.soup(markup)
        assert markup == soup.encode()
        assert "http://www.w3.org/1999/xhtml" == soup.html["xmlns"]
        assert "http://www.w3.org/1998/Math/MathML" == soup.html["xmlns:mathml"]
        assert "http://www.w3.org/2000/svg" == soup.html["xmlns:svg"]

    def test_multivalued_attribute_value_becomes_list(self):
        markup = b''
        soup = self.soup(markup)
        assert ["foo", "bar"] == soup.a["class"]

    #
    # Generally speaking, tests below this point are more tests of
    # Beautiful Soup than tests of the tree builders. But parsers are
    # weird, so we run these tests separately for every tree builder
    # to detect any differences between them.
    #

    def test_can_parse_unicode_document(self):
        # A seemingly innocuous document... but it's in Unicode! And
        # it contains characters that can't be represented in the
        # encoding found in the  declaration! The horror!
        markup = 'Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!'
        soup = self.soup(markup)
        assert "Sacr\xe9 bleu!" == soup.body.string

    def test_soupstrainer(self):
        """Parsers should be able to work with SoupStrainers."""
        strainer = SoupStrainer("b")
        soup = self.soup("A bold  statement", parse_only=strainer)
        assert soup.decode() == "bold"

    def test_single_quote_attribute_values_become_double_quotes(self):
        self.assert_soup("", '')

    def test_attribute_values_with_nested_quotes_are_left_alone(self):
        text = """a"""
        self.assert_soup(text)

    def test_attribute_values_with_double_nested_quotes_get_quoted(self):
        text = """a"""
        soup = self.soup(text)
        soup.foo["attr"] = 'Brawls happen at "Bob\'s Bar"'
        self.assert_soup(
            soup.foo.decode(),
            """a""",
        )

    def test_ampersand_in_attribute_value_gets_escaped(self):
        self.assert_soup(
            '',
            '',
        )

        self.assert_soup(
            'foo',
            'foo',
        )

    def test_escaped_ampersand_in_attribute_value_is_left_alone(self):
        self.assert_soup('')

    def test_entities_in_strings_converted_during_parsing(self):
        # Both XML and HTML entities are converted to Unicode characters
        # during parsing.
        text = "
<<sacré bleu!>>
"
        expected = (
            "
<<sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>
"
        )
        self.assert_soup(text, expected)

    def test_smart_quotes_converted_on_the_way_in(self):
        # Microsoft smart quotes are converted to Unicode characters during
        # parsing.
        quote = b"
\x91Foo\x92
"
        soup = self.soup(quote, from_encoding="windows-1252")
        assert (
            soup.p.string
            == "\N{LEFT SINGLE QUOTATION MARK}Foo\N{RIGHT SINGLE QUOTATION MARK}"
        )

    def test_non_breaking_spaces_converted_on_the_way_in(self):
        soup = self.soup("  ")
        assert soup.a.string == "\N{NO-BREAK SPACE}" * 2

    def test_entities_converted_on_the_way_out(self):
        text = "
<<sacré bleu!>>
"
        expected = "
<<sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>>
".encode(
            "utf-8"
        )
        soup = self.soup(text)
        assert soup.p.encode("utf-8") == expected

    def test_real_iso_8859_document(self):
        # Smoke test of interrelated functionality, using an
        # easy-to-understand document.

        # Here it is in Unicode. Note that it claims to be in ISO-8859-1.
        unicode_html = '
Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!
'

        # That's because we're going to encode it into ISO-8859-1,
        # and use that to test.
        iso_latin_html = unicode_html.encode("iso-8859-1")

        # Parse the ISO-8859-1 HTML.
        soup = self.soup(iso_latin_html)

        # Encode it to UTF-8.
        result = soup.encode("utf-8")

        # What do we expect the result to look like? Well, it would
        # look like unicode_html, except that the META tag would say
        # UTF-8 instead of ISO-8859-1.
        expected = unicode_html.replace("ISO-8859-1", "utf-8")

        # And, of course, it would be in UTF-8, not Unicode.
        expected = expected.encode("utf-8")

        # Ta-da!
        assert result == expected

    def test_real_shift_jis_document(self):
        # Smoke test to make sure the parser can handle a document in
        # Shift-JIS encoding, without choking.
        shift_jis_html = (
            b"
"
            b"\x82\xb1\x82\xea\x82\xcdShift-JIS\x82\xc5\x83R\x81[\x83f"
            b"\x83B\x83\x93\x83O\x82\xb3\x82\xea\x82\xbd\x93\xfa\x96{\x8c"
            b"\xea\x82\xcc\x83t\x83@\x83C\x83\x8b\x82\xc5\x82\xb7\x81B"
            b"
"
        )
        unicode_html = shift_jis_html.decode("shift-jis")
        soup = self.soup(unicode_html)

        # Make sure the parse tree is correctly encoded to various
        # encodings.
        assert soup.encode("utf-8") == unicode_html.encode("utf-8")
        assert soup.encode("euc_jp") == unicode_html.encode("euc_jp")

    def test_real_hebrew_document(self):
        # A real-world test to make sure we can convert ISO-8859-9 (a
        # Hebrew encoding) to UTF-8.
        hebrew_document = b"Hebrew (ISO 8859-8) in Visual Directionality
Hebrew (ISO 8859-8) in Visual Directionality
\xed\xe5\xec\xf9"
        soup = self.soup(hebrew_document, from_encoding="iso8859-8")
        # Some tree builders call it iso8859-8, others call it iso-8859-9.
        # That's not a difference we really care about.
        assert soup.original_encoding in ("iso8859-8", "iso-8859-8")
        assert soup.encode("utf-8") == (
            hebrew_document.decode("iso8859-8").encode("utf-8")
        )

    def test_meta_tag_reflects_current_encoding(self):
        # Here's the  tag saying that a document is
        # encoded in Shift-JIS.
        meta_tag = (
            ''
        )

        # Here's a document incorporating that meta tag.
        shift_jis_html = (
            "\n%s\n"
            ''
            "Shift-JIS markup goes here."
        ) % meta_tag
        soup = self.soup(shift_jis_html)

        # Parse the document, and the charset is seemingly unaffected.
        parsed_meta = soup.find("meta", {"http-equiv": "Content-type"})
        content = parsed_meta["content"]
        assert "text/html; charset=x-sjis" == content

        # But that value is actually a ContentMetaAttributeValue object.
        assert isinstance(content, ContentMetaAttributeValue)

        # And it will take on a value that reflects its current
        # encoding.
        assert "text/html; charset=utf8" == content.substitute_encoding("utf8")

        # No matter how the  tag is encoded, its charset attribute
        # will always be accurate.
        assert b"charset=utf8" in parsed_meta.encode("utf8")
        assert b"charset=shift-jis" in parsed_meta.encode("shift-jis")

        # For the rest of the story, see TestSubstitutions in
        # test_tree.py.

    def test_html5_style_meta_tag_reflects_current_encoding(self):
        # Here's the  tag saying that a document is
        # encoded in Shift-JIS.
        meta_tag = ''

        # Here's a document incorporating that meta tag.
        shift_jis_html = (
            "\n%s\n"
            ''
            "Shift-JIS markup goes here."
        ) % meta_tag
        soup = self.soup(shift_jis_html)

        # Parse the document, and the charset is seemingly unaffected.
        parsed_meta = soup.find("meta", id="encoding")
        charset = parsed_meta["charset"]
        assert "x-sjis" == charset

        # But that value is actually a CharsetMetaAttributeValue object.
        assert isinstance(charset, CharsetMetaAttributeValue)

        # And it will take on a value that reflects its current
        # encoding.
        assert "utf8" == charset.substitute_encoding("utf8")

        # No matter how the  tag is encoded, its charset attribute
        # will always be accurate.
        assert b'charset="utf8"' in parsed_meta.encode("utf8")
        assert b'charset="shift-jis"' in parsed_meta.encode("shift-jis")

    def test_python_specific_encodings_not_used_in_charset(self):
        # You can encode an HTML document using a Python-specific
        # encoding, but that encoding won't be mentioned _inside_ the
        # resulting document. Instead, the document will appear to
        # have no encoding.
        for markup in [
            b'' b''
        ]:
            soup = self.soup(markup)
            for encoding in PYTHON_SPECIFIC_ENCODINGS:
                if encoding in (
                    "idna",
                    "mbcs",
                    "oem",
                    "undefined",
                    "string_escape",
                    "string-escape",
                ):
                    # For one reason or another, these will raise an
                    # exception if we actually try to use them, so don't
                    # bother.
                    continue
                encoded = soup.encode(encoding)
                assert b'meta charset=""' in encoded
                assert encoding.encode("ascii") not in encoded

    def test_tag_with_no_attributes_can_have_attributes_added(self):
        data = self.soup("text")
        data.a["foo"] = "bar"
        assert 'text' == data.a.decode()

    def test_closing_tag_with_no_opening_tag(self):
        # Without BeautifulSoup.open_tag_counter, the  tag will
        # cause _popToTag to be called over and over again as we look
        # for a  tag that wasn't there. The result is that 'text2'
        # will show up outside the body of the document.
        soup = self.soup("
text1
text2
")
        assert "
text1
text2
" == soup.body.decode()

    def test_not_really_a_namespace(self):
        # There's some markup here that looks a bit like a namespace ({value: 20000.00}) but isn't.
        markup = ''
        soup = self.soup(markup)

        # We don't care what the parser makes of this, but it needs to be able to parse this markup without crashing.
        soup.body.decode()

    def test_worst_case(self):
        """Test the worst case (currently) for linking issues."""

        soup = self.soup(BAD_DOCUMENT)
        self.linkage_validator(soup)


class XMLTreeBuilderSmokeTest(TreeBuilderSmokeTest):
    def test_pickle_and_unpickle_identity(self):
        # Pickling a tree, then unpickling it, yields a tree identical
        # to the original.
        tree = self.soup("foo")
        dumped = pickle.dumps(tree, 2)
        loaded = pickle.loads(dumped)
        assert loaded.__class__ == BeautifulSoup
        assert loaded.decode() == tree.decode()

    def test_docstring_generated(self):
        soup = self.soup("")
        assert soup.encode() == b'\n'

    def test_xml_declaration(self):
        markup = b"""\n"""
        soup = self.soup(markup)
        assert markup == soup.encode("utf8")

    def test_python_specific_encodings_not_used_in_xml_declaration(self):
        # You can encode an XML document using a Python-specific
        # encoding, but that encoding won't be mentioned _inside_ the
        # resulting document.
        markup = b"""\n"""
        soup = self.soup(markup)
        for encoding in PYTHON_SPECIFIC_ENCODINGS:
            if encoding in (
                "idna",
                "mbcs",
                "oem",
                "undefined",
                "string_escape",
                "string-escape",
            ):
                # For one reason or another, these will raise an
                # exception if we actually try to use them, so don't
                # bother.
                continue
            encoded = soup.encode(encoding)
            assert b'' in encoded
            assert encoding.encode("ascii") not in encoded

    def test_processing_instruction(self):
        markup = b"""\n"""
        soup = self.soup(markup)
        assert markup == soup.encode("utf8")

    def test_real_xhtml_document(self):
        """A real XHTML document should come out *exactly* the same as it went in."""
        markup = b"""


Hello.
Goodbye.
"""
        soup = self.soup(markup)
        assert soup.encode("utf-8") == markup

    def test_nested_namespaces(self):
        doc = b"""





"""
        soup = self.soup(doc)
        assert doc == soup.encode()

    def test_formatter_processes_script_tag_for_xml_documents(self):
        doc = """
  
"""
        soup = BeautifulSoup(doc, "lxml-xml")
        # lxml would have stripped this while parsing, but we can add
        # it later.
        soup.script.string = 'console.log("< < hey > > ");'
        encoded = soup.encode()
        assert b"< < hey > >" in encoded

    def test_can_parse_unicode_document(self):
        markup = 'Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!'
        soup = self.soup(markup)
        assert "Sacr\xe9 bleu!" == soup.root.string

    def test_can_parse_unicode_document_begining_with_bom(self):
        markup = '\N{BYTE ORDER MARK}Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!'
        soup = self.soup(markup)
        assert "Sacr\xe9 bleu!" == soup.root.string

    def test_popping_namespaced_tag(self):
        markup = 'b2012-07-02T20:33:42Zcd'
        soup = self.soup(markup)
        assert str(soup.rss) == markup

    def test_docstring_includes_correct_encoding(self):
        soup = self.soup("")
        assert (
            soup.encode("latin1") == b'\n'
        )

    def test_large_xml_document(self):
        """A large XML document should come out the same as it went in."""
        markup = (
            b'\n'
            + b"0" * (2**12)
            + b""
        )
        soup = self.soup(markup)
        assert soup.encode("utf-8") == markup

    def test_tags_are_empty_element_if_and_only_if_they_are_empty(self):
        self.assert_soup("
", "
")
        self.assert_soup("
foo
")

    def test_namespaces_are_preserved(self):
        markup = 'This tag is in the a namespaceThis tag is in the b namespace'
        soup = self.soup(markup)
        root = soup.root
        assert "http://example.com/" == root["xmlns:a"]
        assert "http://example.net/" == root["xmlns:b"]

    def test_closing_namespaced_tag(self):
        markup = '
20010504
'
        soup = self.soup(markup)
        assert str(soup.p) == markup

    def test_namespaced_attributes(self):
        markup = ''
        soup = self.soup(markup)
        assert str(soup.foo) == markup

    def test_namespaced_attributes_xml_namespace(self):
        markup = 'bar'
        soup = self.soup(markup)
        assert str(soup.foo) == markup

    def test_find_by_prefixed_name(self):
        doc = """

    foo
    bar
    baz

"""
        soup = self.soup(doc)

        # There are three  tags.
        assert 3 == len(soup.find_all("tag"))

        # But two of them are ns1:tag and one of them is ns2:tag.
        assert 2 == len(soup.find_all("ns1:tag"))
        assert 1 == len(soup.find_all("ns2:tag"))

        assert 1, len(soup.find_all("ns2:tag", key="value"))
        assert 3, len(soup.find_all(["ns1:tag", "ns2:tag"]))

    def test_copy_tag_preserves_namespace(self):
        xml = """
"""

        soup = self.soup(xml)
        tag = soup.document
        duplicate = copy.copy(tag)

        # The two tags have the same namespace prefix.
        assert tag.prefix == duplicate.prefix

    def test_worst_case(self):
        """Test the worst case (currently) for linking issues."""

        soup = self.soup(BAD_DOCUMENT)
        self.linkage_validator(soup)


class HTML5TreeBuilderSmokeTest(HTMLTreeBuilderSmokeTest):
    """Smoke test for a tree builder that supports HTML5."""

    def test_real_xhtml_document(self):
        # Since XHTML is not HTML5, HTML5 parsers are not tested to handle
        # XHTML documents in any particular way.
        pass

    def test_html_tags_have_namespace(self):
        markup = ""
        soup = self.soup(markup)
        assert "http://www.w3.org/1999/xhtml" == soup.a.namespace

    def test_svg_tags_have_namespace(self):
        markup = ""
        soup = self.soup(markup)
        namespace = "http://www.w3.org/2000/svg"
        assert namespace == soup.svg.namespace
        assert namespace == soup.circle.namespace

    def test_mathml_tags_have_namespace(self):
        markup = "5"
        soup = self.soup(markup)
        namespace = "http://www.w3.org/1998/Math/MathML"
        assert namespace == soup.math.namespace
        assert namespace == soup.msqrt.namespace

    def test_xml_declaration_becomes_comment(self):
        markup = ''
        soup = self.soup(markup)
        assert isinstance(soup.contents[0], Comment)
        assert soup.contents[0] == '?xml version="1.0" encoding="utf-8"?'
        assert "html" == soup.contents[0].next_element.name
beautifulsoup4-4.14.0/bs4/tests/test_builder.py0000644000000000000000000000212513615410400016364 0ustar00import pytest # type:ignore
from unittest.mock import patch
from bs4.builder import DetectsXMLParsedAsHTML


class TestDetectsXMLParsedAsHTML:
    @pytest.mark.parametrize(
        "markup,looks_like_xml",
        [
            ("No xml declaration", False),
            ("obviously HTMLActually XHTML", False),
            ("            <    html>Tricky XHTML", False),
            ("", True),
        ],
    )
    def test_warn_if_markup_looks_like_xml(self, markup, looks_like_xml):
        # Test of our ability to guess at whether markup looks XML-ish
        # _and_ not HTML-ish.
        with patch("bs4.builder.DetectsXMLParsedAsHTML._warn") as mock:
            for data in markup, markup.encode("utf8"):
                result = DetectsXMLParsedAsHTML.warn_if_markup_looks_like_xml(data)
                assert result == looks_like_xml
                if looks_like_xml:
                    assert mock.called
                else:
                    assert not mock.called
                mock.reset_mock()
beautifulsoup4-4.14.0/bs4/tests/test_builder_registry.py0000644000000000000000000001172613615410400020323 0ustar00"""Tests of the builder registry."""

import warnings
from typing import Type
import pytest # type:ignore

from bs4 import BeautifulSoup
from bs4.builder import (
    builder_registry as registry,
    TreeBuilder,
    TreeBuilderRegistry,
)
from bs4.builder._htmlparser import HTMLParserTreeBuilder

from . import (
    HTML5LIB_PRESENT,
    LXML_PRESENT,
)

if HTML5LIB_PRESENT:
    from bs4.builder._html5lib import HTML5TreeBuilder

if LXML_PRESENT:
    from bs4.builder._lxml import (
        LXMLTreeBuilderForXML,
        LXMLTreeBuilder,
    )


# TODO: Split out the lxml and html5lib tests into their own classes
# and gate with pytest.mark.skipIf.
class TestBuiltInRegistry(object):
    """Test the built-in registry with the default builders registered."""

    def test_combination(self):
        assert registry.lookup("strict", "html") == HTMLParserTreeBuilder
        if LXML_PRESENT:
            assert registry.lookup("fast", "html") == LXMLTreeBuilder
            assert registry.lookup("permissive", "xml") == LXMLTreeBuilderForXML
        if HTML5LIB_PRESENT:
            assert registry.lookup("html5lib", "html") == HTML5TreeBuilder

    def test_lookup_by_markup_type(self):
        if LXML_PRESENT:
            assert registry.lookup("html") == LXMLTreeBuilder
            assert registry.lookup("xml") == LXMLTreeBuilderForXML
        else:
            assert registry.lookup("xml") is None
            if HTML5LIB_PRESENT:
                assert registry.lookup("html") == HTML5TreeBuilder
            else:
                assert registry.lookup("html") == HTMLParserTreeBuilder

    def test_named_library(self):
        if LXML_PRESENT:
            assert registry.lookup("lxml", "xml") == LXMLTreeBuilderForXML
            assert registry.lookup("lxml", "html") == LXMLTreeBuilder
        if HTML5LIB_PRESENT:
            assert registry.lookup("html5lib") == HTML5TreeBuilder

        assert registry.lookup("html.parser") == HTMLParserTreeBuilder

    def test_beautifulsoup_constructor_does_lookup(self):
        with warnings.catch_warnings(record=True):
            # This will create a warning about not explicitly
            # specifying a parser, but we'll ignore it.

            # You can pass in a string.
            BeautifulSoup("", features="html")
            # Or a list of strings.
            BeautifulSoup("", features=["html", "fast"])
            pass

        # You'll get an exception if BS can't find an appropriate
        # builder.
        with pytest.raises(ValueError):
            BeautifulSoup("", features="no-such-feature")


class TestRegistry(object):
    """Test the TreeBuilderRegistry class in general."""

    def setup_method(self):
        self.registry = TreeBuilderRegistry()

    def builder_for_features(self, *feature_list: str) -> Type[TreeBuilder]:
        cls = type(
            "Builder_" + "_".join(feature_list), (object,), {"features": feature_list}
        )

        self.registry.register(cls)
        return cls

    def test_register_with_no_features(self):
        builder = self.builder_for_features()

        # Since the builder advertises no features, you can't find it
        # by looking up features.
        assert self.registry.lookup("foo") is None

        # But you can find it by doing a lookup with no features, if
        # this happens to be the only registered builder.
        assert self.registry.lookup() == builder

    def test_register_with_features_makes_lookup_succeed(self):
        builder = self.builder_for_features("foo", "bar")
        assert self.registry.lookup("foo") is builder
        assert self.registry.lookup("bar") is builder

    def test_lookup_fails_when_no_builder_implements_feature(self):
        assert self.registry.lookup("baz") is None

    def test_lookup_gets_most_recent_registration_when_no_feature_specified(self):
        self.builder_for_features("foo")
        builder2 = self.builder_for_features("bar")
        assert self.registry.lookup() == builder2

    def test_lookup_fails_when_no_tree_builders_registered(self):
        assert self.registry.lookup() is None

    def test_lookup_gets_most_recent_builder_supporting_all_features(self):
        self.builder_for_features("foo")
        self.builder_for_features("bar")
        has_both_early = self.builder_for_features("foo", "bar", "baz")
        has_both_late = self.builder_for_features("foo", "bar", "quux")
        self.builder_for_features("bar")
        self.builder_for_features("foo")

        # There are two builders featuring 'foo' and 'bar', but
        # the one that also features 'quux' was registered later.
        assert self.registry.lookup("foo", "bar") == has_both_late

        # There is only one builder featuring 'foo', 'bar', and 'baz'.
        assert self.registry.lookup("foo", "bar", "baz") == has_both_early

    def test_lookup_fails_when_cannot_reconcile_requested_features(self):
        self.builder_for_features("foo", "bar")
        self.builder_for_features("foo", "baz")
        assert self.registry.lookup("bar", "baz") is None
beautifulsoup4-4.14.0/bs4/tests/test_css.py0000644000000000000000000004431713615410400015537 0ustar00import types
import pytest # type:ignore

from bs4 import (
    BeautifulSoup,
    ResultSet,
)

from typing import (
    Any,
    List,
    Tuple,
    Type,
)

from packaging.version import Version

from . import (
    SoupTest,
    SOUP_SIEVE_PRESENT,
)

SOUPSIEVE_EXCEPTION_ON_UNSUPPORTED_PSEUDOCLASS: Type[Exception]
if SOUP_SIEVE_PRESENT:
    from soupsieve import __version__, SelectorSyntaxError

    # Some behavior changes in soupsieve 2.6 that affects one of our
    # tests.  For the test to run under all versions of Python
    # supported by Beautiful Soup (which includes versions of Python
    # not supported by soupsieve 2.6) we need to check both behaviors.
    SOUPSIEVE_EXCEPTION_ON_UNSUPPORTED_PSEUDOCLASS = SelectorSyntaxError
    if Version(__version__) < Version("2.6"):
        SOUPSIEVE_EXCEPTION_ON_UNSUPPORTED_PSEUDOCLASS = NotImplementedError


@pytest.mark.skipif(not SOUP_SIEVE_PRESENT, reason="Soup Sieve not installed")
class TestCSSSelectors(SoupTest):
    """Test basic CSS selector functionality.

    This functionality is implemented in soupsieve, which has a much
    more comprehensive test suite, so this is basically an extra check
    that soupsieve works as expected.
    """

    HTML = """



The title



Hello there.





An H1


Some text


Some more text


An H2


Another

Bob

Another H2

me

span1a1
span1a2 test

span2a1

















English


English UK


English US


French





"""

    def setup_method(self):
        self._soup = BeautifulSoup(self.HTML, "html.parser")

    def assert_css_selects(
        self, selector: str, expected_ids: List[str], **kwargs: Any
    ) -> None:
        results = self._soup.select(selector, **kwargs)
        assert isinstance(results, ResultSet)
        el_ids = [el["id"] for el in results]
        el_ids.sort()
        expected_ids.sort()
        assert expected_ids == el_ids, "Selector %s, expected [%s], got [%s]" % (
            selector,
            ", ".join(expected_ids),
            ", ".join(el_ids),
        )

    assertSelect = assert_css_selects

    def assert_css_select_multiple(self, *tests: Tuple[str, List[str]]):
        for selector, expected_ids in tests:
            self.assert_css_selects(selector, expected_ids)

    def test_precompiled(self):
        sel = self._soup.css.compile("div")

        els = self._soup.select(sel)
        assert len(els) == 4
        for div in els:
            assert div.name == "div"

        el = self._soup.select_one(sel)
        assert "main" == el["id"]

    def test_one_tag_one(self):
        els = self._soup.select("title")
        assert len(els) == 1
        assert els[0].name == "title"
        assert els[0].contents == ["The title"]

    def test_one_tag_many(self):
        els = self._soup.select("div")
        assert len(els) == 4
        for div in els:
            assert div.name == "div"

        el = self._soup.select_one("div")
        assert "main" == el["id"]

    def test_select_one_returns_none_if_no_match(self):
        match = self._soup.select_one("nonexistenttag")
        assert None is match

    def test_tag_in_tag_one(self):
        self.assert_css_selects("div div", ["inner", "data1"])

    def test_tag_in_tag_many(self):
        for selector in ("html div", "html body div", "body div"):
            self.assert_css_selects(selector, ["data1", "main", "inner", "footer"])

    def test_limit(self):
        self.assert_css_selects("html div", ["main"], limit=1)
        self.assert_css_selects("html body div", ["inner", "main"], limit=2)
        self.assert_css_selects(
            "body div", ["data1", "main", "inner", "footer"], limit=10
        )

    def test_tag_no_match(self):
        assert len(self._soup.select("del")) == 0

    def test_invalid_tag(self):
        with pytest.raises(SelectorSyntaxError):
            self._soup.select("tag%t")

    def test_select_dashed_tag_ids(self):
        self.assert_css_selects("custom-dashed-tag", ["dash1", "dash2"])

    def test_select_dashed_by_id(self):
        dashed = self._soup.select('custom-dashed-tag[id="dash2"]')
        assert dashed[0].name == "custom-dashed-tag"
        assert dashed[0]["id"] == "dash2"

    def test_dashed_tag_text(self):
        assert self._soup.select("body > custom-dashed-tag")[0].text == "Hello there."

    def test_select_dashed_matches_find_all(self):
        assert self._soup.select("custom-dashed-tag") == self._soup.find_all(
            "custom-dashed-tag"
        )

    def test_header_tags(self):
        self.assert_css_select_multiple(
            ("h1", ["header1"]),
            ("h2", ["header2", "header3"]),
        )

    def test_class_one(self):
        for selector in (".onep", "p.onep", "html p.onep"):
            els = self._soup.select(selector)
            assert len(els) == 1
            assert els[0].name == "p"
            assert els[0]["class"] == ["onep"]

    def test_class_mismatched_tag(self):
        els = self._soup.select("div.onep")
        assert len(els) == 0

    def test_one_id(self):
        for selector in ("div#inner", "#inner", "div div#inner"):
            self.assert_css_selects(selector, ["inner"])

    def test_bad_id(self):
        els = self._soup.select("#doesnotexist")
        assert len(els) == 0

    def test_items_in_id(self):
        els = self._soup.select("div#inner p")
        assert len(els) == 3
        for el in els:
            assert el.name == "p"
        assert els[1]["class"] == ["onep"]
        assert not els[0].has_attr("class")

    def test_a_bunch_of_emptys(self):
        for selector in ("div#main del", "div#main div.oops", "div div#main"):
            assert len(self._soup.select(selector)) == 0

    def test_multi_class_support(self):
        for selector in (
            ".class1",
            "p.class1",
            ".class2",
            "p.class2",
            ".class3",
            "p.class3",
            "html p.class2",
            "div#inner .class2",
        ):
            self.assert_css_selects(selector, ["pmulti"])

    def test_multi_class_selection(self):
        for selector in (".class1.class3", ".class3.class2", ".class1.class2.class3"):
            self.assert_css_selects(selector, ["pmulti"])

    def test_child_selector(self):
        self.assert_css_selects(".s1 > a", ["s1a1", "s1a2"])
        self.assert_css_selects(".s1 > a span", ["s1a2s1"])

    def test_child_selector_id(self):
        self.assert_css_selects(".s1 > a#s1a2 span", ["s1a2s1"])

    def test_attribute_equals(self):
        self.assert_css_select_multiple(
            ('p[class="onep"]', ["p1"]),
            ('p[id="p1"]', ["p1"]),
            ('[class="onep"]', ["p1"]),
            ('[id="p1"]', ["p1"]),
            ('link[rel="stylesheet"]', ["l1"]),
            ('link[type="text/css"]', ["l1"]),
            ('link[href="blah.css"]', ["l1"]),
            ('link[href="no-blah.css"]', []),
            ('[rel="stylesheet"]', ["l1"]),
            ('[type="text/css"]', ["l1"]),
            ('[href="blah.css"]', ["l1"]),
            ('[href="no-blah.css"]', []),
            ('p[href="no-blah.css"]', []),
            ('[href="no-blah.css"]', []),
        )

    def test_attribute_tilde(self):
        self.assert_css_select_multiple(
            ('p[class~="class1"]', ["pmulti"]),
            ('p[class~="class2"]', ["pmulti"]),
            ('p[class~="class3"]', ["pmulti"]),
            ('[class~="class1"]', ["pmulti"]),
            ('[class~="class2"]', ["pmulti"]),
            ('[class~="class3"]', ["pmulti"]),
            ('a[rel~="friend"]', ["bob"]),
            ('a[rel~="met"]', ["bob"]),
            ('[rel~="friend"]', ["bob"]),
            ('[rel~="met"]', ["bob"]),
        )

    def test_attribute_startswith(self):
        self.assert_css_select_multiple(
            ('[rel^="style"]', ["l1"]),
            ('link[rel^="style"]', ["l1"]),
            ('notlink[rel^="notstyle"]', []),
            ('[rel^="notstyle"]', []),
            ('link[rel^="notstyle"]', []),
            ('link[href^="bla"]', ["l1"]),
            ('a[href^="http://"]', ["bob", "me"]),
            ('[href^="http://"]', ["bob", "me"]),
            ('[id^="p"]', ["pmulti", "p1"]),
            ('[id^="m"]', ["me", "main"]),
            ('div[id^="m"]', ["main"]),
            ('a[id^="m"]', ["me"]),
            ('div[data-tag^="dashed"]', ["data1"]),
        )

    def test_attribute_endswith(self):
        self.assert_css_select_multiple(
            ('[href$=".css"]', ["l1"]),
            ('link[href$=".css"]', ["l1"]),
            ('link[id$="1"]', ["l1"]),
            (
                '[id$="1"]',
                ["data1", "l1", "p1", "header1", "s1a1", "s2a1", "s1a2s1", "dash1"],
            ),
            ('div[id$="1"]', ["data1"]),
            ('[id$="noending"]', []),
        )

    def test_attribute_contains(self):
        self.assert_css_select_multiple(
            # From test_attribute_startswith
            ('[rel*="style"]', ["l1"]),
            ('link[rel*="style"]', ["l1"]),
            ('notlink[rel*="notstyle"]', []),
            ('[rel*="notstyle"]', []),
            ('link[rel*="notstyle"]', []),
            ('link[href*="bla"]', ["l1"]),
            ('[href*="http://"]', ["bob", "me"]),
            ('[id*="p"]', ["pmulti", "p1"]),
            ('div[id*="m"]', ["main"]),
            ('a[id*="m"]', ["me"]),
            # From test_attribute_endswith
            ('[href*=".css"]', ["l1"]),
            ('link[href*=".css"]', ["l1"]),
            ('link[id*="1"]', ["l1"]),
            (
                '[id*="1"]',
                [
                    "data1",
                    "l1",
                    "p1",
                    "header1",
                    "s1a1",
                    "s1a2",
                    "s2a1",
                    "s1a2s1",
                    "dash1",
                ],
            ),
            ('div[id*="1"]', ["data1"]),
            ('[id*="noending"]', []),
            # New for this test
            ('[href*="."]', ["bob", "me", "l1"]),
            ('a[href*="."]', ["bob", "me"]),
            ('link[href*="."]', ["l1"]),
            ('div[id*="n"]', ["main", "inner"]),
            ('div[id*="nn"]', ["inner"]),
            ('div[data-tag*="edval"]', ["data1"]),
        )

    def test_attribute_exact_or_hypen(self):
        self.assert_css_select_multiple(
            ('p[lang|="en"]', ["lang-en", "lang-en-gb", "lang-en-us"]),
            ('[lang|="en"]', ["lang-en", "lang-en-gb", "lang-en-us"]),
            ('p[lang|="fr"]', ["lang-fr"]),
            ('p[lang|="gb"]', []),
        )

    def test_attribute_exists(self):
        self.assert_css_select_multiple(
            ("[rel]", ["l1", "bob", "me"]),
            ("link[rel]", ["l1"]),
            ("a[rel]", ["bob", "me"]),
            ("[lang]", ["lang-en", "lang-en-gb", "lang-en-us", "lang-fr"]),
            ("p[class]", ["p1", "pmulti"]),
            ("[blah]", []),
            ("p[blah]", []),
            ("div[data-tag]", ["data1"]),
        )

    def test_quoted_space_in_selector_name(self):
        html = """
nope

        
yes

        """
        soup = BeautifulSoup(html, "html.parser")
        [chosen] = soup.select('div[style="display: right"]')
        assert "yes" == chosen.string

    def test_unsupported_pseudoclass(self):
        with pytest.raises(SOUPSIEVE_EXCEPTION_ON_UNSUPPORTED_PSEUDOCLASS):
            self._soup.select("a:no-such-pseudoclass")

        with pytest.raises(SelectorSyntaxError):
            self._soup.select("a:nth-of-type(a)")

    def test_nth_of_type(self):
        # Try to select first paragraph
        els = self._soup.select("div#inner p:nth-of-type(1)")
        assert len(els) == 1
        assert els[0].string == "Some text"

        # Try to select third paragraph
        els = self._soup.select("div#inner p:nth-of-type(3)")
        assert len(els) == 1
        assert els[0].string == "Another"

        # Try to select (non-existent!) fourth paragraph
        els = self._soup.select("div#inner p:nth-of-type(4)")
        assert len(els) == 0

        # Zero will select no tags.
        els = self._soup.select("div p:nth-of-type(0)")
        assert len(els) == 0

    def test_nth_of_type_direct_descendant(self):
        els = self._soup.select("div#inner > p:nth-of-type(1)")
        assert len(els) == 1
        assert els[0].string == "Some text"

    def test_id_child_selector_nth_of_type(self):
        self.assert_css_selects("#inner > p:nth-of-type(2)", ["p1"])

    def test_select_on_element(self):
        # Other tests operate on the tree; this operates on an element
        # within the tree.
        inner = self._soup.find("div", id="main")
        selected = inner.select("div")
        # The 
 tag was selected. The