Test if target is a valid host

[J0WI]

See #10869

[cschanaj]

@J0WI thank you for opening this issue. I would like to add some details here.

In #10869 I checked if a target is valid by matching it against the Public Suffix List. While this test does not guarantee each target has a working DNS record, it has the advantage that no network request is required to perform the checking.

It was found that AM-Best-Company.xml#L11 with a invalid target passed the comprehensive fetch test and got re-activated in #10538. Despite @Bisaloo's assertion, I would like to hear from @Hainish for an explaination.

[cschanaj]

@Hainish @J0WI I have implemented the logic which check each target is having a valid gTLD/ ccTLD in the rule-test in #11187. Besides, I noted that the following type of rewrite will pass the fetch-test without any Travis complaint

<ruleset name="cschanaj">
  <target host="dns-record-does-not-exist.example.com" />
  
  <rule from="^http://dns-record-does-not-exist\.example\.com/"
          to="https://example.com/" />
</ruleset>

Is this an intended behavior? Should we check each target for an existing DNS record? See cschanaj/https-everywhere/pull/6

Besides, should we allow IP target as in Xxxbunker.com.xml#L44?

Please be noted that the code does not compile currently.

[cschanaj]

@Hainish can you have a look on #10877 (comment) and #11187 ?

[Hainish]

Not quite sure how AM-Best-Company.xml#L11 got re-enabled. Checking targets against the Public Suffix List seems reasonable.

[cschanaj]

@Hainish For any target if HTTPS connection works, the fetch test will ignore any result for the HTTP connection, see check_rules.py#L131. In particular, it will ignore the could not resolve host error. As a result, any rewrite in the form of #10877 (comment) will always pass the fetch test.

It seems to be a bug to me. A quick fix is to throw an error if HTTP cannot be resolved. Can you confirm this? Besides, what should be done for ip target as in Xxxbunker.com.xml#L44?

[cschanaj]

All ruleset in the code base will pass the proposed rule-test once #11263 is merged, it will be also safe to merge #11187 (hopefully before next release). See also #11264.

Updated #11187 to include the changes in #11264. thanks!