DOC: Please describe all arguments so their functionality is clearly understood.

[T145]

Example questions include:

• What is the "mining subsystem?"
• What are the details regarding the "Continuous Integration mechanism" and how is it configured?
• How is a source determined to ever be "MALICIOUS" or "SANE?"
• Is there a way to disable "splitted" output to prevent data duplication?
• What are "Analytic" files supposed to represent?

[T145]

1. I'm aware of what CI is, and am primarily asking to see if the documentation fully answers what PyFunceble should be doing in a CI environment. After reading the raw config file it becomes obvious, but the documentation doesn't seem to flush it out as well.
2. I'm not sure what you mean. There's a malicious and sane output type that never seems to have anything generated in it, so this question is specifically meant to expand on what they are.
3. (/4) From your Analytic description, it sounds like splitted already does exactly that and adds more details. It seems like removing Analytic output altogether and renaming splitted to analytic would be a simple solution. Then the analytic config option would control splitted generation.

These questions are just examples to encourage thought behind how the documentation explains PyFunceble's functionality. It appears that there are options that serve more than one purpose and/or deliver multiple output artifacts, in which case what goes on needs more detail than what could be read from the variable name.

[funilrys]

Hi @T145 Thanks for the feedback. I plan to improve the documentation for the next maintenance session starting with your bullet points.

As English isn't my primary language please excuse the unclarities.

---

What is the "mining subsystem?"

It is "documented" here.

It is called "mining" for historical reason. Back in 2017 @mitchellkrogza and I talked about what we called a "mining" mechanism for funceble.
The principle was simple: When active, PyFunceble will spawn some extra processes that will be fed with all the ACTIVE domains caught by the main processes. The mining process consists of registering, following, storing, and testing all domains in an HTTP redirection path.

What does it mean?

Let's assume the following:

https://example.com --> https://example.org/awesome --> https://example.de/sleep --> https://example.co.jp/awake

Without the "mining" mechanism you won't even know the existence of example.org, example.de and example.co.jp. With the mining mechanism, PyFunceble will test them and therefore produce outputs with them. In a highly automated environment, this gave us the ability to keep track of moving or new redirections behind really bad actors - back then.

---

What are the details regarding the "Continuous Integration mechanism" and how is it configured?

It is "documented" here.

This is a feature that is only relevant if you run PyFunceble within the following "CI Engines":

• TravisCI
• JenkinsCI
• GitHub Actions
• GitLab CI/CD

If PyFunceble runs within one of those "engines" and the --ci option or its configuration counterpart is active, PyFunceble will run for around 30 minutes before stopping and committing the filesystem changes to your Git Repository. If used along the auto continue and a database, the next "run" after the commit will restart PyFunceble with the same parameter. But on the 2nd execution, PyFunceble will just try to continue where it stopped. It's a tricky feature but when your CI engine is under your control, this is a nice feature 😄

---

How is a source determined to ever be "MALICIOUS" or "SANE?"

That's indeed something that is not "deeply" documented. My excuse for this. It all started with the #68 issue.

A source is MALICIOUS when listed in Alienvault's IP Reputation dataset.

---

Is there a way to disable "splitted" output to prevent data duplication?

Yes! You can use the --unified-results or its configuration counterpart.

This is something I really need to do better ... There is room for improvement on that side.

---

What are "Analytic" files supposed to represent?

That's also something that isn't documented enough ... It's another historical "feature".

Back in 2017-2018, I was really biased when it came to the status of the subject. I indeed was following RFCs as much as possible. But as soon as I started to work with @mitchellkrogza on bigger projects we soon realized that the technical power of mankind surpasses the power of the most powerful RFCs.
Most people think that the worldwide DNS infrastructure follows the RFCs but we had our fun back then.... When cross-checking the results of Py-Funceble with some of our very own "human inputs" we noticed for example that around ~100 domains didn't have NS records but still had A records .... That's actually impossible you may think ... Well, that's the kind of fun we had back then ...

Therefore, PyFunceble tries to follow the RFCs, but when it doesn't, a record within the analytic output folder is generated so that "humans" are informed of something that PyFunceble found "weird" but is somehow ACTIVE or INACTIVE.