Module launched via `fuchsia-franzisrunner`¶

Glossary¶

Endpoint — an HTTP request description in the extended HAR format.
Tag — an entity described by a label (format — string) and attributes (format — object in JSON notation). It is the result of the module operation.
Vulnerability report — a tag with attributes matching the JSON schema.

`fuchsia-franzisrunner` component¶

The fuchsia-franzisrunner component is a software component whose tasks are to launch Fuchsia scan modules and write tags to the database. Launched modules are indifferent to the database schema, but must follow the fuchsia-franzisrunner communication protocol described below to interact correctly.

Communication protocol between `fuchsia-franzisrunner` and the launched module¶

Module input data¶

When the module is started, the fuchsia-franzisrunner component transmits input data to the module by writing it to the stdin standard input stream.

Acceptable input data:

endpoint in JSON format;
tag from any other module in JSON format;
resource URL in string format.

Note

The fuchsia-franzisrunner component starts one module process and passes only one endpoint to it as input. If N endpoints need to be passed (N > 0), fuchsia-franzisrunner will start N module processes and pass one endpoint to each of them.

Each entity from the list above corresponds to a mode of operation, which is selected using the -mode flag. To get a sample of input data with more complex conditions, filters can be used via the -filter flag. The -label flag is used to determine the tags to be received (filtering by the label with which the tag was written to the database).

The default values for the flags used are shown in the table below.

Flag	Default value
`-mode`	`deps`
`-filter`	`none`
`-label`	not set

For more information about the operating modes and filters used, see the table below.

Operating mode (`-mode`)	Filter (`-filter`)	Data transmitted to the module
`deps`	`none`	All found endpoints in JSON format
	`tagged`	All found endpoints in JSON format that have a tag with a label corresponding to the value of the passed `-label` flag
	`associated-by-label`	All found endpoints in JSON format associated with a web resource that has a tag with a label corresponding to the value of the passed `-label` flag
`pages`	`none`	All found URLs of web pages in string representation, additionally filtered by the `Content-Type` header value (corresponds to the `text/html` or `application/xhtml+xml`)
	`root`	Only the root URL of the scanned application
	`all`	All found web page URLs in string representation, without additional filtering by the `Content-Type` header value
	`tagged`	All found web page URLs in string representation that have a tag with a label corresponding to the value of the passed `-label` flag
`tags`	`none`	Attributes of tags in JSON format that have a label corresponding to the value of the passed `-label` flag

Example¶

Note

In the following examples ... replaces the set of available flags, which is optional for specific examples. first-scanner is the value of the -label flag.

The module is designed to operate on all found endpoints:
```
fuchsia-franzisrunner -mode deps -filter none ...
```
This is equivalent to the example below due to the use of default values:
```
fuchsia-franzisrunner ...
```

The module is designed to operate only on the root URL of the scanned application:
```
fuchsia-franzisrunner -mode pages -filter root ...
```

The module is designed to operate on all found tags received from another module:
```
fuchsia-franzisrunner -mode tags -filter none -label first-scanner ...
```
This is equivalent to the example below due to the use of default values:
```
fuchsia-franzisrunner -mode tags -label first-scanner ...
```

The module is designed to operate only on endpoints that have a tag from another module:
```
fuchsia-franzisrunner -mode deps -filter tagged -label first-scanner ...
```

The module is designed to operate on all found URLs of web pages, filtered by Content-Type:
```
fuchsia-franzisrunner -mode pages -filter none ...
```
This is equivalent to the example below due to the use of default values:
```
fuchsia-franzisrunner -mode pages ...
```

Module output data¶

For correct operation, the module must write only the result of its work to the standard output stream stdout. All logs and other output should be sent to the standard error output stream stderr. The logs written to stderr are stored separately. They can be fetched using the fuchsiactl logs command.

The module should write the result of its operation in NDJSON format (a stream of JSON objects separated by a line break). The result can be:

empty output;
one JSON object;
multiple JSON objects separated by a line break.

To further identify the results of the module's operation, labels obtained with the -label flag are used.

If all module results can be perceived in the same way, the label is set with the -scanner flag.
If one module can generate results with different labels, the -labelsFromScanner flag and a special JSON tag format must be used.

The -scanner flag must be set in both cases.

Examples¶

The running module writes tags with the same example-scanner labels.

The launch will look like this:

fuchsia-franzisrunner -scanner example-scanner ...

The scanner can send the following set of tags to the standard output stream:

{"type": "IssueFound", "issue": "...", "url": "https://example.com/path1"}
{"type": "IssueFound", "issue": "...", "url": "https://example.com/path2"}

The module will result in two tags with label=example-scanner being written to the database.

Attributes of the first tag will include the object:

{"type": "IssueFound", "issue": "...", "url": "https://example.com/path1"}

Attributes of the second tag will include the object:

{"type": "IssueFound", "issue": "...", "url": "https://example.com/path2"}

The running module writes tags with different labels.

The launch will look like this:

fuchsia-franzisrunner -scanner example-scanner -labelsFromScanner ...

The scanner can send the following set of tags in a special format to the standard output stream:

{"label": "example-scanner1", "attributes": {"type": "IssueFound", "issue": "...", "url": "https://example.com/path1"}}
{"label": "example-scanner2", "attributes": {"type": "IssueFound", "issue": "...", "url": "https://example.com/path2"}}

The module will result in the following being written to the database:

one tag with label'=example-scanner1', whose attributes will be the object:

{"type": "IssueFound", "issue": "...", "url": "https://example.com/path1"}

one tag with label'=example-scanner2', whose attributes will be the object:

{"type": "IssueFound", "issue": "...", "url": "https://example.com/path2"}

If the result needs to be displayed in the “Issues” section of the dashboard, then the tag's label should be the module name, and it's attributes should be a JSON object corresponding to the schema. If correct display is not necessary (for example, if the module stores some intermediate information for the next module), any valid JSON object can be used as the tag attributes format.

`fuchsia-franzisrunner` flags¶

The full list of flags can be found by running the fuchsia-franzisrunner -help command. This section will describe the main flags.

-mode — the format of input data to be sent to the module. The following modes are currently supported:
- deps — all found endpoints are fed to the module as input data;
- tags — tags from the database saved by other modules are given as input data. When using this mode, you must also use the -label flag to specify the label of the module, the tags of which will be used as input data. For example, a module has written a tag in {"key": "value"} format with the test-scanner label. If the module uses these tags as input, running fuchsia-franzisrunner would look like this:
```
fuchsia-franzisrunner -mode tags -label test-scanner ...
```
- pages — URLs of found web pages are given to the module as input data. Resources are filtered by the Content-Type header value. In this mode, only resources with the Content-Type corresponding to text/html or application/xhtml+xml will be used;

-filter — additional conditions regulating which input data will be given to the module. The following modes are currently supported:
- none — no additional conditions;
- root — only get the root URL of the scanned application. Only works with -mode=pages;
- all — get all found web page URLs without additional Content-Type header restrictions. Only works with -mode=pages;
- tagged — get all found endpoints or page URLs that have a tag with a label corresponding to the value of the passed -label flag. Only works with -mode=deps or -mode=pages;
- associated-by-label — get all found endpoints related to a web resource that has a tag with a label corresponding to the value of the passed -label flag. Only works with -mode=deps;

-scanner — the name of the scanner to be run, which will be used as a label when writing the results to the database;
-label — a label for selecting certain conditions (get tags with a certain label; get endpoints that have an tag with a certain label);
-concurrency — the number of module processes running at once (for example, with -concurrency 4, 4 module processes with different inputs will be running simultaneously);
-autocommit — tells fuchsia-franzisrunner to close the database transaction after each recorded tag, rather than at the end of all running module processes;
-labelsFromScanner — informs fuchsia-franzisrunner that the module will record its results with different labels;
-domainList — a list of allowed domains to match relative to the policy selected in -domainScope. If -domainList is empty, it will be automatically set from the domain of the target being scanned;
-pageDeduplication — deduplication of resources on which the module is run (in the pages mode). Enabled by default. To disable it, use the flag with the value false;
-depDeduplication — deduplication of endpoints on which the module is run. Supported values:
- none — deduplication is disabled;
- default — deduplication by normalization;
- extended — advanced deduplication;

-domainScope — describes the startup policies on endpoints based on the domain in the found HTTP request. Valid values:
- any — run on any endpoints regardless of domain;
- same — the domain completely matches one from the allowed list -domainList;
- subdomain — the domain completely matches one from the allowed list or is a subdomain of a domain from the allowed list -domainList;
- secondlevel/second-level — the domain completely matches one from the allowed list, or the second-level domain matches one from the allowed list -domainList. To search for second-level domains, use “Public Suffix List”. It is a default value;

-scanId — the scan number whose entities fuchsia-franzisrunner will work with. If this flag is not specified, all scans available in the database at the time of launch will be used. It is filled in with a templated value;
-config — the path to the fuchsiad configuration file. It is filled in with a templated value.

Environment variables¶

The fuchsia-franzisrunner component passes settings to modules using environment variables:

FUCHSIA_PROXY — HTTP proxy address in ip:port format. It is strongly recommended to use FUCHSIA_PROXY as a proxy, even if the module was not planned to use a proxy. The Fuchsia scanner uses its own proxy that logs requests and performs authorization. If the Upstream Proxy transfer function was used when starting the scan, then requests will be routed correctly when using FUCHSIA_PROXY.
FUCHSIA_RPS_LIMIT — value that limits RPS, if this parameter was set in the scan settings. The Fuchsia scanner's internal proxy handles request limiting on its own (based on this setting). However, if the module performs specific attacks (for example, time-based checks), it is recommended to configure RPS limits to avoid false positives.

Module configuration files¶

For the module to work as part of the Fuchsia scanner, its configuration file must meet certain conditions.

Location of the configuration file¶

If fuchsiad is deployed on the system using deb packages, the module configuration files must be located in the /usr/lib/fuchsia/pipeline.d directory, unless otherwise specified in the scan_pipeline directive of the fuchsiad configuration file.
If fuchsiad is deployed on the system using Docker Compose, files should be stored in /etc/fuchsia/pipeline.d, unless otherwise specified in the scan_pipeline directive of the fuchsiad configuration file.

Once the module's configuration file is in the directory with similar files from other modules, the module must be registered in the fuchsiad pipeline by running the fuchsiactl reload command to update the configuration. To make sure that the module is registered in the pipeline, run the command fuchsiactl list_modules.

Name of the configuration file¶

The file must be named in the format N-scanner-name.yml, where N — module priority, scanner-name — module name. The priority is determined as follows: first come the modules that identify the attack surface (priority up to and including 30), then come the signature analysis modules (priority up to and including 60). Next are the fuzzing modules. For example, endpoint fuzzing modules have a priority of 80. The priority, in addition to determining the launch order, affects the parallelism of the modules being launched. Only modules with the same priority can be run in parallel.

Priority 75 is suggested for modules that use page URLs;
Priority 80 is suggested for modules that use endpoints.

Structure of the configuration file¶

To run the module as part of the Fuchsia scanner, a description of how it should be run must be added to the yaml configuration. It may look like this:

name: requester
args_template:
  [
    "fuchsia-franzisrunner",
    "-config",
    "{{.ConfigPath}}",
    "-scanId",
    "{{.ScanId}}",
    "-scanner",
    "requester",
    "-concurrency",
    "4",
    "--",
    "/usr/bin/requester",
  ]
may_fail: true

The module runs according to the fuchsia-franzisrunner protocol, which makes it easy to integrate into the Fuchsia scanner.

The parameters {{.ConfigPath}} and {{.ScanId}} are templated values that are processed by the Fuchsia scanner using the Go text/template library.

The `name` parameter¶

The name of the module. It may be used in the future, for example, to query the log entries of a specific module.

fuchsiactl logs --scan 100 --modules requester

The `args_template` parameter¶

Parameterized command to start the scan module. According to the fuchsia-franzisrunner protocol, it consists of the fuchsia-franzisrunner launch command, the “--” separator, and the module launch command. Templated values {{.ScanId}} and others can be used in all parts of this command.

A complete list of parameters that can be passed to the module using the Fuchsia scanner is specified in the module configuration file:

{{.Request}} — contains the Scan Request object that is received by the Fuchsia scanner to create a scan. Contains the scan configuration;

Usage example:

[
  "fuchsia-franzisrunner",
  "-config",
  "{{.ConfigPath}}",
  "-scanId",
  "{{.ScanId}}",
  "--",
  "crawler",
  "-dirbusting={{if .Request.GetWithDirbuster}}true{{else}}false{{end}}",
  "--url",
  "{{.URL}}",
]

{{.URL}} — contains the URL where the scan was run;
{{.ScanId}} — contains the ID of the current scan. Required for proper operation of fuchsia-franzisrunner;
{{.ConfigPath}} — contains the path to the fuchsiad configuration file. Required for proper operation of fuchsia-franzisrunner;

{{.Files}} — used to transfer files from the the Fuchsia scanner to the scan module;

Usage example:

[
  "fuchsia-franzisrunner",
  "-scanId",
  "{{.ScanId}}",
  "-config",
  "{{.ConfigPath}}",
  "--",
  "fuchsia-openapi-hars-generator",
  "--mode",
  "module",
  "--spec",
  "{{.Files.OpenAPISpec}}",
]

{{.ExtraSettings}} — used to transfer arbitrary additional parameters to the scan module (check --extra-setting flag of fuchsiactl);

Usage example:

[
  "fuchsia-franzisrunner",
  "-config",
  "{{.ConfigPath}}",
  "-scanId",
  "{{.ScanId}}",
  "-scanner",
  "wpscan-wrapper",
  "--",
  "/usr/bin/wpscan-wrapper",
  "--api-token",
  "{{.ExtraSettings.api_token}}",
]

When a scan is started, an additional variable is given to the input as follows:

fuchsiactl scan --url https://example.com --extra-setting=wpscan-wrapper:api_token=xxxxxxxxx

The `may_fail` parameter¶

If may_fail is false or not set, and the module terminates with a non-zero return code, then the next modules in the queue will not be started, and the status of the entire scan will be erroneous.

The `priority` parameter¶

Lets you specify the priority of a module directly from the configuration file, rather than in its name. Setting the priority from the configuration file is more prioritized than from the configuration file name.

For example, a configuration file named 60-requester.yml has the following contents:

name: requester
args_template:
  [
    "fuchsia-franzisrunner",
    "-scanId",
    "{{.ScanId}}",
    "-config",
    "{{.ConfigPath}}",
    "--",
    "requester",
  ]
priority: 30
may_fail: true

Since the priority parameter in the configuration file has a value of 30, the requester module will have a priority of 30, and the value 60 in the configuration file name will be ignored.

The `condition` parameter¶

The module launch condition. Processed using the Go text/template library in the same context as the args_template parameter. A similar parameter list can be used in the condition. If the condition value is not set, the module will always run. If condition is set, and the result of template processing returns an empty string, the module will not be started.

Usage example:

name: openapi-hars-generator
args_template:
  [
    "fuchsia-franzisrunner",
    "-scanId",
    "{{.ScanId}}",
    "-config",
    "{{.ConfigPath}}",
    "--",
    "fuchsia-openapi-hars-generator",
    "--mode",
    "module",
    "--spec",
    "{{.Files.OpenAPISpec}}",
  ]
condition: '{{ index .Files "OpenAPISpec" }}'
may_fail: true

Scanning module examples¶

A module that accepts endpoints as input¶

A module written in Go

package main

import (
    "crypto/tls"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
)

type KeyValue struct {
    Name        string `json:"name"`
    Value       string `json:"value"`
    Type        string `json:"type,omitempty"`
    Filename    string `json:"filename,omitempty"`
    ContentType string `json:"content_type,omitempty"`
}

type Parameters []*KeyValue

type PostData struct {
    MimeType string     `json:"mimeType"`
    Text     string     `json:"text"`
    Params   Parameters `json:"params,omitempty"`
}

type HAR struct {
    Method      string     `json:"method"`
    URL         string     `json:"url"`
    HTTPVersion string     `json:"httpVersion"`
    Headers     Parameters `json:"headers"`
    QueryString Parameters `json:"queryString"`
    BodySize    int        `json:"bodySize"`
    PostData    *PostData  `json:"postData,omitempty"`
}

type Result struct {
    StatusCode int
    URL        string
}

func main() {
    var proxyAddr string

    if x, ok := os.LookupEnv("FUCHSIA_PROXY"); ok {
        proxyAddr = "http://" + x
    }

    transport := &http.Transport{
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
    }

    if proxyAddr != "" {
        proxy, err := url.ParseRequestURI(proxyAddr)
        if err != nil {
            fmt.Fprintf(os.Stderr, "failed to parse proxy URL: %v\n", err)
            os.Exit(1)
        }
        transport.Proxy = http.ProxyURL(proxy)
    }

    c := http.Client{
        Transport: transport,
    }

    dec := json.NewDecoder(os.Stdin)
    h := HAR{}
    err := dec.Decode(&h)
    if err != nil && err != io.EOF {
        fmt.Fprintf(os.Stderr, "failed to read HAR: %v\n", err)
        os.Exit(1)
    }

    r, err := c.Get(h.URL)
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to create request: %v\n", err)
        os.Exit(1)
    }
    res := &Result{
        StatusCode: r.StatusCode,
        URL:        h.URL,
    }
    reportJson, err := json.Marshal(res)
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to marshal result: %v\n", err)
        os.Exit(1)
    }
    fmt.Println(string(reportJson))
}

The configuration for this module may look like this:

name: requester
args_template:
  [
    "fuchsia-franzisrunner",
    "-config",
    "{{.ConfigPath}}",
    "-scanId",
    "{{.ScanId}}",
    "-scanner",
    "requester",
    "-concurrency",
    "4",
    "--",
    "/usr/bin/requester",
  ]

A module that accepts page URLs as input¶

A module written in JavaScript

const puppeteer = require("puppeteer");

(async () => {
  const readline = require("readline");
  const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
    terminal: false,
  });

  let url;
  rl.on("line", (input) => {
    url = input;
    rl.close();
  });

  rl.on("close", async () => {
    if (!url) {
      console.error("No URL provided");
      process.exit(1);
    }

    const proxy = process.env.FUCHSIA_PROXY;
    let browser;

    try {
      const args = proxy ? [`--proxy-server=${proxy}`] : [];
      browser = await puppeteer.launch({
        headless: "new",
        args: args,
      });

      const page = await browser.newPage();
      const response = await page.goto(url, {
        waitUntil: "networkidle0",
      });

      const result = {
        StatusCode: response.status(),
        URL: url,
      };

      console.log(JSON.stringify(result));
    } catch (err) {
      console.error(`Failed to process the request: ${err.message}`);
      process.exit(1);
    } finally {
      if (browser) {
        await browser.close();
      }
    }
  });
})();

The following open-page Shell script can be used for an easy startup:

#!/usr/bin/env bash

export PUPPETEER_CACHE_DIR=/usr/lib/fuchsia/page-opener/node_modules/puppeteer/.cache/
export PUPPETEER_EXECUTABLE_PATH=/usr/bin/fuchsia-chrome
exec node /usr/lib/fuchsia/page-opener/open-page.js "$@"

The configuration for this module may look like this:

name: page-opener
args_template:
  [
    "fuchsia-franzisrunner",
    "-config",
    "{{.ConfigPath}}",
    "-scanId",
    "{{.ScanId}}",
    "-scanner",
    "requester",
    "-concurrency",
    "4",
    "-mode",
    "pages",
    "--",
    "node",
    "/usr/bin/open-page",
  ]
may_fail: true

Module launched via fuchsia-franzisrunner¶

Glossary¶

fuchsia-franzisrunner component¶

Communication protocol between fuchsia-franzisrunner and the launched module¶

Module input data¶

Example¶

Module output data¶

Examples¶

fuchsia-franzisrunner flags¶

Environment variables¶

Module configuration files¶

Location of the configuration file¶

Name of the configuration file¶

Structure of the configuration file¶

The name parameter¶

The args_template parameter¶

The may_fail parameter¶

The priority parameter¶

The condition parameter¶

Scanning module examples¶

A module that accepts endpoints as input¶

A module that accepts page URLs as input¶

Module launched via `fuchsia-franzisrunner`¶

`fuchsia-franzisrunner` component¶

Communication protocol between `fuchsia-franzisrunner` and the launched module¶

`fuchsia-franzisrunner` flags¶

The `name` parameter¶

The `args_template` parameter¶

The `may_fail` parameter¶

The `priority` parameter¶

The `condition` parameter¶