Add generate-config script and document config options

Dockerfile

@@ -50,7 +50,7 @@ WORKDIR /sydent
 USER sydent:sydent
 VOLUME ["/data"]
 
-RUN python3 /sydent/scripts/generate-key >> /data/sydent.conf
+RUN python3 /sydent/scripts/generate-config
 
 EXPOSE 8090/tcp
 CMD [ "python", "-m", "sydent.sydent" ]

README.rst

@@ -44,18 +44,18 @@ With the virtualenv activated, you can run Sydent using::
 
     python -m sydent.sydent
 
-If this is the first time Sydent is run, then it will generate a configuration file in
-``sydent.conf`` with some defaults and then stop. You must run the generate-key script
-and update the config with this key before Sydent will start.
+Before you run Sydent for the first time, you must run the generate-config script.
 
 You should not write anything in the ``[DEFAULT]`` section. If a
 setting is defined in both the ``[DEFAULT]`` section and another section in the configuration
 file, then the value in the other section is used.
 
 You'll most likely want to change the server name (``server.name``) and specify an email server
-(look for the settings starting with ``email.``).
+(look for the settings starting with ``email.``). 
 
-By default, Sydent will listen on ``0.0.0.0:8090``. This can be changed by changing the values for
+See `<docs/configuration.md>` for more information on how to configure Sydent.
+
+By default, Sydent will listen on ``[::]:8090``. This can be changed by changing the values for
 the configuration settings ``clientapi.http.bind_address`` and ``clientapi.http.port``.
 
 Sydent uses SQLite as its database backend. By default, it will create the database as ``sydent.db``
@@ -109,7 +109,9 @@ However, you then have to pay attention to the file permissions.
 Environment variables
 ---------------------
 
-.. warning:: These variables are only taken into account at first start and are written to the configuration file.
+.. warning:: These variables are taken into account when generate-config is run and
+are written to the config file. They are only used by Sydent if they are not
+overridden by the config file.
 
 +--------------------+-----------------+-----------------------+
 | Variable Name      | Sydent default  | Dockerfile default    |
@@ -190,3 +192,34 @@ Replication
 
 It is possible to configure a mesh of Sydent instances which replicate identity bindings
 between each other. See `<docs/replication.md>`_.
+
+Email obfuscation
+=================
+When a user is invited to a room via their email address, that invite is
+displayed in the room list using an obfuscated version of the user's email
+address.
+
+The amount of obfuscation is controlled by the 
+`email.third_party_invite_username_obfuscate_characters` and
+`email.third_party_invite_domain_obfuscate_characters` config options. The first
+sets the number of characters from the beginning to reveal of the email's username
+portion (left of the '@' sign). The second, the number of characters from the
+beginning to reveal of the email's domain portion (right of the '@' sign).
+
+The '@' sign is always included. 
+
+If the string is longer than a configured limit below, it is truncated to
+that limit with '...' added. For shorter strings, the following rules are
+used:
+
+* If the string has more than 5 characters, it is truncated to 3 characters
+  + '...' (e.g. 'username' would become 'use...')
+
+* If the string has between 2 and 5 characters inclusive, it is truncated
+  to 1 character + '...' (e.g. 'user' would become 'u...')
+
+* If the string is 1 character long, it is converted to just '...'
+  (e.g. 'a' would become '...')
+
+This ensures that a full email address is never shown, even if it is extremely
+short.

changelog.d/404.misc

@@ -0,0 +1 @@
+Replace the CONFIG_DEFAULTS dict with a generate-config script and documentation on all the config options.

docs/configuration.md

@@ -0,0 +1,174 @@
+# Configuration
+
+## Config file structure
+Sydent uses Python's `configparser` libary to parse the config file. 
+This libary's documentation can be found [here](https://docs.python.org/3/library/configparser.html).
+
+As an example:
+```
+[DEFAULT]
+tail.length = 10
+
+[elephant]
+name = Albert
+trunk.length =
+#wings.width =
+
+[mouse]
+name = Terrance
+tail.length = 5
+
+[human]
+```
+
+This config file is then translated to the following Python dict:
+```python
+cfg = {
+    elephant: {
+        name: "Albert",
+        trunk.length: "",
+        tail.length: "10",
+    },
+    mouse: {
+        name: "Terrance",
+        tail.length: "5",
+    },
+    human: {
+        tail.length: "10",
+    }
+}
+```
+
+The important things to note are:  
+1. Commenting out `wings.width` essentially sets it to `None`. But an empty
+value for `trunk.length` sets it to an empty string.
+
+2. Even though the `human` section was empty, it still has a `tail.length`
+value of `"10"` (due to the `DEFAULT` section).  
+
+3. Everything is a string and it's up to the program to decide what type to
+cast things into.
+
+
+## Example config file
+
+An example minimal config file can be found in
+[docs/example_config.conf](https://github.com/matrix-org/sydent/blob/main/docs/example_config.conf).
+
+
+## Writing your own config
+
+Your config file need only contain entries for things you are overwriting from the 
+default. 
+
+The DEFAULT section should be empty and is a deprecated way of configuring Sydent.
+
+You can generate a new config file by running the [generate-config script](https://github.com/matrix-org/sydent/blob/main/scripts/generate-config)
+
+## Config options
+
+The string `XYZ` in an option name means that there is a family of config options,
+each with a different substitution for `XYZ`.
+
+See [docs/example_config.conf](https://github.com/matrix-org/sydent/blob/main/docs/example_config.conf)
+for an example of this in the [sms] section.
+
+### Definitions of emptiness:
+
+Consider this snippet of config:
+```
+this.option.is.empty = 
+this.option.is.non-empty = petal
+```
+* `this.option.is.empty` has been set to the empty string. This is **EMPTY**.
+* `this.option.was.never.mentioned` was never mentioned. This is **UNSET**.
+* `this.option.is.non-empty` is neither **EMPTY** nor **UNSET**. This is **NON-EMPTY**.
+
+Leaving an option *empty*, overrides the default value with an empty string.  
+Leaving an option *unset* uses the defualt value for that option.
+
+### [general]
+
+Name                    | Description
+---------               | -------
+`address_lookup_limit`  | The maximum number of addresses that someone can query in a single /lookup request. DEFAULTS TO `1000`
+`brand.default`         | The default brand of templates to use. See [docs/templates.md](https://github.com/matrix-org/sydent/blob/main/docs/templates.md) for more information. DEFAULTS TO `matrix-org`
+`delete_tokens_on_bind` | Whether to delete invite tokens after successful binding has taken place. DEFAULTS TO `true`
+`enable_v1_associations`| Whether clients and homeservers can register an association using v1 API endpoints. DEFAULTS TO `true`
+`ip.blacklist`          | A comma-seperated list of CIDR IP address ranges to block outbound requests to. DEFAULTS TO a list of private IP ranges to prevent DNS rebinding attacks. This list can be found in [sydent/util/ip_range.py](https://github.com/matrix-org/sydent/blob/main/sydent/util/ip_range.py).
+`ip.whitelist`          | A comma-seperated of IP address CIDR ranges that should be allowed for outbound requests. This is useful for specifying exceptions to wide-ranging blacklisted target IP ranges. This list overrides the blacklist. DEFAULTS TO EMPTY
+`log.level`             | The log level to use. This can be set to any level used by the Python `logging` module. DEFAULTS TO `INFO`
+`log.path`              | The path of the file to write the logs to. Leaving this empty prints logs to stderr. DEFAULTS TO EMPTY
+`pidfile.path`          | The file to save Sydent's process ID (PID) to. DEFAULTS TO the value stored in `SYDENT_PID_FILE` environment variable, or `sydent.pid` if that is unset.
+`prometheus_addr`       | The local IPv4 or IPv6 address for Prometheus to bind to. This must be set to enable the Prometheus client. DEFAULTS TO UNSET
+`prometheus_port`       | The port for Prometheus to bind to. This must be set to enable the Prometheus client. DEFAULTS TO UNSET
+`server.name`           | The domain name of the server. DEFAULTS TO value returned by `os.uname()[1]`
+`sentry.dsn`            | The Data Source Name (DSN) for Sentry to use. This must be set to enable Sentry. DEFAULTS TO UNSET
+`templates.path`        | The path to the root template directory. See [docs/templates.md](https://github.com/matrix-org/sydent/blob/main/docs/templates.md) for more information. DEFAULTS TO `res`
+`terms.path`            | The path to the file where the terms and conditions are configured, or empty if no terms are being used. DEFAULTS TO EMPTY
+
+
+### [db]
+
+Name        | Description
+---------   | -------
+`db.file`   | The path to the SQLite database file for Sydent to use. It can be set to `:memory:` to use a temporary database in RAM instead of on disk. DEFAULTS TO the value stored in the `SYDENT_DB_PATH` environment variable or `sydent.db` if that is unset.
+
+
+### [http]
+
+Name                            | Description
+---------                       | -------
+`client_http_base`              | The base url of Sydent. This should be of the form `scheme://base.url.com/here`. DEFAULTS TO EMPTY
+`clientapi.http.bind_address`   | The local IPv4 or IPv6 address for the Identity Server API to bind to. DEFAULTS TO `::` (i.e. bind to all)
+`clientapi.http.port`           | The port for the Identity Server API to bind to. DEFAULTS TO `8090`
+`federation.verifycerts`        | Whether or not Sydent should verify the TLS certificates of homeservers it communicates with. DEFAULTS TO `true`
+`internalapi.http.bind_address` | The local IPv4 or IPv6 address for the Internal Testing API to bind to. DEFAULTS TO `::1`
+`internalapi.http.port`         | The port for the Internal Testing API to bind to. This must be non-empty to enable the Internal Testing API. Enabling this allows for binding and unbinding between identifiers and matrix IDs without any authentication. DEFAULTS TO EMPTY
+`obey_x_forwarded_for`          | Whether or not Sydent should pay attention to X-Forwarded-For headers. DEFAULTS TO `false`
+`replication.https.bind_address`| The local IPv4 or IPv6 address for the Replication API to bind to. DEFAULTS TO `::` (i.e. bind to all)
+`replication.https.cacert`      | The file path to a root CA certificate. If this is SET then certificates of other Sydent servers signed by this CA will be trusted. This is useful for testing or when it's not practical to get the client cert signed by a real root CA but should never be used on a production server. DEFAULTS TO UNSET
+`replication.https.certfile`    | The file path to a TLS certificate and private key. This file should contain **both** the public certificate and the private key used to generate it. This must be non-empty to enable the Replication API. DEFAULTS TO UNSET
+`replication.https.port`        | The port for the Replication API to bind to. DEFAULTS TO `4434`
+
+
+### [email]
+
+Name                        | Description
+---------                   | -------
+`email.default_web_client_location` | The web client location which will be used in store invites if one is not provided by the homeserver. See [docs/templates.md](https://github.com/matrix-org/sydent/blob/main/docs/templates.md) for more information. This should be of the form 'scheme://base.url.com/here'. DEFAULTS TO `https://app.element.io`
+`email.from`                | The email address that all emails should appear to have been sent from. This should take the form: `Display Name Here <actual.email@example.com>`. DEFAULTS TO `Sydent <noreply@example.com>`
+`email.hostname`            | The fully qualified domain name (FQDN) to use with HELO/EHLO command when connecting to the SMTP server. DEFAULTS TO result of `socket.getfqdn()`
+`email.invite.subject`      | The subject line of emails that invite someone to a room. This is a string template using `"%(variable)s"` substitution and can use any of the parameters sent to the `/store-invite` API. See See [docs/templates.md](https://github.com/matrix-org/sydent/blob/main/docs/templates.md) for more information. DEFAULTS TO `%(sender_display_name)s has invited you to chat`
+`email.invite.subject_space`| The subject line of emails that invite someone to a space. This is a string template using `"%(variable)s"` substitution and can use any of the parameters sent to the `/store-invite` API. See [docs/templates.md](https://github.com/matrix-org/sydent/blob/main/docs/templates.md) for more information. DEFAULTS TO `%(sender_display_name)s has invited you to a space`
+`email.smtphost`            | The address of the SMTP server to use. DEFAULTS TO `localhost`
+`email.smtppassword`        | The password to connect to the SMTP server with. DEFAULTS TO EMPTY
+`email.smtpport`            | The port to connect to the SMTP server on. DEFAULTS TO `25`
+`email.smtpusername`        | The username to connect to the SMTP server with. DEFAULTS TO EMPTY
+`email.third_party_invite_username_obfuscate_characters` | The number of characters to from the beginning to reveal of an email's username portion (left of the '@' sign). See the [README](https://github.com/matrix-org/sydent/blob/main/README.rst) for more information on email obfuscation. DEFAULTS TO `3`
+`email.third_party_invite_domain_obfuscate_characters` | The number of characters to from the beginning to reveal of an email's domain portion (right of the '@' sign). See the [README](https://github.com/matrix-org/sydent/blob/main/README.rst) for more information on email obfuscation. DEFAULTS TO `3`
+`email.tlsmode`             | The security mode to use when connecting with the SMTP server. This can take one of the following options: `NONE`, `TLS`, `SSL`, `STARTTLS`. Any other value is equivalent to `NONE`. DEFAULTS TO `NONE`
+
+
+### [sms]
+
+Name                    | Description
+---------               | -------
+`bodyTemplate`          | The template to use for SMS validation texts. The string '{token}' will get replaced with the validation code. DEFAULTS TO `Your code is {token}`
+`username`              | The username to use when connecting to the SMS sender at https://smsc.openmarket.com/sms/v4/mt. DEFAULTS TO EMPTY
+`password`              | The password to use when connecting to the SMS sender at https://smsc.openmarket.com/sms/v4/mt. DEFAULTS TO EMPTY
+`originators.XYZ`       | The list of originators to use when sending an SMS to a number with country code `XYZ`. The originator is chosen deterministically from this list so if someone requests multiple codes, they come from a consistent number. The originators must be in form: `long:<number>`, `short:<number>` or `alpha:<text>`, separated by commas. DEFAULTS TO UNSET
+`originators.default`   | The default originator to use when sending an SMS. This must be of the form `long:<number>`, `short:<number>` or `alpha:<text>`. DEFAULTS TO UNSET
+`smsrule.XYZ`           | Whether or not to allow verification texts to numbers with country code `XYZ`. If this option is set to `reject` then that country is blacklisted, otherwise it is allowed. DEFAULTS TO UNSET
+
+### [crypto]
+
+Name                | Description
+---------           | -------
+`ed25519.signingkey`| The key used to sign JSON sent by this server. A new key can be generated by running the [generate-key script](https://github.com/matrix-org/sydent/blob/main/scripts/generate-key). This must be set to start the server. NO DEFAULT
+
+### [peer.XYZ]
+
+Name                    | Description
+---------               | -------
+`base_replication_url`  | The base url of the peer with name `XYZ`. This should be of the form `https://internal-address.example.com:4434`. See [docs/replication.md](https://github.com/matrix-org/sydent/blob/main/docs/replication.md) for more information. DEFAULTS TO UNSET

docs/example_config.conf

@@ -0,0 +1,39 @@
+[general]
+server.name = sydent.mydomain.com
+log.path = /home/sydent/sydent.log
+terms.path = /home/sydent/terms.yaml
+templates.path = /home/sydent/templates
+brand.default = example-client
+
+[db]
+db.file = /home/sydent/sydent.db
+
+[http]
+client_http_base = http://sydent.mydomain.com
+
+[email]
+email.from = My Domain <noreply@mydomain.com>
+email.smtphost = relay.emailserver.com
+email.smtpport = 587
+email.smtpusername = myUsername
+email.smtppassword = myPassword
+email.hostname = sydent.mydomain.com
+email.tlsmode = TLS
+email.default_web_client_location = https://app.example-client.com
+
+[sms]
+username = openmarketUserName
+password = openmarketPassword
+
+# Send texts to people with UK (44) numbers from 12345
+originators.44 = short:12345
+originators.default = alpha:mydomain
+
+# Don't sent verification texts to people with US/Canada (1) or French (33) phone numbers
+smsrule.1 = reject
+smsrule.33 = reject
+
+[crypto]
+# Run the generate-key script to produce a random signing key
+# (There is no example key here to prevent people forgetting to generate a new one!)
+ed25519.signingkey = 

scripts/generate-config

@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+
+# Generates a config file for Sydent to use
+
+
+import argparse
+import os
+from configparser import ConfigParser
+
+import signedjson.key
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("--server-name", help="The server domain name. The value in SYDENT_SERVER_NAME takes precedence.")
+
+    parser.add_argument(
+        "--pid-file",
+        default="sydent.pid",
+        help="The file where the PID of the running Sydent process will be written. The value in SYDENT_PID_FILE takes precedence. Default: %(default)s",
+    )
+
+    parser.add_argument(
+        "--db-path",
+        default="sydent.db",
+        help="The SQLite Database file for Sydent to use. The value in SYDENT_DB_PATH takes precedence. Default: %(default)s",
+    )
+
+    parser.add_argument(
+        "-o",
+        "--output-file",
+        default="sydent.conf",
+        help="The file to write the configuration to. The value in SYDENT_CONF takes precedence. Default: %(default)s",
+    )
+
+    args = parser.parse_args()
+
+    signing_key = signedjson.key.generate_signing_key(0)
+    sk_str = "%s %s %s" % (
+        signing_key.alg,
+        signing_key.version,
+        signedjson.key.encode_signing_key_base64(signing_key),
+    )
+
+    cfg = ConfigParser()
+
+    # [general]
+    cfg.add_section("general")
+
+    name = os.environ.get("SYDENT_SERVER_NAME", args.server_name)
+    if name is not None:
+        cfg.set("general", "server.name", name)
+
+    pidfile = os.environ.get("SYDENT_PID_FILE", args.pid_file)
+    cfg.set("general", "pidfile.path", pidfile)
+
+    # [db]
+    cfg.add_section("db")
+
+    dbpath = os.environ.get("SYDENT_DB_PATH", args.db_path)
+    cfg.set("db", "db.file", dbpath)
+
+    # [http]
+    cfg.add_section("http")
+
+    # [email]
+    cfg.add_section("email")
+
+    # [sms]
+    cfg.add_section("sms")
+
+    # [crypto]
+    cfg.add_section("crypto")
+
+    cfg.set("crypto", "ed25519.signingkey", sk_str)
+
+    outputfile = os.environ.get("SYDENT_CONF", args.output_file)
+    if not os.path.exists(outputfile):
+        with open(outputfile, "w") as configfile:
+            cfg.write(configfile)
+    else:
+        print(f"ERROR: Cannot overwrite existing file {outputfile}")