Strac Endpoint Data Protection is plugin-based, cross-platform service application designed to audit and process sensitive files and actions on customer managed workstations. The application is customizable via the addition of managers, processors, and detectors. Its modular architecture allows for the seamless integration of new functionalities, making it adaptable to a wide range of use cases in file analysis and security auditing.
Important
As of 2025-05-22, the Strac Endpoint Agent repository has been made public.
- Auditor
- Architecture
- Usage
- Local Development & Packaging
- Packaging Auditor
- Auditor Components
- Configuration
- Extending Auditor
- Future Work
The architecture of auditor revolves around three main component types:
-
Managers: Orchestrate the overall workflow and lifecycle of the application. They manage resources, coordinate tasks, and facilitate communication between components.
-
Processors: Handle file parsing and data extraction. Each processor is responsible for processing specific file types or formats, converting them into a standardized format for analysis.
-
Detectors: Analyze the processed data to detect specific patterns, conditions, or anomalies. Detectors implement the logic for identifying sensitive information, security threats, or other defined criteria.
There are two main local storage mechanisms for the application, both of which are located in operating system specific directories. Both files can be cleaned by running sudo auditor reset:
-
SQLite: Used for queueing, records management, and other stateful data. This file is managed by the ORM abstractions in
src/storage/database.pyand should not be modified directly. The database configuration settings are all prefixed withDB_. -
Log Files: Used for detailed debugging logs. Currently this means that the logs also show line numbers and file names. This should be modified if this is undesirable to have on a customer machine. The logging configuration settings are all prefixed with
LOG_.
Auditor is designed to be run interactively by a superuser or non-interactively by root, and is installed on the hosts executable path (/usr/local/bin/auditor). Once installed, it's respective services can be interacted with via the auditor command line interface.
The currently available CLI commands and their options:
| Command | Options | Description |
|---|---|---|
start |
scanner, access, network, browser, usb |
Start a specific service |
stop |
scanner, access, network, browser, usb |
Stop a specific service |
restart |
scanner, access, network, browser, usb |
Restart a specific service |
status |
scanner, access, network, browser, usb |
Get status of a specific service |
uninstall |
- | Uninstall local installation |
show |
config, logs, system |
Show configuration or logs |
reset |
- | Reset local installation |
version |
- | Show version |
Notes:
- all commands must be run with
sudoprivileges scannerandnetworkservices currently show an invalid license error message as they arenoopshow configdisplays all safe to show configuration valuesshow logsdisplays the last 120 lines of the local log fileshow systemdisplays the a few values of the current host systemresetwill remove all data and reset the application to a clean stateuninstallwill remove data and uninstall the application completely
After installing the auditor application from a Strac built .pkg, you can use the following commands to start and stop the services:
Tip
You can enable DEBUG logs by setting the AUDITOR_DEBUG environment variable to true.
Start the browser service:
sudo auditor start browserStart the access service:
sudo auditor start accessStop the access service:
sudo auditor stop accessShow the current configuration:
sudo auditor show configShow the help menu:
sudo auditor --help# Install python3.12 from the Python.org website
# download the installer to your desktop
curl -o ~/Desktop/python-3.12.10-macos11.pkg https://www.python.org/ftp/python/3.12.10/python-3.12.10-macos11.pkg
# run the installer and check all options
open ~/Desktop/python-3.12.10-macos11.pkg
# Install Homebrew (if not already installed)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Reload your shell after Homebrew install
exec zsh
# Install binary requirements
brew install poppler upx openblas gcc ruby git most wget nano 1password-cli 1password bat
# Add Ruby to the path
echo 'export GEM_HOME=~/.ruby' >> ~/.bashrc
echo 'export PATH=~/.ruby/bin:$PATH' >> ~/.bashrc
echo 'export PATH="/usr/local/opt/ruby/bin:$PATH"' >> ~/.zshrc
source ~/.bashrc
# Install fpm via ruby gem - DO NOT use brew install fpm as it installs something else
gem install fpm
# Clone the Auditor repository
git clone https://github.com/strac-io/auditor.git
cd auditor
# Create a local virtual environment
python3 -m venv .venv
# Activate the virtual environment
source .venv/bin/activate
# Install the required packages into the virtual environment
python3 -m pip install -r requirements-dev.txt
# Confirm the version of the application
cd src
sudo python3 cli.py versionInstall the 1Password CLI and then follow these instructions to setup the 1Password CLI.
# Install 1Password CLI
brew install 1password-cli# Install binary requirements
sudo apt-get install python3 upx gcc ruby poppler-utils libopenblas-dev qhull-dev libqhull-dev unrar
# Install fpm via ruby gem - DO NOT use apt install fpm as it installs something else
mkdir ~/.ruby
echo 'export GEM_HOME=~/.ruby' >> ~/.bashrc
echo 'export PATH=~/.ruby/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
gem install fpm
sudo cp -Rf ~/.ruby/bin/fpm /usr/local/bin/fpm
# Clone the Auditor repository (if not already done)
# git clone https://github.com/strac-io/auditor.git
# cd auditor
# Create a local virtual environment (if not already done)
# python3 -m venv .venv
# Activate the virtual environment
source .venv/bin/activate
# Update pip
python3 -m pip install --compile --no-cache-dir -U pip
# Install the required packages into the virtual environment
python3 -m pip install --compile --no-cache-dir -r requirements-dev.txt
# Confirm the version of the application
cd src
sudo python3 cli.py version# Install python3.12 from the Python.org website
# download the installer to your desktop
curl -o ~/Desktop/python-3.12.9-macos11.pkg https://www.python.org/ftp/python/3.12.9/python-3.12.9-macos11.pkg
# run the installer and check all options
open ~/Desktop/python-3.12.9-macos11.pkg
## Install Homebrew (if not already installed)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Reload your shell after Homebrew install
exec zsh
# Install binary requirements
brew install poppler upx openblas gcc ruby
# Install fpm via ruby gem - DO NOT use brew install fpm as it installs something else
gem install fpm
# Clone the Auditor repository
git clone https://github.com/strac-io/auditor.git
cd auditor
# Create a local virtual environment
python3 -m venv .venv
# Activate the virtual environment
source .venv/bin/activate
# Install the required packages into the virtual environment
python3 -m pip install -r requirements-dev.txt
# Confirm the version of the application
cd src
sudo python3 cli.py versionInstall the 1Password CLI and then follow these instructions to setup the 1Password CLI.
# Install 1Password CLI
brew install 1password-cliVersioning is handled automatically with the make bump-version command. This command increments the patch version number inside the .version file and replaces the config.py value with the same value. To adjust the build version, edit the .version file directly, not the config.py file. it should only be run once per release, not once per architecture.
Importing the macOS signing certificates is handled with the make import-macos-certificates command. This command will fetch the certificates from the auditor-secrets vault using the 1Password CLI and import them into your local login keychain. You will need to have the 1Password CLI installed and configured to use this command.
Note
You will only need to run this command once per machine.
The building of macOS auditor insatller packages is handled with the make command. The following commands should be used on an arm64 machine to produce arm64 macOS installers and on an x86_64 machine to produce x86_64 macOS installers. Note that you will need to have the 1Password CLI installed and configured to use these commands.
# enter the root directory of the auditor repository
cd auditor
# activate the arm64 virtual environment
source .venv/bin/activate
# increment the version number
make bump-version
# build the installers. Replace "clientA" with the customer name you want to use. CUSTOMER will default to "test" if not specified
make CUSTOMER="clientA" arm-macos-installers
# deactivate the virtual environment
deactivatebuild/- Temporary build artifactsdist/- Final distributable packages and executablespyinstaller/- PyInstaller outputs
assets/- Additional assets used for the installerpkgbuild/- macOS installer assets (e.g.entitlements.plist)customer_scripts/- Customer specific pre and post install scripts (e.g.clientA)
Note
The customer_scripts directory contains customer specific pre and post install scripts that specify what should happen when the installer is run. These scripts are used to customize the installer for the customer and to startup speicifc auditor managers, automatically. These scripts are only installed and executed in the MDM installer.
- PyInstaller - Bundles Python applications into standalone executables. Known for its straightforward approach and broad compatibility across different operating systems.
- macOS: pkgbuild - Creates native
.pkginstallers - Linux: fpm - Simplifies creation of
.deband.rpmpackages (implementation pending)
UPX is utilized to compress the resulting executable. It reduce the overall file size and improves load times.
The signing and notarization process is fully automated and does not need to be run manually. The following docs are provided for reference only.
- apple developer admin account (see your apple developer account admin)
- xcode installed
- strac's "Developer ID Application" certificate from developer.apple.com (see your apple developer account)
- a PyInstaller built binary
# 1. create Certificate Signing Request in Keychain Access
# 2. download Developer ID Application certificate from developer.apple.com
# 3. install certificate to KeychainThis should already be in ./assets/pkgbuild/entitlements.plist, but if not, here's the template:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<!-- required for binaries built by PyInstaller -->
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
</dict>
</plist>codesign --force --timestamp --sign "F80534528D879A29E0222B68355E0848BD709CCA" --entitlements ./assets/pkgbuild/entitlements.plist --options runtime ./dist/pyinstaller/auditor-X.X.X-ARCH/auditorThis includes a macOS compatible bash script that installs launchd daemons for both the access and uploads managers. This script is run immediately after success installation and also kickstarts both managers.
pkgbuild --root ./dist/pyinstaller/auditor-X.X.X-ARCH/auditor \
--scripts assets/pkgbuild/Scripts \
--identifier "com.strac.auditor" \
--version "1.0" \
--install-location "/Applications" \
--sign "0EDE190D1910FDCD363710B46E85A439DBD6E06E" \
auditor-X.X.X-ARCH.pkgpkgbuild --root ./dist/pyinstaller/auditor-X.X.X-ARCH/auditor \
--identifier "com.strac.auditor" \
--version "1.0" \
--install-location "/Applications" \
--sign "0EDE190D1910FDCD363710B46E85A439DBD6E06E" \
auditor-X.X.X-ARCH.pkgproductbuild --sign "Developer ID Installer: Strac Incorporated (992GD587TM)" --package auditor-X.X.X-ARCH.pkg auditor-X.X.X-ARCH-READY.pkg# cleanup pkgs
rm -rf auditor-X.X.X-ARCH.pkg
mv auditor-X.X.X-ARCH-READY.pkg auditor-X.X.X-ARCH.pkg
# submit for notarization and wait for completion
xcrun notarytool submit auditor-X.X.X-ARCH.pkg --keychain-profile "notary-strac.io" --wait
# staple the notarization
xcrun stapler staple auditor-X.X.X-ARCH.pkg
# verify the notarization
xcrun stapler validate auditor-X.X.X-ARCH.pkg- Export the existing certificates from your local keychain:
# export the Developer ID Application certificate
security export -k login.keychain -t identities -f pkcs12 -P "YOUR_PASSWORD" -o developer_id_app.p12 "Developer ID Application: Strac Incorporated (992GD587TM)"
# export the Developer ID Installer certificate
security export -k login.keychain -t identities -f pkcs12 -P "YOUR_PASSWORD" -o developer_id_installer.p12 "Developer ID Installer: Strac Incorporated (992GD587TM)"-
Zip up the certificates and transfer the newly exported
.p12files to their new destination machine using a secure method (like AirDrop, secure file transfer, or private network share). -
Import the certificates into the keychain on the new machine:
# Import Developer ID Application certificate
security import developer_id_app.p12 -k login.keychain -P "YOUR_PASSWORD" -T /usr/bin/codesign
# Import Developer ID Installer certificate
security import developer_id_installer.p12 -k login.keychain -P "YOUR_PASSWORD" -T /usr/bin/productbuild- Setup the notarization profile on the new machine:
Note
You will need to retrieve the apple-id, team-id, and password from the auditor-secrets 1Password Vault.
xcrun notarytool store-credentials "notary-strac.io" --apple-id "SOME_ID" --team-id "SOME_ID" --password "SOME_PASSWORD"- Verify the certificates and profile are setup correctly:
# List certificates to confirm they're properly imported
security find-identity -v -p codesigningManagers are responsible for orchestrating the processing workflow. They coordinate the operations of processors and detectors, manage systems application execution, tasks and resources, and ensure efficient execution of their respective pipelines.
Available Managers:
| Manager | Description |
|---|---|
| Access_Manager | Manages system call access monitoring, including file monitoring and application execution |
| Browser_Manager | Manages browser download activity monitoring and detection |
| Network_Manager | Manages website blocking rules and the packet filtering service |
| Scanner_Manager | Manages the file scanning processes, including sensitive data detection, queuing, and processing |
| USB__Manager | Manages the USB drive mount scanner |
Processors parse and extract data from various file types. Each processor is designed to handle a specific file format, converting the raw data into a format suitable for analysis by detectors.
Available Processors:
| Processor | Description |
|---|---|
| Archive_Processor | Handles compressed archives (.zip, .rar, .7z) and processes their contents |
| Email_Processor | Parses email files and extracts message contents (.eml, .msg) |
| Excel_Processor | Processes Microsoft Excel files (.xls, .xlsx) |
| GDS_Processor | Processes schematic files (.gds, .gdsii, .oas) |
| Image_Processor | Handles image files, extracting text via OCR when necessary (.png, .jpg, .jpeg, .tiff, .gif, .bmp, .webp) |
| iWork_Processor | Extracts text from iWork documents (.pages, .numbers, .keynote) |
| PDF_Processor | Extracts text and metadata from PDF files (.pdf) |
| PowerPoint_Processor | Extracts content from PowerPoint presentations (.ppt, .pptx) |
| Text_Processor | Processes plain text files, including CSV, JSON, YAML, and more (.txt, .csv, .json, .yaml, .yml) |
| Word_Processor | Extracts text from Microsoft Word documents (.doc, .docx) |
Detectors analyze the processed content to identify specific patterns, sensitive information, or anomalies. They apply detection logic such as pattern matching, keyword search, and content classification.
Available Detectors:
| Detector | Description |
|---|---|
| Confidential_Detector | Identifies confidential or sensitive information |
| DOB_Detector | Identifies Date of Birth |
| Email_Detector | Identifies email addresses and patterns |
| Financial_Detector | Identifies financial data and patterns |
| IBAN_Detector | Identifies International Bank Account Numbers (IBANs) |
| IN_AADHAR_Detector | Identifies Indian AADHAAR numbers |
| IN_NREGA_Detector | Identifies Indian NREGA numbers |
| IP_Detector | Identifies IP addresses and patterns |
| PCI_Detector | Detects Payment Card Industry-related sensitive data |
| Phone_Number_Detector | Identifies phone numbers and patterns |
| UK_NHS_Detector | Identifies United Kingdom National Health Service (NHS) numbers |
| UK_NINO_Detector | Identifies United Kingdom National Insurance Number (NINO) numbers |
| UK_UTR_Detector | Identifies United Kingdom Unique Taxpayer Reference (UTR) numbers |
| US_Driver_License_Detector | Identifies United States Driver License numbers |
| US_License_Plate_Detector | Identifies United States License Plate numbers |
| US_Passport_Detector | Identifies United States Passport numbers |
| US_SSN_Detector | Detects United States Social Security Number (SSN) patterns |
| US_Taxpayer_ID_Detector | Identifies United States Taxpayer Identification Number (TIN) numbers |
| VIN_Detector | Identifies Vehicle Identification Numbers (VINs) |
Auditor's behavior can be customized through the config.py file and other configuration settings. This allows users to enable or disable components, set specific processing parameters, and adjust the application's operation to suit different environments and requirements.
TODO: Add configuration sync with the Strac API.
To configure which plugins are loaded for the Scanner Manager:
- Enable or disable plugin requirements installation: This is controlled by the
SCANNER_SKIP_PLUGIN_REQUIREMENTSflag in theconfig.pyfile. Setting it toFalseensures that the required dependencies for each plugin are installed.
SCANNER_SKIP_PLUGIN_REQUIREMENTS = False # Set to True to skip installing plugin requirements- Enable or disable specific processors or detectors: Modify the
SCANNER_ENABLED_PROCESSORSandSCANNER_ENABLED_DETECTORSlists in theconfig.pyfile to include only the plugins you want to load.
# Enabled processor modules for different file types
SCANNER_ENABLED_PROCESSORS = [
"pdf_processor",
"archive_processor",
"text_processor",
"word_processor",
"excel_processor",
"powerpoint_processor",
"image_processor",
"email_processor",
]
# Enabled detector modules for content analysis
SCANNER_ENABLED_DETECTORS = [
"pci_detector",
"confidential_detector",
"pattern_detector",
"keyword_detector",
]Each processor defines the file extensions it supports. To configure file extensions for processors:
- Modify the
supported_extensionsattribute in the processor's class.
For example, to add support for .md files in the Text_Processor:
class Processor:
# ...
supported_extensions = [
".csv",
".env",
".htm",
".html",
".json",
".rtf",
".txt",
".xml",
".yaml",
".yml",
".md",
]To configure directories, filenames, or file extensions to ignore during scanning, modify the following settings in the config.py file:
# Directories to ignore during scanning
SCANNER_IGNORE_DIRECTORIES = [
"_",
".",
"Shared",
"node_modules",
"Pictures",
"Applications",
"Library",
]
# Filenames to ignore
SCANNER_IGNORE_FILENAMES = ["thumbs.db", ".ds_store", "package.json"]
# File extensions to ignore
SCANNER_IGNORE_EXTENSIONS = [".tmp", ".log", ".py"]To configure the Strac API settings, including the Customers API key and related parameters, update the relevant fields in the config.py file:
STRAC_API_KEY = "YOUR_API_KEY_HERE" # Replace with your actual API key
STRAC_API_HEADERS = {
"x-api-key": STRAC_API_KEY,
"clientId": "your_client_id",
}
STRAC_API_BASE_URL = "https://api.test.yourapi.com"
STRAC_API_ENDPOINT_CREATE_DOCUMENT = f"{STRAC_API_BASE_URL}/documents"
STRAC_API_ENDPOINT_CREATE_DOCUMENT_LARGE = f"{STRAC_API_BASE_URL}/documents/url"
STRAC_API_ENDPOINT_DETECT = f"{STRAC_API_BASE_URL}/detect"
STRAC_API_ENDPOINT_PROCESS_MESSAGE = f"{STRAC_API_BASE_URL}/endpoint-dlp/process-message"
STRAC_API_ENDPOINT_CONFIG = f"{STRAC_API_BASE_URL}/endpoint-dlp/config"
STRAC_API_DOCUMENT_SIZE_LIMIT = 6.3 * 1024 * 1024 # 6.3 MBNote: Ensure that you securely handle your STRAC_API_KEY. Do not hardcode sensitive information in your configuration files if they are part of version control. Consider using environment variables or a secure secrets management system.
Adding functionality to auditor can be done by creating new managers, processors, or detectors. The following sections provide guidance and examples on how to develop and integrate new components.
To create a new manager, there is no current base manager class to inherit from nor is there a contract that is needed to be met. Implement the methods required for your custom needs:
import logging
class CustomManager:
def __init__(self):
self.logger = logging.getLogger("manager-custom")
def start(self):
# Startup code if needed
pass
def process(self, data):
# Processing logic if needed
passTo develop a new processor:
- Create a new directory under
src/processorswith the name of your processor (e.g.,custom_processor). - Within this directory, create an
__init__.pyfile and your processor implementation file (e.g.,custom_processor.py). - Inherit from the base
Processorclass and implement the necessary methods.
Example:
import logging
class Processor:
name = "Custom Processor"
version = "1.0"
supported_extensions = [".custom"]
def __init__(self):
self.logger = logging.getLogger("processor-custom")
async def process_file_async(self, file_path, scanner):
# Asynchronous processing logic
pass
def process_file(self, file_path, scanner):
# Synchronous processing logic
passTo create a new detector:
- Create a new directory under
src/detectorswith the name of your detector (e.g.,custom_detector). - Within this directory, create an empty
__init__.pyfile, your detector implementation file (e.g.,custom_detector.py), and arequirements.txtfile. - Inherit from the base
Detectorclass and implement the required methods.
Example:
import logging
class Detector:
description = "Custom Detector" # include a brief description of this detector
version = "1.0" # include the version of this detector
def __init__(self):
self.name = "detector-custom" # include the name of this detector in this format
self.logger = logging.getLogger(self.name)
async def process_text(self, text_content):
# Detection logic
findings = []
# Analyze text_content and populate findings
return findingssome-package==1.2.3
some-other-package==3.2.1After creating new components:
- Register the component by adding it to the appropriate enabled list in
config.py.
SCANNER_ENABLED_PROCESSORS = [
# Existing processors
"custom_processor", # Add your new processor
]
SCANNER_ENABLED_DETECTORS = [
# Existing detectors
"custom_detector", # Add your new detector
]- Install any required dependencies by adding them to a
requirements.txtfile inside your component's directory.
For example, if your processor requires somepackage:
somepackage==1.2.3
- Ensure the application installs plugin requirements by setting
SCANNER_SKIP_PLUGIN_REQUIREMENTStoFalseinconfig.py.
After creating your new Manager:
- Integrate with the CLI by adding your new manager to
config.pyand creating the appropriate cli commands.
The following sections outline the steps to run auditor as a background service on Linux and macOS. They should be implemented if auditor is to be run as a background service (i.e. not interactively/headless).
- Copy the service file to the
systemddirectory:
sudo cp dist/auditor.service /etc/systemd/system/- Reload
systemdand start the service:
sudo systemctl daemon-reload
sudo systemctl enable auditor
sudo systemctl start auditorExample auditor.service file:
[Unit]
Description=Auditor Service
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/auditor
Restart=always
RestartSec=42s
[Install]
WantedBy=multi-user.targetThis has ben implemented in cli.py and as a postinstall script for kickstarting the acess manager and the downloads manager only. This implementation is specific to the clientB customer and needs to be revisited for a more generic implementation.
This is likely where we want to be creating and storing our releases as a first step, prior to sharing with customers.
-
Tagging a Release:
To tag a new release, use the following command in your terminal:
git tag -a vX.Y.Z -m "Release version X.Y.Z"Replace
X.Y.Zwith the version number you are releasing. The-mflag allows you to add a message to the tag. -
Pushing the Tag to GitHub:
After creating the tag, push it to the GitHub repository:
git push origin vX.Y.Z
-
Creating a Release on GitHub:
- Go to your repository on GitHub.
- Click on the "Releases" tab.
- Click the "Draft a new release" button.
- In the "Tag version" dropdown, select the tag you just pushed (e.g.,
vX.Y.Z). - Fill in the release title and description.
- Use the "Attach binaries by dropping them here or selecting them" section to upload the
auditorinstallers. - Click the "Publish release" button.