Local POSIX Access Design Strategies

[chirayupatel9]

DataFed Local POSIX Access Design Strategies Overview

This document provides a comprehensive and detailed overview of five different design strategies for enabling local POSIX access to DataFed repositories. Each strategy represents a distinct architectural approach to balancing security, performance, usability, and operational complexity in high-performance computing (HPC) environments.

The strategies address the fundamental challenge of providing secure, efficient access to repository data while maintaining the performance characteristics that HPC workflows demand. They range from simple file linking approaches to complex namespace isolation systems, each offering different trade-offs that must be carefully considered based on specific use case requirements.

---

Design Strategy 1: Temporary POSIX User Mapping with ACLs

Overview: Create temporary POSIX users on the repository systems that map to application users, using Access Control Lists (ACLs) to grant read/write permissions without changing file ownership.

Implementation:

• When a user requests local access, the hub instructs the repository to:
• Create a temporary POSIX user account (or reuse from a pool)
• Set ACLs on the requested files: setfacl -m u:tempuser:rw file
• Provide SSH/SFTP credentials to the user
• Monitor access and revoke after session ends or timeout

Approach: Creates temporary POSIX user accounts with file-specific ACLs for direct repository access. This strategy leverages the native Linux user management and access control systems to provide secure, isolated access to repository files.

Key Components

Temporary POSIX Accounts

• Dynamic user creation: Temporary accounts (df_tmp_007, df_tmp_008, etc.) are created on-demand when users request access
• Isolated user environment: Each temporary user has a minimal home directory and restricted shell access
• Automatic cleanup: Users are automatically removed after TTL expiration or explicit revocation

POSIX ACLs (Access Control Lists)

• Fine-grained permissions: Uses setfacl to grant read, write, or execute permissions on specific files
• User-specific access: Each temporary user gets precisely the permissions they need for their workflow
• Permission inheritance: Can set default ACLs on directories for new files created during the session

SSH Key-based Authentication

• Ephemeral SSH keys: Short-lived SSH key pairs generated for each access session
• Secure key distribution: Keys are securely delivered to users through the DataFed Hub
• Key rotation: New keys are generated for each session to prevent key reuse attacks

Automatic Cleanup Mechanisms

• Systemd timers: Automated cleanup processes that run at regular intervals
• ACL revocation: Automatic removal of user-specific ACLs when sessions expire
• User account cleanup: Complete removal of temporary user accounts and associated data

Implementation Flow

Repository Side Setup

# 1. Create temporary user account
sudo useradd -m -s /bin/bash -c "Temporary DataFed user" df_tmp_007

# 2. Set file-specific ACLs for the user
setfacl -m u:df_tmp_007:rw /repo/project42/inputs/config.json
setfacl -m u:df_tmp_007:r /repo/project42/reference/data.csv

# 3. Generate SSH key pair for the user
sudo -u df_tmp_007 ssh-keygen -t ed25519 -N "" -f /tmp/g-007_key

# 4. Set up TTL timer for automatic cleanup
systemctl start datafed-cleanup@df_tmp_007.timer

User Side Access

# 1. Download SSH key from DataFed Hub
# 2. Set proper permissions on the private key
chmod 600 ~/g-007_key

# 3. Connect to repository as temporary user
ssh -i ~/g-007_key df_tmp_007@repo.example.com

# 4. Access authorized files directly
cat /repo/project42/inputs/config.json
cp /repo/project42/reference/data.csv ~/local_analysis/

Pros

• ✅ Native POSIX performance: Direct filesystem access without virtualization overhead
• ✅ Precise per-file access control: ACLs provide granular permission management
• ✅ Clean audit trail: All access is logged through standard Linux mechanisms
• ✅ No file duplication: Files remain in their original location
• ✅ Standard Linux tools: Uses well-established user management and security practices
• ✅ HPC compatibility: Works seamlessly with Slurm, MPI, and other HPC tools

Cons

• ❌ Temporary user account creation: Requires system-level user management operations
• ❌ SSH key management overhead: Keys must be generated, distributed, and cleaned up
• ❌ Potential privilege escalation: If not properly isolated, temporary users could access system resources
• ❌ File permission complexity: ACLs can become complex with many users and files
• ❌ Session persistence: Users lose access if their SSH session disconnects

Use Cases

• HPC workflows requiring direct file access: Jobs that need to read/write large datasets during computation
• Short-term data analysis sessions: Interactive analysis sessions lasting minutes to hours
• Users needing read/write access to specific files: Researchers who need to modify input files or write results
• Batch processing workflows: Automated jobs that process multiple files in sequence

Security Considerations

• Temporary users should have minimal system access (no sudo, limited PATH, restricted shell)
• SSH keys must have appropriate expiration and be securely distributed
• ACLs should be regularly audited to ensure they don't accumulate over time
• User isolation should prevent access to system files and other users' data

User Sequence Diagram

sequenceDiagram
  participant Bob as Bob (User)
  participant DataFed as DataFed Client
  participant Hub as DataFed Hub
  participant Repo as Repository Service
  participant KDC as Kerberos Service
  participant POSIX as POSIX Filesystem

  Bob ->> DataFed: Login to DataFed (datafed login)
  DataFed ->> Hub: Verify User
  Hub -->> DataFed: ACK
  Bob ->> DataFed: User requests Repo addr (datafed connect $repo_id)
  DataFed ->> Hub: datafed connect $repo_id
  Hub -->> DataFed: Return Repo Addr
  DataFed ->> Repo: Tries to connect
  alt unable to connect
    Repo -> Bob: Timeout.
  end
  Repo -->> DataFed: Acknowledge connected, return Auth Addr
  DataFed ->> DataFed: Check Ticket
  alt no Ticket
    Bob ->> DataFed: enter username/password
    DataFed ->> KDC: New Ticket request
    KDC -->> DataFed: User Ticket
    DataFed ->> DataFed: Cache Ticket
  end
  DataFed ->> Repo: TGT + datafed token
  Repo ->> Hub: Verifies Datafed User and User logged in with kerberos is same
  Hub ->> Hub: Add Lock flag with username
    Hub ->> Repo: Ack, Lock Flag, RW Perm
  Repo ->> POSIX: setfacl -m u:bob:r /mnt/data/file.txt
  alt Session expires or logout
  Note over Repo, POSIX: Monitor session or token expiry
    Repo -->> DataFed: returns: /mnt/data/file.txt
    DataFed -->> Bob: cat: /mnt/data/file.txt
    Repo ->> POSIX: setfacl -x u:bob /mnt/data/file.txt
  end

Loading

---

Steps:

1. Bob logs into DataFed using the CLI command datafed login. The DataFed client contacts the Hub to verify the user.

2. Once authenticated, Bob requests access to a file by calling datafed connect $repo_id.

3. The Hub resolves the address of the repository holding the file and returns it to the DataFed client.

4. The DataFed client attempts to connect to the Repo service. If unsuccessful, a timeout is returned.

5. Upon successful connection, the Repo checks whether Bob has a valid Kerberos ticket.

6. If no valid ticket is found:

• The client prompts Bob to enter credentials.

• A Kerberos Ticket Granting Ticket (TGT) is requested from the Kerberos Service (KDC).

• The TGT is cached locally.

7. The Repo service grants Bob temporary access by setting POSIX ACL:

setfacl -m u:bob:r /mnt/data/file.txt

8. A session monitor is launched on the Repo side, watching for the Kerberos ticket’s expiration.

9. When the Kerberos session ends (via kdestroy or timeout), the Repo revokes Bob’s access:

setfacl -x u:bob /mnt/data/file.txt

---

Here is a simple implementation

Repository / Admin Side

(executed by the “repo‑controller” service running as root)

Allocate POSIX permission to an user for a file.txt.

setfacl -m u:bob:rw file.txt

u: User permissions (named user, other than owner)
:rw Read / Write on the file

Set Permissions for a Specific Group

setfacl -m g:developers:r file.txt

g: Group permissions (named group, other than owning group)
:r Grants group read permission

Set Default ACL on a Directory

setfacl -d -m u:alice:rwx /data/shared

New files in /data/shared will grant alice rwx.

Set Multiple ACLs at Once

setfacl -m u:alice:rw,g:devs:rw file.txt

Sets permissions for user and group simultaneously.

Remove All ACLs (reset to basic permissions)

setfacl -b file.txt

Clears all extended ACLs (restores chmod-based permissions).

View Current ACLs

getfacl file.txt

Sample output:

Copy
Edit
# file: file.txt
# owner: root
# group: root
user::rw-
user:bob:rw-
group::r--
mask::rw-
other::r--

Grant ACLs only on authorised paths

FILES=(/repo/project42/inputs/config.json)
TTL=8h

for f in "${FILES[@]}"; do
    setfacl -m u:$TMP_USER:rx "$f"
done
# (Use --default ACLs if directories need traversal; remove w bit for RO.)

Generate short‑lived SSH key + drop into Hub

sudo -u "$TMP_USER" ssh-keygen -t ed25519 -N "" -f /tmp/g-007_key
# send /tmp/g-007_key.pub to Hub → delivered to Bob

Audit logging

logger -p local0.info "GRANT g-007 user=bob tmp=$TMP_USER paths=${FILES[*]} ttl=$TTL"
(Add /var/log/datafed_acl/... rotating logs if desired.)

User / Client Side (Bob)

Download / save the temp SSH private key

Hub UI →  “Download SSH key pair”
Files: g‑007_key  and  g‑007_key.pub

SSH to the repository host as df_tmp_007

chmod 600 ~/g-007_key
ssh -i ~/g-007_key df_tmp_007@repo.example

Prompt now shows:

df_tmp_007@repo-host:~$

Read or copy the authorised file (RO example)

cp /repo/project42/inputs/config.json  $HOME/
# editing will fail because file is read‑only

Optional: keep session running during HPC job

Bob can launch the HPC job from this shell or simply leave the shell open while the job reads the file under the temp UID.

Early exit

exit       ends SSH session

ACL remains valid until TTL timer fires (unless Bob presses “Revoke now” in Hub).

What happens if Milly tries to delete while Bob is reading?

• Bob (temp UID) has the file open.

• Milly requests DELETE via Hub; repo controller checks lsof +D file or fuser and sees an open FD under df_tmp_007.

• Hub responds “File is currently in use by Bob – try later.”

• After Bob’s FD closes (shell exit or job close()), Milly’s next DELETE will succeed.

• (Kernel enforces the lock: inode can’t be deleted while a link count > 0 and FD open.)

How every commitment is met

Requirement	Fulfilled by
User‑specific mount namespace?	Not needed here – temp UID + ACL already limits view.
Bind only authorised files?	setfacl -m u:df_tmp_xxx gives precise per‑file perms.
UID remap	Temp account’s UID matches POSIX entry; files show as owned by repo_svc, but Bob accesses via ACL.
Jailed environment	User logs in as isolated temp account; no sudo, no home dir.
Automatic revocation	systemd timer + revoke_acl.sh removes ACLs & account.

---

Design Strategy 2: FUSE-based Virtual Filesystem

Overview: Implement a FUSE (Filesystem in Userspace) layer that presents authorized files to users while maintaining the underlying service account ownership.

Implementation:

• Deploy a FUSE daemon on each repository
• User authenticates to the FUSE mount point
• FUSE daemon queries hub for authorization in real-time or uses cached tokens
• Files appear with virtual permissions based on user's authorization

Approach: Uses FUSE (Filesystem in Userspace) to create virtual filesystem mounts with JWT token authentication. This strategy provides a transparent filesystem interface while maintaining centralized access control through token validation.

Key Components

FUSE Daemon (df-fuse)

• Virtual filesystem layer: Creates a mount point that appears as a normal directory to users
• Token validation: Checks JWT tokens for each filesystem operation
• Permission mapping: Translates token permissions to filesystem access rights
• File operation proxying: Intercepts all filesystem calls and validates them before execution

JWT Token Authentication

• Time-limited access: Tokens have configurable TTL (Time To Live) values
• Scope-based permissions: Tokens specify which files and directories are accessible
• Permission granularity: Read-only, read-write, or execute permissions can be specified
• User identification: Tokens contain user identity information for audit purposes

Automatic Token Refresh

• Background refresh: Tokens are automatically renewed before expiration
• Long-running job support: Jobs can continue running without manual token renewal
• Refresh policies: Configurable refresh intervals and failure handling
• Seamless user experience: Users don't need to manually handle token expiration

File Locking Mechanisms

• Concurrent access control: Prevents multiple users from simultaneously modifying the same file
• Lock coordination: Coordinates locks across multiple mount points and users
• Deadlock prevention: Implements timeout and retry mechanisms for lock acquisition
• Lock persistence: Locks survive across token refreshes and network interruptions

Implementation Flow

Token Request and Mount

# 1. User requests access token from DataFed Hub
datafed token request --scope /project42/inputs --permissions rw --ttl 12h

# 2. Hub returns JWT token
# Token contains: user_id, scope, permissions, expiration, signature

# 3. User mounts repository using token
datafed mount --token ~/token.jwt --mount-point ~/datafed_repo

# 4. FUSE daemon validates token and creates virtual filesystem
# Files appear at ~/datafed_repo/ with appropriate permissions

File Operations

# 1. User performs normal filesystem operations
ls ~/datafed_repo/
cat ~/datafed_repo/config.json
cp ~/datafed_repo/data.csv ~/local_analysis/

# 2. FUSE daemon intercepts each operation
# 3. Token is validated for each file access
# 4. Operations are proxied to underlying repository
# 5. Results are returned to user

Token Refresh Process

# 1. FUSE daemon monitors token expiration
# 2. At 75% of TTL, automatic refresh is initiated
# 3. New token is requested from Hub
# 4. Old token is replaced with new one
# 5. User operations continue uninterrupted

Pros

• ✅ Transparent filesystem access: Users interact with files as if they were local
• ✅ Automatic token refresh: Long-running jobs don't require manual intervention
• ✅ File locking prevents conflicts: Multiple users can safely access the same repository
• ✅ No temporary accounts needed: All access is controlled through tokens
• ✅ Centralized access control: Hub maintains complete control over permissions
• ✅ Cross-platform compatibility: Works on Linux, macOS, and Windows

Cons

• ❌ FUSE overhead: 25-50% performance impact compared to native filesystem access
• ❌ Mount point management complexity: Users must manage mount points and ensure proper unmounting
• ❌ Potential compatibility issues: Some HPC tools may not work properly with FUSE mounts
• ❌ Memory usage: FUSE daemon consumes additional memory for caching and operation tracking
• ❌ Network dependency: Token validation requires network connectivity to Hub

Use Cases

• Long-running HPC simulations: Jobs that need persistent access to repository files
• Interactive data exploration: Users who need to browse and analyze multiple files
• Multi-user collaborative environments: Teams working on shared datasets
• Cross-platform workflows: Users working on different operating systems
• File-based applications: Software that expects standard filesystem semantics

Performance Characteristics

• Read operations: Generally good performance with caching, but slower than native for large files
• Write operations: Performance depends on underlying storage and network conditions
• Metadata operations: Directory listings and file information queries are fast
• Concurrent access: Multiple users can access the same files simultaneously
• Cache efficiency: Frequently accessed files benefit from FUSE-level caching

User sequence Diagram

sequenceDiagram
  participant Bob as HPC User (Bob)
  participant DataFed as DataFed Client
  participant Hub as DataFed Hub
  participant Repo as Repo + Mount Manager
  participant FUSE as FUSE or bindfs Mountpoint
  participant FS as POSIX Filesystem
  participant KDC as Kerberos Server
  participant DB as Metadata Database
  Bob ->> DataFed: Login to DataFed (datafed login)
  DataFed ->> Hub: Verify User
  Hub -->> DataFed: ACK
  Bob ->> DataFed: User requests Repo addr (datafed connect $repo_id)
  DataFed ->> Hub: datafed connect $repo_id
  Hub -->> DataFed: Return Repo Addr
  DataFed ->> Repo: Tries to connect
  alt unable to connect
    Repo -> Bob: Timeout.
  end
  Repo -->> DataFed: Acknowledge connected, return Auth Addr
  DataFed ->> DataFed: Check Ticket
  alt no Ticket
    Bob ->> DataFed: enter username/password
    DataFed ->> KDC: New Ticket request
    KDC -->> DataFed: User Ticket
    DataFed ->> DataFed: Cache Ticket
  end
  DataFed ->> Repo: TGT + datafed token
  Repo ->> Hub: Verifies Datafed User and User logged in with kerberos is same
  Hub ->> Hub: Add Lock flag with username
    Hub ->> Repo: Ack, Lock Flag, RW Perm

  %% ---------- Authorization + Access Setup ----------
  DataFed ->> DataFed: datafed access d/456 --lock
  DataFed ->> Hub: Check permissions for Bob on d/456
  Hub -->> DataFed: Access granted

  DataFed ->> Repo: Request hard link for Bob to d/456 with lock
  Repo ->> FS: ln /mnt/shared/master/file_456.txt /mnt/shared/bob/file_bob.txt
  FS -->> Repo: Hard link created
  Repo ->> DB: Record lock (locked=true, locked_by=Bob)
  DB -->> Repo: Lock recorded
  Repo -->> DataFed: ACK - file_bob.txt ready

  %% ---------- File Access Phase with Lock ----------
  DataFed ->> DataFed: file_bob.txt ready in /mnt/shared/bob

  %% ---------- Bob Opens File (Implicit Locking) ----------
  DataFed ->> FUSE: open("/mnt/shared/bob/file_bob.txt", O_RDWR)
  FUSE ->> Repo: Check write permission
  Repo ->> DB: Check if file is already locked
  alt Not locked
    Repo ->> DB: Set locked=true, locked_by=Bob
    DB -->> Repo: Lock set
    Repo -->> FUSE: Allow open
    FUSE -->> Bob: File opened for writing
  else Locked by another user
    Repo -->> FUSE: Deny write (EACCES)
    FUSE -->> Bob: Permission denied
  end

  %% ---------- Session Completion + Cleanup ----------
  Bob ->> DataFed: file access complete
  DataFed ->> Repo: Dispose hard link /mnt/shared/bob/file_bob.txt
  Repo ->> DB: Clear lock
  Repo ->> FS: unlink /mnt/shared/bob/file_bob.txt
  FS -->> Repo: Link removed
  Repo -->> DataFed: Cleanup done

Loading

Steps:

1. User Login: Bob (the HPC User) initiates a login to the DataFed Client using the command datafed login.

2. Verification: The DataFed Client contacts the DataFed Hub to verify Bob's identity. The Hub acknowledges the verification.

3. Repo Connection: Bob requests to connect to a specific repository (datafed connect $repo_id).

4. Address Retrieval: The DataFed Client asks the Hub for the repository's network address, and the Hub provides it.

5. Direct Connection & Auth: The DataFed Client connects to the Repo. The Repo acknowledges the connection and provides an authentication address.

6. Kerberos Ticket Check: The DataFed Client checks for a cached Kerberos authentication ticket.

7. If no ticket exists: Bob is prompted for his username/password, which the client uses to request a new ticket from the Kerberos Server (KDC). The client then caches this new ticket.

8. Authentication: The DataFed Client sends the Kerberos ticket (TGT) and a DataFed token to the Repo.

9. Final Verification: The Repo contacts the Hub to ensure the DataFed user and the Kerberos user are the same. The Hub confirms this, adds a lock flag for the user's session, and grants Read/Write permissions to the Repo.

10. Access Request: Through the client, Bob requests locked access to a data object (e.g., datafed access d/456 --lock).

11. Permission Check: The DataFed Client asks the Hub if Bob has permission to access d/456. The Hub grants access.

12. Hard Link Creation: The client requests that the Repo create a locked hard link to the data for Bob.

13. Filesystem Operation: The Repo instructs the POSIX Filesystem (FS) to create a hard link from the master file to a file in Bob's personal directory (e.g., ln /mnt/shared/master/file_456.txt /mnt/shared/bob/file_bob.txt).

14. Database Update: After the FS confirms the link is created, the Repo records the lock in the Metadata Database (DB), marking it as locked by Bob.

15. Confirmation: The Repo notifies the DataFed Client that the personal, locked file (file_bob.txt) is ready for use.

16. File Open Request: Bob attempts to open his personal file (file_bob.txt) for writing. This request is intercepted by the FUSE mountpoint.

17. Permission Check: FUSE asks the Repo to verify write permissions.

18. Lock Verification: The Repo checks the DB to see if the file is locked.

19. If not locked: The Repo immediately sets the lock for Bob in the DB and tells FUSE to allow the file to be opened. Bob can now write to the file.

20. If locked by another user: The Repo tells FUSE to deny the request, and Bob receives a "Permission denied" error.

21. Access Complete: Bob signals that he is finished with the file.

22. Cleanup Request: The DataFed Client tells the Repo to dispose of the hard link.

23. Lock Release: The Repo updates the DB to clear the lock on the file.

24. File Unlink: The Repo instructs the FS to remove the hard link from Bob's directory.

25. Final Confirmation: After the FS confirms the removal, the Repo notifies the DataFed Client that the cleanup is complete.

---

Here is a simple scenario

Use case scenario

Don is a scientist, fuse mounting few directories in to his local computer with the help of sshfs.

Susie is a grad student who wants to have access to those directories as she needs to read those files for some research purposes.

Later Don gives Susie access to those files by creating a fuseaccess group so other users in the system doesn't get access those files.

---

Create fuseaccess group

sudo addgroup fuseaccess

Add Don and Susie to the fuseaccess group

sudo usermod -aG fuseaccess don
sudo usermod -aG fuseaccess susie

Verify group membership

getent group fuseaccess
groups don
groups susie

Create a secure mount point

sudo mkdir /mnt/shared_securedata
sudo chown don:fuseaccess /mnt/shared_securedata
sudo chmod 770 /mnt/shared_securedata

Additional security hardening

# Set sticky bit to prevent users from deleting files they don't own
sudo chmod 1770 /mnt/shared_securedata

# Verify permissions
ls -la /mnt/shared_securedata

To get id's

id -u
getent group fuseaccess

To Mount

Basic mount with security options

sshfs don@localhost:/somepath /mnt/shared_securedata \
  -o allow_other,default_permissions,uid=1000,gid=1002,umask=007

Enhanced mount with additional security and performance options

sshfs don@localhost:/somepath /mnt/shared_securedata \
  -o allow_other,default_permissions,uid=1000,gid=1002,umask=007 \
  -o compression=no,large_read,large_write \
  -o ServerAliveInterval=60,ServerAliveCountMax=3 \
  -o IdentityFile=/home/don/.ssh/id_rsa \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=/home/don/.ssh/known_hosts

somepath can be anything in don's local directory /home/don/code

uid = user id

gid = group id

umask=007: sets permission to 770 on files/dirs (owner/group full, others none)

Mount option explanations:

• allow_other: Allows users other than the mounting user to access the filesystem
• default_permissions: Enables permission checking on the client side
• compression=no: Disables compression for better performance on fast networks
• large_read,large_write: Enables larger read/write operations for better performance
• ServerAliveInterval=60: Sends keep-alive packets every 60 seconds
• ServerAliveCountMax=3: Maximum number of unacknowledged keep-alive packets
• IdentityFile: Specifies the private key file for authentication
• StrictHostKeyChecking=yes: Ensures host key verification
• UserKnownHostsFile: Specifies the known hosts file

Error Handling and Troubleshooting

Check if mount was successful

mount | grep fuse
df -h /mnt/shared_securedata

Common issues and solutions

# If mount fails with permission denied
sudo chmod 755 /mnt/shared_securedata

# If SSH connection is unstable
sshfs don@localhost:/somepath /mnt/shared_securedata \
  -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3

# If performance is slow
sshfs don@localhost:/somepath /mnt/shared_securedata \
  -o cache=yes,kernel_cache,auto_cache,compression=no

Debug SSHFS issues

# Enable verbose logging
sshfs don@localhost:/somepath /mnt/shared_securedata -o debug

# Check SSH connection separately
ssh -v don@localhost

To unmount

# Check current mounts
mount | grep fuse

# Graceful unmount
fusermount -u /mnt/shared_securedata

# Force unmount if needed (use with caution)
sudo umount -f /mnt/shared_securedata

Monitoring and Maintenance

Check mount status and performance

# Monitor I/O statistics
iostat -x 1

# Check disk usage
du -sh /mnt/shared_securedata

# Monitor network connectivity
ping -c 3 localhost

Security monitoring

# Check for unauthorized access attempts
sudo tail -f /var/log/auth.log | grep ssh

# Monitor file access
sudo auditctl -w /mnt/shared_securedata -p wa -k shared_data_access

Regular maintenance tasks

# Verify group membership periodically
getent group fuseaccess

# Check mount point permissions
ls -la /mnt/shared_securedata

Summary of sshfs

Aspect	Performance for Large Files
❌ Network bottleneck	1 GB+ file must go over TCP (SSH) from remote → local
❌ Single-threaded SFTP	SFTP protocol used by sshfs is not parallel or pipelined
❌ Encryption cost	Every byte is encrypted/decrypted by SSH (unless using RDMA or NFS instead)
❌ File seek/read inefficiency	Poor for random access into large files (e.g., scientific datasets)

❌ Not recommended for large file access in HPC. SSHFS is convenient but slow and not scalable.

---

Design Strategy 3: Bind Mount with Namespace Isolation

Overview: Use Linux namespaces and bind mounts to give users isolated views of authorized files.

Implementation:

• Create user-specific mount namespaces
• Bind mount only authorized files/directories into user's namespace
• Use user namespaces to remap UIDs so files appear owned by the user
• Provide containerized or jailed environment for access

Approach: Creates isolated Linux namespaces with bind-mounted repository files for secure, jailed access. This strategy provides the highest level of security by completely isolating user processes from the host system while maintaining native filesystem performance.

Key Components

User Namespace Isolation

• Process isolation: Each user gets a completely isolated process tree
• User ID mapping: User UIDs are remapped within the namespace for security
• Capability restrictions: Users have minimal Linux capabilities within their namespace
• Resource limits: Memory, CPU, and other resources can be strictly controlled

Mount Namespace Isolation

• Private mount propagation: Changes to the mount tree don't affect the host system
• Bind mount precision: Only authorized files are visible through bind mounts
• Read-only enforcement: Files can be mounted read-only to prevent modification
• Mount point cleanup: All mounts are automatically cleaned up when the namespace is destroyed

PID Namespace Isolation

• Process tree isolation: Users see only their own processes within the namespace
• PID remapping: Process IDs are remapped for additional security
• Signal isolation: Signals sent to processes don't affect processes outside the namespace
• Process monitoring: Host system can monitor namespace processes without interference

UID/GID Remapping

• Identity translation: User appears as root within the namespace but has limited access
• File ownership: Files appear to be owned by the user for proper access control
• Permission enforcement: Standard Unix permissions work normally within the namespace
• Security boundaries: Users cannot access files outside their authorized scope

Implementation Flow

Repository Side Namespace Creation

# 1. Create isolated namespace with unshare
unshare --mount --user --pid --fork bash

# 2. Set up UID/GID mapping for the user
echo deny > /proc/$$/setgroups
newuidmap $$ 0 1003 1  # Map namespace UID 0 to host UID 1003 (Bob)
newgidmap $$ 0 1003 1  # Map namespace GID 0 to host GID 1003

# 3. Make mount propagation private
mount --make-rprivate /

# 4. Create view directory structure
mkdir -p /mnt/view
chown 0:0 /mnt/view

# 5. Bind mount authorized files
mount --bind /repo/project42/inputs/config.json /mnt/view/config.json
mount -o remount,bind,ro /mnt/view/config.json  # Make read-only

# 6. Start user shell in the namespace
exec capsh --drop=all -- -c 'exec /bin/bash -l'

User Side Access

# 1. User receives PID file from repository
# 2. User enters the namespace using df_attach
df_attach /run/datafed/attach/g-123.pid

# 3. User sees isolated environment with only authorized files
ls /  # Shows only /mnt/view and basic system directories
cat /config.json  # Access to the authorized file
pwd  # Shows user is in their home directory

Automatic Cleanup

# 1. TTL timer expires or user exits
# 2. Namespace process tree is terminated
# 3. All bind mounts are unmounted
umount -R /mnt/view

# 4. Namespace is destroyed
# 5. Audit log is written
logger -p local0.info "Namespace session ended for user $USER"

Pros

• ✅ Complete process isolation: Users cannot access any system resources outside their namespace
• ✅ Native filesystem performance: Direct access to files without virtualization overhead
• ✅ Precise file visibility control: Only authorized files are visible to users
• ✅ No file duplication: Files remain in their original location
• ✅ Maximum security: Users are completely jailed from the host system
• ✅ Resource control: CPU, memory, and other resources can be strictly limited

Cons

• ❌ Requires kernel namespace support: Needs Linux kernel 3.8+ with namespace support enabled
• ❌ Complex setup and management: Requires careful configuration and monitoring
• ❌ Potential AppArmor/SELinux conflicts: Security modules may interfere with namespace operations
• ❌ Root privileges required for setup: Repository daemon needs elevated privileges
• ❌ Debugging complexity: Issues within namespaces can be difficult to diagnose
• ❌ Resource overhead: Each namespace consumes additional kernel resources

Use Cases

• High-security environments: Government, military, or financial research requiring maximum isolation
• Multi-tenant systems: Shared HPC clusters where user isolation is critical
• Research requiring complete isolation: Sensitive research that cannot risk data leakage
• Compliance-driven deployments: Environments requiring strict security certifications
• Untrusted user access: Providing access to users whose trust level is unknown

Security Features

• Process isolation: Users cannot see or interact with other users' processes
• File system isolation: Users only see files they're authorized to access
• Network isolation: Users can be restricted from network access if needed
• Capability dropping: Users have minimal Linux capabilities within their namespace
• Resource limits: Memory, CPU, and other resources can be strictly controlled

User Sequence Diagram

sequenceDiagram
  participant Bob as HPC User (Bob)
  participant DataFed as DataFed Client
  participant Hub as DataFed Hub
  participant Repo as Repo + Mount Manager
  participant FUSE as FUSE or bindfs Mountpoint
  participant DB as Metadata DB
  participant FS as POSIX Filesystem
  participant KDC as Kerberos Service

  Bob ->> DataFed: Login to DataFed (datafed login)
  DataFed ->> Hub: Verify User
  Hub -->> DataFed: ACK
  Bob ->> DataFed: User requests Repo addr (datafed connect $repo_id)
  DataFed ->> Hub: datafed connect $repo_id
  Hub -->> DataFed: Return Repo Addr
  DataFed ->> Repo: Tries to connect
  alt unable to connect
    Repo -> Bob: Timeout.
  end
  Repo -->> DataFed: Acknowledge connected, return Auth Addr
  DataFed ->> DataFed: Check Ticket
  alt no Ticket
    Bob ->> DataFed: enter username/password
    DataFed ->> KDC: New Ticket request
    KDC -->> DataFed: User Ticket
    DataFed ->> DataFed: Cache Ticket
  end
  DataFed ->> Repo: TGT + datafed token
  Repo ->> Hub: Verifies Datafed User and User logged in with kerberos is same
  Hub ->> Hub: Add Lock flag with username
Hub ->> Repo: Ack, Lock Flag, RW Perm
Repo ->> DB: Locks the file

  Repo ->> FS: Resolve path for /d/abc123
  FS -->> Repo: /repo/data/abc123.txt

  %% Create namespace for user access
  Repo ->> Repo: unshare --user --mount --map-root-user --fork
  Repo ->> Repo: mount --make-private /
  Repo ->> Repo: mkdir /mnt/view

  %% Mount only authorized file
  Repo ->> Repo: mount --bind /repo/data/abc123.txt /mnt/view/abc123.txt
  Repo ->> Repo: mount -o remount,bind,ro /mnt/view/abc123.txt

  %% UID/GID remapping for user namespace
  Repo ->> Repo: newuidmap/newgidmap to map Bob's UID/GID

  %% Start jailed shell or session
  Repo ->> Bob: Namespace shell with /mnt/view/abc123.txt mounted

  Bob ->> FS: Reads file /mnt/view/abc123.txt
  FS -->> Bob: File content

  %% Bob exits jail
  Bob ->> Repo: Exit
  Repo ->> Repo: umount /mnt/view/abc123.txt
Repo ->> DB: Resets Lock flag
  Repo ->> Hub: Release Lock Flag
  Hub -->> Repo: ACK
  Repo ->> DataFed: Operation complete
  DataFed -->> Bob: Session ended

Loading

Steps:

1. User Login: Bob (the HPC User) logs into the DataFed Client with datafed login.

2. Verification: The client contacts the DataFed Hub to verify Bob's credentials.

3. Repo Connection: Bob requests a connection to a specific repository via the client (datafed connect $repo_id). The client gets the repository's address from the Hub.

4. Authentication Handshake: The client connects to the Repo, which acknowledges the connection.

5. Kerberos Ticket: The client checks for a valid Kerberos ticket. If one isn't found, it prompts Bob for his credentials to get a new ticket from the Kerberos Service (KDC).

6. Token Submission: The client sends the Kerberos ticket and a DataFed token to the Repo.

7. Final Permission Grant: The Repo confirms with the Hub that the Kerberos and DataFed users match. The Hub applies a session lock for the user and grants the Repo read/write permissions for the session.

8. File Lock: The Repo immediately locks the requested file in the Metadata DB to prevent conflicts.

9. Path Resolution: The Repo asks the POSIX Filesystem (FS) for the true, physical path of the requested data object (e.g., /d/abc123 resolves to /repo/data/abc123.txt).

10. Create Sandbox: The Repo creates a new, isolated user and mount namespace (unshare). This is a lightweight container or "jail" that separates the user's session from the main system.

11. Create View: Inside this sandbox, the Repo creates a new mount point (e.g., /mnt/view).

12. Mount File: The Repo uses a bind mount to make the specific data file (abc123.txt) appear inside the user's sandboxed view. It then remounts this file as read-only to ensure data integrity.

13. Map User ID: The system maps Bob's user/group ID into the namespace so file ownership and permissions appear correct to him.

14. Start Jailed Session: The Repo provides Bob with a shell session that is locked inside this secure namespace. From Bob's perspective, the filesystem only contains the single file he requested.

15. Read File: Bob reads the file from its path inside his isolated view (/mnt/view/abc123.txt).

16. Content Served: The FS provides the file's content to Bob.

17. Exit: Bob exits the jailed shell session.

18. Unmount: The Repo automatically unmounts the file from the sandboxed view.

19. Release Locks: The Repo resets the lock flag in the Metadata DB and notifies the Hub to release the session lock.

20. Complete: The Repo signals to the DataFed Client that the operation is complete, and the client informs Bob that his session has ended.

---

Here is a simple implementation

Scenario

---

Create a test file

repo_host$ nano ~/mytest.txt

Give permissions to the files

$ chmod o+rx /home/cpatel
$ chmod go+r  /home/cpatel/mytest.txt

Copy the file to tmp location

$ cp mytest.txt /tmp/mytest.txt

Creating the namespace with unshare

$ sudo unshare --user --map-root-user --mount --pid --fork bash

isolates the entire filesystem namespace from any other namespaces on the system

repo_namespace$ mount --make-rprivate /
repo_namespace$ mkdir -p /mnt/view

Read only file path

$# RO_SRC=/tmp/mytest.txt

File path in mount

$# LANDING=/mnt/view/mytest.txt

Create a directory in namespace with the landing path

$# mkdir -p "$(dirname "$LANDING")"

Create a bind mount in the namespace (add comments )

$# mount --verbose --bind "$RO_SRC" "$LANDING"

Mount the created bind mount

$# mount --verbose -o remount,bind,ro "$LANDING"

Check if it has mounted or not

$# mount | grep "$LANDING"

Unmount

$# umount -R /mnt/view/mytest.txt

Retrying without sudo

$ unshare --map-root-user --mount --uts --pid --fork --mount-proc bash

unshare: write failed /proc/self/uid_map: Operation not permitted

$ sudo dmesg | grep -i "apparmor=\"DENIED\""

[  274.186824] audit: type=1400 audit(1752600835.021:127): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=1989 comm="unshare" capability=21  capname="sys_admin"

[  470.742352] audit: type=1400 audit(1752601031.577:129): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=2005 comm="unshare" capability=21  capname="sys_admin"

This tells us definitively that:

An AppArmor security profile is blocking the action.

The profile is named unprivileged_userns.

It is specifically denying the unshare command the sys_admin capability, which is required to create the user namespace mapping, even though it's a "fake" root inside the namespace.

This command installs the app armor utils and allow unprivileged users to create and interact with user namespaces.
Log any actions within those namespaces that would normally violate the associated AppArmor profile for "unprivileged userns".

$ sudo apt-get update && sudo apt-get install apparmor-utils
$ sudo aa-complain unprivileged_userns

Let's create the files we will use to test our jail's isolation.

Create a working directory

$ mkdir -p ~/namespace-test
$ cd ~/namespace-test

Create a file that we WILL allow into the jail

$ echo "This is the secret, authorized content." > authorized_file.txt

Create a file that should NOT be visible from the jail

$ echo "This is sensitive data that should be hidden." > unauthorized_file.txt

$ echo "--- Host View ---"
$ ls -l

This is the core command. We use unshare to launch a new bash shell inside a new set of namespaces, mapping our user to root within them.

$ unshare --map-root-user --mount --uts --pid --fork --mount-proc bash

Prevent any mount changes from propagating back to the host system

$# mount --make-private /

Create the directory for our jail in a location you have write access to

$# mkdir ~/jail

Create a directory inside the jail to hold our authorized content

$# mkdir ~/jail/safe_place

Create an empty placeholder file that will serve as the mount point

$# touch ~/jail/safe_place/secret.txt

Use a bind mount to make ONLY the authorized file visible inside the jail

$# mount --bind ~/namespace-test/authorized_file.txt ~/jail/safe_place/secret.txt

Verify that the unauthorized file is NOT in our prepared jail

$# echo "Searching for the unauthorized file inside the prepared jail..."
$# find ~/jail -name "unauthorized_file.txt" 

Create a 'bin' directory in the jail

$# mkdir ~/jail/bin

Copy the bash executable

$# cp /bin/bash ~/jail/bin/

Create directories for the necessary librarie

$# mkdir -p ~/jail/lib/x86_64-linux-gnu
$# mkdir -p ~/jail/lib64

Copy the libraries that bash depends on

$# cp /lib/x86_64-linux-gnu/libc.so.6 ~/jail/lib/x86_64-linux-gnu/
$# cp /lib/x86_64-linux-gnu/libtinfo.so.6 ~/jail/lib/x86_64-linux-gnu/
$# cp /lib64/ld-linux-x86-64.so.2 ~/jail/lib64/

Enter the chroot jail

$# chroot ~/jail /bin/bash

Check the contents of your new root directory.

$# $# echo "--- Jailed View ---"
$# $# ls /

Confirm you can read the authorized file.

$# $# cat /safe_place/secret.txt

Confirm you CANNOT see anything from the host filesystem.

$# $# ls /home

Exit the Environment

$# $# exit

---

Design Strategy 4: Capability-Based Token System

Overview: Implement a capability token system where the hub issues time-limited tokens that can be exchanged for temporary file access.

Implementation:

• Hub generates signed tokens containing: user ID, file paths, permissions, expiry
• Repository daemon validates tokens and creates temporary hard links in user-accessible directories
• Set permissions on link parent directory (not the files themselves)
• Clean up links after token expiry

Approach: Creates hard links to repository files in private directories, controlled by JWT capability tokens. This strategy provides a simple, efficient way to grant access to repository files while maintaining security through token validation.

Key Components

Hard Link Creation

• Inode sharing: Hard links point to the same inode as the original file, sharing storage
• Private directories: Each user gets a private directory for their file links
• Path isolation: Users cannot see other users' hard links or access unauthorized files
• No duplication: Files are not copied, only linked, saving storage space

JWT Capability Tokens

• One-time use: Each token can only be redeemed once to prevent reuse attacks
• Scope definition: Tokens specify exactly which files and directories are accessible
• Permission encoding: Read, write, or execute permissions are encoded in the token
• Time limitation: Tokens have configurable expiration times for automatic cleanup

Private Directory Structure

• User isolation: Each user gets a unique directory under /tmp/datafed/
• Token-specific paths: Directories include token IDs for additional security
• Permission enforcement: Directories have restrictive permissions (700) to prevent access by other users
• Automatic cleanup: Directories are removed when tokens expire or are revoked

File Access Control

• Permission inheritance: Hard links inherit the permissions of the original file
• User ownership: Links appear to be owned by the user for proper access control
• ACL support: Additional access control can be applied through POSIX ACLs
• Audit logging: All file access is logged for security monitoring

Implementation Flow

Token Request and Link Creation

# 1. User requests capability token from DataFed Hub
datafed token request --scope /project42/inputs/config.json --permissions r --ttl 1h

# 2. Hub returns JWT token with unique identifier
# Token contains: jti (unique ID), user_id, scope, permissions, expiration

# 3. User exchanges token for hard link access
datafed link --token ~/token.jwt

# 4. Repository validates token and creates hard link
mkdir -p /tmp/datafed/u1003/tk-9a12
ln /repo/project42/inputs/config.json /tmp/datafed/u1003/tk-9a12/config.json
chmod 700 /tmp/datafed/u1003/tk-9a12

# 5. Repository returns access path to user
echo "/tmp/datafed/u1003/tk-9a12"

File Access and Usage

# 1. User sets environment variable for easy access
export CFG=/tmp/datafed/u1003/tk-9a12/config.json

# 2. User accesses file normally
cat $CFG
cp $CFG ~/local_analysis/
python3 analyze.py $CFG

# 3. File operations work at native speed
# No virtualization or network overhead

Automatic Cleanup

# 1. TTL timer expires or token is revoked
# 2. Repository removes hard links
find /tmp/datafed -name "tk-9a12" -exec rm -rf {} \;

# 3. Private directories are cleaned up
# 4. Audit log is written with access statistics

Pros

• ✅ Native filesystem performance: Direct access to files without any overhead
• ✅ Simple token-based access control: Easy to implement and manage
• ✅ No file duplication: Files share storage through inode linking
• ✅ Easy cleanup and revocation: Simple unlink operations remove access
• ✅ HPC compatibility: Works seamlessly with all HPC tools and workflows
• ✅ Low resource usage: Minimal memory and CPU overhead

Cons

• ❌ Filenames may be visible: Directory listings might reveal file names to unauthorized users
• ❌ Requires careful permission management: File permissions must be properly configured
• ❌ Hard link cleanup complexity: Finding and removing all hard links can be challenging
• ❌ Limited file hiding: Cannot completely hide file existence from users
• ❌ Permission inheritance: Hard links inherit original file permissions
• ❌ Cross-filesystem limitations: Hard links cannot cross filesystem boundaries

Use Cases

• Large file access in HPC environments: Jobs that need to read/write multi-gigabyte datasets
• Simple read/write operations: Basic file access without complex security requirements
• Performance-critical workflows: Applications that cannot tolerate any performance overhead
• Batch processing jobs: Automated workflows that process multiple files
• Interactive analysis: Users who need to quickly access and modify files

Security Considerations

• Token validation: All tokens must be cryptographically validated
• One-time use: Tokens should only be redeemable once
• Scope enforcement: Token scope must be strictly enforced
• Permission inheritance: Careful attention to file permissions and ACLs
• Directory isolation: User directories must be properly isolated
• Audit logging: All access should be logged for security monitoring

User Sequence Diagram

sequenceDiagram
  participant Bob as HPC User (Bob)
  participant DataFed as DataFed Client
  participant Hub as DataFed Hub
  participant KDC as Kerberos Server
  participant Repo as Repo
  participant FUSE as FUSE or bindfs Mountpoint
  participant FS as POSIX Filesystem

  Bob ->> DataFed: Login to DataFed (datafed login)
  DataFed ->> Hub: Verify User
  Hub -->> DataFed: ACK
  Bob ->> DataFed: User requests Repo addr (datafed connect $repo_id)
  DataFed ->> Hub: datafed connect $repo_id
  Hub -->> DataFed: Return Repo Addr
  DataFed ->> Repo: Tries to connect
  alt unable to connect
    Repo -> Bob: Timeout.
  end
  Repo -->> DataFed: Acknowledge connected, return Auth Addr
  DataFed ->> DataFed: Check Ticket
  alt no Ticket
    Bob ->> DataFed: enter username/password
    DataFed ->> KDC: New Ticket request
    KDC -->> DataFed: User Ticket
    DataFed ->> DataFed: Cache Ticket
  end
  DataFed ->> Repo: TGT + datafed token
  Repo ->> Hub: Verifies Datafed User and User logged in with kerberos is same
  Hub ->> Hub: Add Lock flag with username
    Hub ->> Repo: Ack, Lock Flag, RW Perm

  %% ---------- Authorization + Access Setup ----------
  Bob ->> DataFed: datafed access d/456
  DataFed ->> Hub: Check permissions for Bob on d/456
  Hub -->> DataFed: Access granted

  DataFed ->> Repo: Request hard link for Bob to d/456
  Repo ->> FS: ln /mnt/shared/master/file_456.txt /mnt/shared/bob/file_bob.txt
  FS -->> Repo: Hard link created
  Repo -->> DataFed: ACK - file_bob.txt ready

  %% ---------- File Access Phase ----------
  DataFed ->> DataFed: file_bob.txt ready in /mnt/shared/bob
  DataFed ->> FUSE: Open /mnt/shared/bob/file_bob.txt
  FUSE ->> FS: Read/write access
  FS -->> FUSE: File data
  FUSE -->> Bob: File read/write complete

  %% ---------- Session Completion + Cleanup ----------
  Bob ->> DataFed: File access / Session complete
  DataFed ->> Repo: Dispose hard link /mnt/shared/bob/file_bob.txt
  Repo ->> FS: unlink /mnt/shared/bob/file_bob.txt
  FS -->> Repo: Link removed
  Repo -->> DataFed: Cleanup done

Loading

Steps:

1. User Login: Bob (the HPC User) initiates a login to the DataFed Client using the command datafed login.

2. Verification: The DataFed Client contacts the DataFed Hub to verify Bob's identity. The Hub acknowledges the verification.

3. Repo Connection: Bob requests a connection to a specific repository (datafed connect $repo_id). The client asks the Hub for the repository's network address and receives it.

4. Authentication Handshake: The DataFed Client connects to the Repo, which acknowledges the connection and returns its authentication address.

5. Kerberos Ticket Check: The client checks for a cached Kerberos authentication ticket. If one is not found, it prompts Bob for his credentials and requests a new ticket from the Kerberos Server (KDC).

6. Token Submission: The client sends the Kerberos ticket (TGT) and a DataFed token to the Repo.

7. Final Permission Grant: The Repo confirms with the Hub that the Kerberos user and DataFed user are the same. The Hub applies a session lock for the user and grants the Repo read/write permissions for the session.

8. Access Request: Bob, through the client, requests access to a data object (datafed access d/456).

9. Permission Check: The DataFed Client checks with the Hub to confirm Bob has permission to access d/456. The Hub grants access.

10. Hard Link Creation: The DataFed Client requests that the Repo create a hard link to the file for Bob.

11. Filesystem Operation: The Repo instructs the POSIX Filesystem (FS) to create a hard link from the master file to a file in Bob's personal directory (e.g., ln /mnt/shared/master/file_456.txt /mnt/shared/bob/file_bob.txt).

12. Confirmation: The FS confirms the link is created, and the Repo notifies the DataFed Client that the file is ready for use at the new path (file_bob.txt).

13. File Open: Bob opens the file for reading or writing. This request goes through the FUSE layer.

14. Direct Access: The FUSE layer forwards the read or write request directly to the underlying Filesystem.

15. Data Transfer: The Filesystem performs the requested operation and returns the file data to FUSE, which then passes it back to Bob. There is no file-level lock check performed at this stage.

16. Access Complete: Bob signals that he is finished with the file or that his session is complete.

17. Cleanup Request: The DataFed Client instructs the Repo to dispose of the hard link.

18. File Unlink: The Repo tells the FS to remove the hard link (unlink /mnt/shared/bob/file_bob.txt).

19. Complete: After the FS confirms the link is removed, the Repo notifies the DataFed Client that cleanup is complete.

---

Here is a simple implementation

$ echo "secret" > file1
$ ln file1 file2  # ln sourceFile destFile

If user does not own the file

$ sudo chown username: file1
$ ln file1 file2  # ln sourceFile destFile

To check hard link is created

$ ls -li file1 file2  # ls -li sourceFile destFile
# returns 
12345 -rw-r--r-- 2 user user 11 Jul 28 10:00 file1
12345 -rw-r--r-- 2 user user 11 Jul 28 10:00 file2

Deleting a hardlink

$ rm file1
# This will delete file1 
$ cat file2
# It will redirect to the inode, still having the contents. 
# So we need to delete file2 as it is hard link to file1's inode

Other way around deleting all the hard links

$ ls -li file1 file2  # ls -li sourceFile destFile
# returns 
12345 -rw-r--r-- 2 user user 11 Jul 28 10:00 file1
12345 -rw-r--r-- 2 user user 11 Jul 28 10:00 file2

# Here 12345 is the inum that is linked to inode
$ find /mnt /home -xdev -inum 12345

# returns
/mnt/temp1/file1
/home/cpatel/tmp3/file2

# Delete these paths
# This will delete all the hard links and the raw data.
$ rm /mnt/temp1/file1
$ rm /home/cpatel/tmp3/file2

Alternatively we can manage the file remove functionality in DataFed Client

datafed removeRawByID $file1_id # Use the id starting with d/_id 

OR

When deleting from Client Web, the raw data can deleted without removing the metadata.

This will go into the shell/bash and look for the file, get the inum of the file, find all the hard links user has created.

The user can create hard link anywhere from /mnt/(Allocation spaces) and ~/

$ find / -xdev -type f -printf '%i %p\n' > inode_map.txt

$ awk '{inode[$1] = inode[$1] ? inode[$1] ORS $2 : $2; count[$1]++}
     END {for (i in inode) if (count[i] > 1) print "Inode:", i, "\n" inode[i] "\n"}' inode_map.txt

But the process of finding all the hard links is costlier

FS Size	# Files	find + stat time
100K files	~1 GB	~2–4 seconds (SSD)
1M files	~10–50 GB	~15–30 seconds
10M+ files	100+ GB	2–5 minutes

Once the user's session is ended, the datafed client has the file and knows where the hardlink is created or atleast the datafed client has the inum and it can find all the hardlinks related to that inum

$ find /mnt /home -xdev -inum 1725343 -print | tee removed_links.log | xargs rm -f

---

Some limitations

1. Cannot Cross Filesystems

   • Hard links must be on the same filesystem (same device).
   • This is enforced by the kernel because hard links point to inodes, which are specific to a filesystem.

2. Cannot Link to Directories (by default)

   • Most systems forbid hard links to directories to prevent infinite loops and filesystem corruption.
   • Only root (with special flags) can do it, and it's extremely rare and discouraged.

3. Indistinguishable from the Original

   • All hard links to a file are equal — there is no "original".

   • This makes it hard to track down which filename created the content.

4. Breaks When Files Are Replaced (Not Edited)

   • If a file is edited, all hard links reflect the change (they share content).

   • But if a file is replaced (e.g., echo "x" > file), a new inode is created, and old links are unaffected — now pointing to stale data.

---

Here is a simple implementation

Scenario

Repo, bob and gray are three regular users in the same group. Bob and gray wants to access a file which is owned by Repo via hard-link. Bob has to update the file and gray will use the different file for a task. Once they are done with the file the hard-link should be removed. The file bob has access, should not able to accessible by gray and vice versa.

Sudo Required commands

For adding users in the group

# Create a group
$ sudo groupadd sharedaccess

# Add users to the group `sharedaccess`
$ sudo usermod -aG sharedaccess bob
$ sudo usermod -aG sharedaccess repo
$ sudo usermod -aG sharedaccess gray

bob
cd /mnt/shared/acl

repo
cd
Add file to the sharedaccess

# Add the dir to the group `sharedaccess`
$ sudo chgrp sharedaccess /mnt/shared/acl/
/mnt/shared

# Change permission so repo, bob and gray have read/write permissions
$ chmod 660 /mnt/shared

Repo has a shared directory in /mnt/shared

Bob and Gray have their own directories in /mnt as access_bob and access_gray respectively.

All the directories in /mnt/shared has the same group as sharedaccess

Repo Creates an Hardlink for bob.

$ ln /mnt/shared/newdata.txt /mnt/shared/access_bob/newdata.txt

bob Updates the file.

$ nano /mnt/shared/access_bob/newdata.txt

# Result
Repo created file
Bob was here

bob ends his session

Repo revokes hard-link for bob

rm -f /mnt/shared/access_bob/newdata.txt

Repo creates an Hard-link for gray

ln /mnt/shared/newdata.txt /mnt/shared/access_gray/newdata.txt

Gray reads the file

more /mnt/shared/access_tester/newdata.txt

# Result
Repo created file
Bob was here

Gray later updates the file

nano /mnt/shared/access_tester/newdata.txt

# Result
Repo created file
Bob was here
Gray modified the file

Repo revokes the permission for Gray when session exits.

rm -f /mnt/shared/access_gray/newdata.txt

---

Design Strategy 5: WebDAV/SFTP Gateway with Authorization Proxy

Overview: Deploy protocol gateways that authenticate users and proxy authorized file operations.

Implementation:

• Run WebDAV or SFTP servers on repositories
• Gateway authenticates users against hub
• Caches authorization decisions with TTL
• Performs file operations as service account on user's behalf

Approach: Provides SFTP access to repository files through a gateway service with token-based authentication. This strategy offers a familiar interface for many users while maintaining security through centralized token validation.

Key Components

SFTP Gateway Service

• Protocol support: Implements standard SFTP protocol for file transfer operations
• User authentication: Validates users through DataFed authentication systems
• File operation proxying: Intercepts SFTP commands and validates them against tokens
• Session management: Maintains user sessions and tracks file access

Token Validation System

• Real-time validation: Tokens are validated for each file operation
• Permission checking: Verifies that users have appropriate permissions for requested operations
• Token caching: Caches validation results to improve performance
• Revocation support: Immediately revokes access when tokens are invalidated

Service Account Operations

• Privileged access: Gateway runs with sufficient privileges to access repository files
• User impersonation: Operates on behalf of authenticated users
• Permission mapping: Translates user permissions to filesystem operations
• Audit logging: Records all file operations for security monitoring

Session-based Access Control

• Connection tracking: Monitors active SFTP connections and user sessions
• Timeout management: Automatically closes inactive sessions
• Resource limits: Enforces limits on concurrent connections and file operations
• Cleanup procedures: Ensures proper cleanup when sessions end

Implementation Flow

Gateway Setup and Configuration

# 1. Install and configure SFTP gateway
apt install openssh-server
mkdir -p /opt/datafed/sftp_gateway

# 2. Configure SSH daemon for SFTP-only access
cat >> /etc/ssh/sshd_config << EOF
Match Group sftp_users
    ChrootDirectory /opt/datafed/sftp_gateway
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no
EOF

# 3. Create SFTP user group
groupadd sftp_users
usermod -aG sftp_users bob

# 4. Start SSH service
systemctl enable --now ssh

User Authentication and Access

# 1. User obtains access token from DataFed Hub
datafed sftp-token --scope /project42 --permissions rw --ttl 2h

# 2. User connects via SFTP using token
sftp -i ~/sftp_token bob@gateway.example.com

# 3. Gateway validates token and establishes session
# 4. User performs SFTP operations
sftp> ls
sftp> get config.json
sftp> put results.txt
sftp> quit

File Operation Processing

# 1. User sends SFTP command (e.g., GET file.txt)
# 2. Gateway validates user's token and permissions
# 3. Gateway checks if file is within user's scope
# 4. Gateway performs operation on underlying filesystem
# 5. Results are returned to user through SFTP protocol

Pros

• ✅ Standard SFTP protocol support: Familiar interface for many users
• ✅ Familiar interface for many users: Leverages existing SFTP client software
• ✅ Good for file transfer workflows: Efficient for moving files in and out of repositories
• ✅ No local filesystem mounting required: Works from any system with SFTP support
• ✅ Cross-platform compatibility: Works on Windows, macOS, and Linux
• ✅ Simple client requirements: Only needs standard SFTP client software

Cons

• ❌ Not suitable for direct file access: Files must be transferred before use
• ❌ Performance overhead for large files: Network transfer adds latency and bandwidth usage
• ❌ Limited integration with HPC workflows: Not suitable for in-place file processing
• ❌ Requires separate client software: Users need SFTP clients installed
• ❌ Session management complexity: Must track and manage active sessions
• ❌ File locking limitations: No native support for file locking or concurrent access

Use Cases

• File transfer operations: Moving files between local systems and repositories
• Users familiar with SFTP: Researchers who prefer SFTP for file management
• Cross-platform compatibility needs: Users working on different operating systems
• Simple file access patterns: Basic read/write operations without complex workflows
• Remote access scenarios: Users accessing repositories from external networks

Performance Characteristics

• Small files: Good performance for files under 100MB
• Large files: Performance degrades with file size due to network transfer overhead
• Concurrent access: Multiple users can access different files simultaneously
• Network dependency: Performance heavily depends on network conditions
• Caching: Limited caching opportunities due to stateless nature of SFTP

User sequence Diagram for webdav

sequenceDiagram
    %% PARTICIPANTS
    actor BobBrowser  as Bob – browser
    actor BobShell    as Bob – shell / CLI
    participant Hub   as DataFed Hub
    participant Client as WebDAV client
    participant Gateway as NGINX + mod_dav
    participant Proxy as Auth‑proxy
    participant FS    as POSIX FS

    %% 0. Bob logs in to the Hub UI
    BobBrowser->>Hub: OAuth login
    Hub-->>BobBrowser: Session cookie

    %% 1. Bob requests WebDAV token
    BobBrowser->>Hub: "Get WebDAV token"<br/>path=/project42/inputs/config.json<br/>mode=RO ttl=1 h
    Hub-->>BobBrowser: token tk‑9a12 (download)<br/>file saved as ~/token.jwt

    %% 2. Bob mounts the share from his shell
    BobBrowser-->>BobShell: info (URL + token file path)
    BobShell->>Client: mount -t davfs https://repo.example/dav ~/repo42 \\ -o token=/home/bob/token.jwt

    %% 3. TLS connection & auth
    Client->>Gateway: PROPFIND /dav/…  (Authorization: Bearer tk‑9a12)
    Gateway->>Proxy: /auth_dav  token=tk‑9a12
    Proxy->>Hub: POST /introspect  tk‑9a12   (cache miss)
    Hub-->>Proxy: 200 OK  user=bob  scope=/project42  cache=15 min
    Proxy-->>Gateway: 200 allow

    %% 4. Data request
    Client->>Gateway: GET /dav/project42/inputs/config.json
    Gateway->>Proxy: /auth_dav  (cache hit) → allow
    Gateway->>FS: open RO /repo/project42/inputs/config.json  (svc UID)
    FS-->>Gateway: fd → data
    Gateway-->>Client: 200 OK  (stream file)
    Client-->>BobShell: file delivered to ~/repo42/…

    %% 5. Hub pushes revoke before TTL
    Hub-->>Proxy: PUSH revoke user=bob
    Proxy->>Proxy: purge bob cache

    %% 6. Next client request is denied
    Client->>Gateway: GET /dav/project42/inputs/config.json
    Gateway->>Proxy: /auth_dav  (cache miss → deny)
    Proxy-->>Gateway: 403 Forbidden
    Gateway-->>Client: 403 Forbidden
    Client-->>BobShell: I/O error (access revoked)

Loading

Steps:

1. Web Login: Bob logs into the DataFed Hub via his web browser, likely using a standard method like OAuth. The Hub responds by giving his browser a session cookie.

2. Token Request: Within the Hub's web UI, Bob requests a WebDAV token. He specifies the exact path he needs access to (/project42/inputs/config.json), the access mode (Read-Only), and a time-to-live (1 hour).

3. Token Download: The Hub generates a unique token (tk‑9a12) and provides it to Bob's browser as a downloadable file, which he saves as ~/token.jwt.

4. Mount Command: Bob switches to his shell and uses a WebDAV client (like davfs) to execute a mount command.

5. Authentication: He provides the repository URL and specifies the path to the downloaded token file, which will be used for authentication instead of a password.

6. First Request: The WebDAV client sends an initial request (PROPFIND to list directory properties) to the NGINX Gateway. It includes the token in the Authorization: Bearer header.

7. Auth Check: The Gateway passes the token to the Auth-proxy for validation.

8. Token Introspection: Since the Auth-proxy has never seen this token before (cache miss), it contacts the DataFed Hub to validate it.

9. Hub Validation: The Hub confirms the token is valid for user "bob" and the requested scope (/project42). It tells the proxy this validation can be cached for 15 minutes.

10. Access Granted: The Auth-proxy gives the Gateway the green light to proceed.

11. File Request: The WebDAV client sends a GET request to read the config.json file.

12. Cached Auth: The Gateway checks with the Auth-proxy again. This time, the proxy finds the valid token in its cache (cache hit) and immediately allows the request without contacting the Hub.

13. File Retrieval: The Gateway reads the file from the POSIX Filesystem.

14. File Delivery: The Gateway streams the file's content back to the WebDAV client, which makes it available to Bob in his local mount point (~/repo42).

15. Revoke Command: For security or administrative reasons, the Hub pushes a command to the Auth-proxy to revoke all access for user "bob".

16. Cache Purge: The Auth-proxy immediately purges its cache, deleting the stored validation for Bob's token.

17. New Request: The WebDAV client sends another request for the file.

18. Auth Failure: The Gateway checks with the Auth-proxy. The proxy can't find the token in its cache and now denies the request (cache miss → deny).

19. Forbidden Error: The proxy returns a 403 Forbidden error to the Gateway, which forwards it to the WebDAV client.

20. I/O Error: The client reports an "I/O error" to Bob's shell, indicating that his access has been revoked and the mount is no longer functional.

User sequence Diagram for SFTP

sequenceDiagram
    %% PARTICIPANTS
    actor Bob            as Bob
    participant Hub      as DataFed Hub
    participant Gateway  as sshd (+ sftp‑subsystem)
    participant Proxy    as Auth‑proxy (path authoriser)
    participant FS       as POSIX FS

    %% 0. Bob asks the Hub for an ephemeral SFTP certificate
    Bob->>Hub: Request SSH cert (TTL 2 h) for /project42 scope
    Hub-->>Bob: hub_bob_cert.pub  (signed OpenSSH certificate)

    %% 1. Bob starts the SFTP session
    Bob->>Gateway: SSH handshake  (cert=hub_bob_cert.pub, user=bob)

    %% 2. sshd validates the certificate
    Gateway->>Hub:  AuthorizedPrincipalsCommand check (bob, cert sig)
    Hub-->>Gateway:  OK  expires=2 h
    Gateway-->>Bob:  SSH success  (opens SFTP subsystem)

    %% 3. Bob downloads a file  (READ)
    Bob->>Gateway:  SFTP: OPEN /project42/inputs/config.json  flags=READ
    Gateway->>Proxy:  /authorize  user=bob  path=/project42/inputs/config.json  mode=RO
    Proxy-->>Gateway:  allow   (cached 15 min)
    Gateway->>FS:  open RO /repo/project42/inputs/config.json  (svc UID)
    FS-->>Gateway:  fd ➜ data
    Gateway-->>Bob:  SFTP: DATA

    %% 4. Bob uploads scratch output  (WRITE)
    Bob->>Gateway:  SFTP: OPEN /project42/scratch/bob_tmp/out.dat  flags=WRITE
    Gateway->>Proxy:  /authorize (cached miss → ask Hub)
    Proxy-->>Gateway:  allow
    Gateway->>FS:  open RW  /repo/project42/scratch/bob_tmp/out.dat
    FS-->>Gateway:  fd
    Gateway-->>Bob:  SFTP: OK  (Bob sends DATA packets)

    %% 5. Hub revokes Bob before TTL expires
    Hub-->>Proxy:  PUSH revoke user=bob
    Proxy->>Proxy: purge bob cache

    %% 6. Next Bob operation is denied, channel closes
    Bob->>Gateway:  SFTP: OPEN /project42/inputs/config.json flags=READ
    Gateway->>Proxy: /authorize  (cache miss, revoked)
    Proxy-->>Gateway: deny
    Gateway-->>Bob:  SFTP: STATUS = PERMISSION DENIED
    Bob-->>Gateway:  SSH disconnect

Loading

Steps:

1. Request Certificate: Bob asks the DataFed Hub for an ephemeral (short-lived) SSH certificate. He specifies the scope of his access (e.g., /project42) and a Time-To-Live (TTL) of 2 hours.

2. Receive Certificate: The Hub generates a unique, signed OpenSSH certificate and sends the file (hub_bob_cert.pub) back to Bob.

3. SSH Handshake: Bob initiates an SFTP connection to the Gateway (sshd server), presenting the certificate as his method of authentication.

4. Certificate Validation: The Gateway uses a command (AuthorizedPrincipalsCommand) to ask the DataFed Hub if the certificate is valid for the user "bob".

5. Hub Confirmation: The Hub verifies the certificate's signature, confirms it's valid, and notes its 2-hour expiration.

6. Connection Success: The Gateway authenticates Bob and opens the SFTP subsystem, establishing the session.

7. Open Request: Bob sends a command to open a file for reading (/project42/inputs/config.json).

8. Authorize Path: The Gateway forwards the request details (user, path, and read-only mode) to the Auth-proxy.

9. Permission Granted: The Auth-proxy confirms Bob is allowed to read this file, allows the request, and caches this permission for 15 minutes.

10. File Retrieval: The Gateway opens the file on the POSIX Filesystem and streams its data back to Bob.

11. Write Request: Bob attempts to open a different file for writing in his scratch directory.

12. Authorize Path (Cache Miss): The Gateway again asks the Auth-proxy for permission. Since this is a new path, it's a cache miss, forcing the proxy to re-evaluate permissions (likely by consulting the Hub or its own policies).

13. Permission Granted: The Auth-proxy allows the write operation.

14. File Creation: The Gateway creates the file on the Filesystem and sends an "OK" back to Bob, who can now begin sending the file's data packets.

15. Push Revocation: The Hub sends a PUSH notification to the Auth-proxy, instructing it to immediately revoke all access for user "bob".

16. Purge Cache: The Auth-proxy purges all of Bob's cached permissions.

17. New Request: Within the same SFTP session, Bob attempts another file operation.

18. Authorization Denied: The Gateway checks with the Auth-proxy. Because the cache was purged and the user is marked as revoked, the proxy denies the request.

19. Permission Denied Error: The Gateway sends a PERMISSION DENIED status to Bob's client.

20. Disconnect: The session can no longer proceed, and Bob disconnects.

---

Here is a simple implementation for SFTP

# Create group and add repo users
groupadd -f sftp_users
usermod -aG sftp_users bob               # repeat for each DataFed user

systemctl reload sshd
Gateway side complete ✔️

User / Client Side (Bob)

Get an ephemeral SFTP certificate from the Hub

# Browser UI or CLI; CLI example:
datafed ssh-cert get --ttl 2h --scope /project42 > ~/hub_bob_cert
chmod 600 ~/hub_bob_cert

Start SFTP session using that cert

sftp -o IdentityFile=~/hub_bob_cert bob@repo.example
Behind the scenes: sshd validates the cert with the Hub; each OPEN/PUT
is authorised by the proxy.

Do normal file work

sftp> get /project42/inputs/config.json
sftp> put results.txt /project42/scratch/bob_tmp/

What happens on revoke / expiry

If the Hub revokes Bob early, his next SFTP command returns:
Permission denied.

When TTL (2 h) is reached, the certificate is invalid; the session closes and Bob must request a fresh cert.

End-to-End Picture

SSH cert   ──▶  sshd (port 2022)  ──┐
                                     │  AuthorizedPrincipalsCommand  → Hub: cert valid?
SFTP OPEN path=/project42/…          │
          ──────────────────────────▶│
                                     │
              sftp_wrapper  (strace) │
                                     │  /authorize?user=bob&path=…&mode=RW
                                     ├─▶ auth_proxy (FastAPI) ──▶ Hub  (cache miss)
                                     │          ▲
                                     │          └── push /revoke bob
                                     │
Allowed ➜ openat() as repo_svc UID   │
  Deny  ──┐                           │
          ▼                           │
      SFTP STATUS=PERMISSION_DENIED ◀─┘

---

Strategy Comparison Matrix

The following matrix provides a detailed comparison of all five strategies across key dimensions:

Feature	Strategy 1 (Temp Users + ACLs)	Strategy 2 (FUSE + Tokens)	Strategy 3 (Namespaces)	Strategy 4 (Hard Links)	Strategy 5 (SFTP Gateway)
Performance	✅ Native filesystem speed No virtualization overhead Direct kernel I/O	⚠️ FUSE overhead (25-50%) Userspace filesystem layer Context switching overhead	✅ Native filesystem speed Direct kernel access No virtualization overhead	✅ Native filesystem speed Direct inode access No performance penalty	❌ Network transfer overhead SFTP protocol overhead Serial file operations
Security	✅ High isolation through user separation POSIX ACL enforcement SSH key authentication	✅ Token-based access control FUSE-level permission checking Session isolation	✅ Complete process isolation Namespace boundaries No privilege escalation paths	⚠️ File visibility concerns Permission inheritance Token-based access control	✅ Gateway isolation Token validation Service account separation
Complexity	⚠️ Medium complexity User account management ACL configuration SSH key handling	⚠️ Medium complexity FUSE daemon management Token refresh logic Mount point handling	❌ High complexity Namespace setup UID/GID mapping Mount propagation	✅ Low complexity Simple link creation Token validation Basic cleanup	✅ Low complexity Standard SFTP setup Token integration Session management
HPC Compatibility	✅ Excellent compatibility Native filesystem access Full POSIX support MPI and Slurm integration	⚠️ Good compatibility FUSE mount points Most HPC tools work Some edge cases	✅ Excellent compatibility Native filesystem access Full POSIX support Complete tool integration	✅ Excellent compatibility Native filesystem access Full POSIX support Seamless workflow integration	❌ Limited compatibility File transfer only No in-place processing Not suitable for compute jobs
File Locking	✅ Native file locking flock() and fcntl() support Concurrent access control HPC workflow support	✅ FUSE-based locking Application-level coordination Concurrent access management Cross-mount coordination	✅ Native file locking flock() and fcntl() support Complete isolation Secure concurrent access	✅ Native file locking flock() and fcntl() support Standard POSIX semantics HPC workflow support	❌ No file locking Transfer-based access No concurrent access control Not applicable
Setup Requirements	⚠️ User account creation SSH key generation ACL configuration Systemd timer setup	✅ Token-based setup FUSE daemon deployment Mount point configuration Token refresh logic	❌ Root privileges required Namespace support Kernel configuration Complex setup scripts	✅ Token-based setup Simple link creation Directory management Basic cleanup scripts	✅ Gateway service setup SSH configuration User group management Token integration
Cleanup	✅ Automatic cleanup Systemd timers ACL revocation User account removal	✅ Token expiry cleanup Mount point unmounting Session termination Resource cleanup	✅ Namespace cleanup Mount unmounting Process termination Resource cleanup	✅ Link removal Directory cleanup Token expiry Simple unlink operations	✅ Session end cleanup Connection termination Resource cleanup Token invalidation

---

Conclusion

Each design strategy offers different trade-offs between performance, security, complexity, and operational requirements. The choice depends on specific requirements and constraints:

Strategy 1 — Temp POSIX users + per-file ACLs

• Hub asks the repo to create (or reuse) a temporary POSIX account, grant setfacl on the specific files, hand the user short-lived SSH/Kerberos creds, and revoke ACLs on logout/TTL. Delivers native POSIX I/O with clean audit/revoke steps.

Strategy 2 — Token-validated FUSE mount (df-fuse)

• User mounts a virtual tree; the FUSE daemon checks the Hub on lookups/opens, refreshes tokens, and unmounts or returns EACCES on revoke/expiry. Great for “show only what’s allowed,” but adds a user-space hop.

Strategy 3 — Namespace “view jail” (bind-mounts)

• Repo spawns an isolated user+mount namespace, bind-mounts just the authorized paths into a minimal view (RO/RW as granted), and tears it down on TTL. Users work at full POSIX speed inside the jail; attach via a small df_attach helper.

Strategy 4 — Hard-link “view dirs” (token-gated)

• Hub issues a short-lived capability token; repo redeems it once and builds a private 0700 view dir populated with hard links to the allowed files; cleanup simply unlinks them at TTL/revoke. Delivers the simplest native POSIX path and immediate revoke (links disappear).

Strategy 5 — SFTP/WebDAV gateway

• Gateway authenticates with short-lived SSH certs (SFTP) or JWTs (WebDAV) and consults an auth-proxy per open; revokes are honored on the next request. Suited for file transfer/export, not in-place compute, because clients copy files locally to open/edit them.

Key trade-offs

• Performance: Strategies 1/3/4 are native POSIX; 2 adds FUSE overhead; 5 is transfer-oriented.
• Isolation/least exposure: 3 shows only mounted paths; 2 hides unauthorized entries via the VFS view; 1 may reveal filenames; 4 exposes only the private view dir.
• Revocation: 4 unlinks instantly; 1 setfacl -x; 2 unmount/deny; 3 kills namespace; 5 denies next request.

The key to successful implementation is understanding the specific requirements of the environment and choosing the strategy (or combination of strategies) that best balances performance, security, complexity, and operational requirements.