syketagCompliance/DesignDoc

# LogJensticks - Application Management System

## Project Overview
LogJensticks is a trucking application management system that automates the processing of job applications through document verification, OCR analysis, and automated checks.

## Architecture

### Infrastructure
- **2 Docker Containers**: Go server + MongoDB database orchestrated with Docker Compose
- **Build System**: Makefile for building, running, and testing
  - `make build` - Compiles code and builds Docker containers
  - `make run` - Starts all containers
  - `make run server` - Starts only the server
  - `make run db` - Starts only MongoDB
  - `make run test` - Runs unit tests

---

## API Endpoints

### Authentication
| Method | Endpoint | Access | Description |
|--------|----------|--------|-------------|
| POST | `/login` | Public | User login with credentials |
| POST | `/logout` | Authenticated | User logout |
| GET | `/health` | Public | Health check endpoint |

> **Unregistered Flow**: Truckers may use the system without registering. Upon first upload/submit the server issues a random `unreg_token` stored in a **secure, HTTP‑only** cookie. "Secure" means the browser will only send it over HTTPS connections, preventing exposure on unencrypted networks. "HTTP‑only" means JavaScript running in the browser cannot read the cookie; this reduces the risk of token theft via XSS. All subsequent requests (save, submit, get, status) require that cookie; the token is a single-use key that only allows access to applications created with the same token. Tokens cannot access others' applications. Registration upgrades the session to a normal authenticated user and allows reuse of prior documents.

### Application Management (Trucker)
| Method | Endpoint | Access | Description |
|--------|----------|--------|-------------|
| POST | `/saveApplication` | Public or cookie | Save application as draft (auto-generates `app_id`). Sets/reads `anon_token` cookie when user is not registered. |
| POST | `/submitApplication` | Public or cookie | Submit application for review (sets `submitted_timestamp`); requires valid token if unregistered. |
| GET | `/getApplication/:applicationID` | Authenticated or cookie | Retrieve application details. Registered users may view their whole history; unregistered users may only view apps tied to their current `unreg_token`.  |
| GET | `/applicationStatus/:applicationID` | Authenticated or cookie | Check status and results with same access rules as above. |

### Application Review (Approver/Manager)
| Method | Endpoint | Access | Approvers Only | Description |
|--------|----------|--------|---|-------------|
| GET | `/approvalQueue` | Yes | - | Get list of pending applications |
| GET | `/getApplication/:id` | Yes | - | Retrieve application for review (shows all details including check results) |
| POST | `/approve/:applicationID` | Yes | - | Approve application with optional internal comments |
| POST | `/reject/:applicationID` | Yes | - | Permanently reject application with internal reason (applicant will not see reason) |
| POST | `/return/:applicationID` | Yes | - | Return application to applicant for corrections with required feedback message |

### User Management (Admin/Manager)
| Method | Endpoint | Access | Description |
|--------|----------|--------|-------------|
| POST | `/createUser/:username/:token` | Token Required | Create new user (password in request body, token must match DB record) |
| GET | `/users` | Admin Only | List all users and their roles |
| PUT | `/updateUser/:username` | Admin Only | Update user details or role |

### Dashboard (Optional)
| Method | Endpoint | Access | Description |
|--------|----------|--------|-------------|
| GET | `/dashboard/stats` | Admin Only | System statistics and metrics |

---

## Frontend Pages

### Public Pages (HTML/CSS/Vue.js)
- `/login` - User authentication form

### Trucker Pages (Authenticated or Unregistered)
- `/apply` - Application submission form (multi-step form with file uploads). Works without login; unregistered users are issued a token cookie. Registered users can save and reuse documents.
- `/applicationConfirmation` - Post-submission confirmation with application ID and notice about token cookie for unregistered users.
- `/applicationStatus` - Track application progress and view check results. Unregistered users may only access apps created with current token.
- `/register` - Optional sign-up page for users who want an account to retain history and reuse files.

### Approver/Manager Pages (Authenticated + Authorized)
- `/approvalQueue` - List of pending applications for review
- `/reviewApplication/:applicationID` - Detailed application review interface with approve/reject actions
- `/dashboard` - Manager dashboard with application statistics

---

## Data Schema

### Application Document Fields

#### User-Provided Fields
- `load_num` (string) - Load number
- `carrier_name` (string) - Carrier/company name
- `MC_num` (string) - Motor Carrier number (validated with regex)
- `driver_name` (string) - Full name of driver
- `driver_license_num` (string) - License number (numbers only)
- `truck_plate_num` (string) - License plate number
- `driver_cell_phone` (string) - Phone number (must be valid format)
- `truck_num` (string) - Truck identifier
- `trailer_num` (string) - Trailer identifier

#### Document Uploads (JPEG, PNG, PDF)
- `driver_with_truck_num` - Picture showing driver, truck, and visible US DOT/MC/KYU numbers
- `license_front` - Front of driver's license
- `license_back` - Back of driver's license
- `apportioned_cab_card` - Apportioned cab card

#### System-Generated Fields (on submission)
- `_id` (UUID) - MongoDB document ID, same as `app_id`
- `app_id` (UUID) - Application ID
- `submitted_timestamp` (UTC timestamp) - When application was submitted
- `status` (string) - Current status: "draft", "submitted", "processing", "approved", "rejected", "returned", "human_review"
- `approver_notes` (string, optional) - Internal notes from approver (not visible to applicant)
- `correction_feedback` (string, optional) - Feedback message sent to applicant when application is returned
- `unreg_token` (string, optional) – Random token assigned to unregistered users; stored in cookie and used to authorize access to this application

#### Other Metadata
- `created_by_user` (string, optional) – Username or ID if submitted by a registered user (otherwise absent)
- `last_modified` (UTC timestamp) – When the record was last updated

#### Check Result Fields (added during processing)
- `[CheckName]` (string) - One field per check with value: "processing", "pass", "fail", or "human_review"

---

## Application Processing Pipeline

### Submission Flow
When an application is submitted, it goes through three stages:

1. **Storage** - Application stored in MongoDB with all user-provided fields and documents
2. **OCR Processing** - Optical Character Recognition extracts data from uploaded document images
3. **Automated Checks** - Validation checks run to verify application completeness and accuracy

### Document File Storage Structure
File uploads are stored with metadata:
```json
{
    "file_uploaded": "<<utc timestamp>>",
    "file_location": "/path/to/file"
}
```

For example, during initial submission:
```json
{
    "_id": "<<uuid>>",
    "app_id": "<<uuid>>",
    "load_num": "5395746",
    "carrier_name": "1 Fab Express",
    "MC_num": "1206193",
    "driver_name": "Fabiano Caruana",
    "driver_license_num": 060414478,
    "truck_plate_num": "C180XZ",
    "driver_cell_phone": 3859827777,
    "truck_num": 1234,
    "trailer_num": 53302,
    "submitted_timestamp": "<<utc timestamp>>",
    "status": "processing",
    "driver_with_truck_num": {
        "file_uploaded": "<<utc timestamp>>",
        "file_location": "/path/to/file"
    },
    "license_front": {
        "file_uploaded": "<<utc timestamp>>",
        "file_location": "/path/to/file"
    },
    "license_back": {
        "file_uploaded": "<<utc timestamp>>",
        "file_location": "/path/to/file"
    },
    "apportioned_cab_card": {
        "file_uploaded": "<<utc timestamp>>",
        "file_location": "/path/to/file"
    }
}
```

### OCR Processing

OCR data extracted from images is stored within the document field's `ocr_results` object:

```json
{
    "driver_with_truck_num": {
        "file_uploaded": "<<utc timestamp>>",
        "file_location": "/path/to/file",
        "ocr_results": {
            "US_DOT": 3418975,
            "MC_num": "1206193",
            "KYU": "639224"
        }
    },
    "license_front": {
        "file_uploaded": "<<utc timestamp>>",
        "file_location": "/path/to/file"
    },
    ...
}
```

#### OCR Job Configuration
Each OCR job is defined with a JSON schema specifying which fields to extract. Fields without a `maps_to` property are stored using their `field_name` as the key:

```json
{
    "document_type": "driver_with_truck_num",
    "fields": [
        { "field_name": "US DOT", "type": "String", "maps_to": "US_DOT" },
        { "field_name": "MC", "type": "String", "maps_to": "MC_num" },
        { "field_name": "KYU", "type": "String" }
    ]
}
```

### Automated Checks System

#### Check Architecture
- **Check**: Abstract base class with a `run(document)` function
- **Primary Checks**: Checks with no dependencies; run immediately upon submission in separate Go routines
- **Dependent Checks**: Checks that depend on other checks completing first; launched as Go routines upon parent check completion

#### Check Lifecycle
1. Check begins → Set field in document to `"processing"`
2. Check executes logic
3. Check completes → Update field to one of: `"pass"`, `"fail"`, or `"human_review"`
4. Dependent checks are launched if check passed
5. Human review queue updated if `"human_review"` or `"fail"` result

#### Check Result Format
Each check adds a field to the document with status:
```json
{
    "app_id": "<<uuid>>",
    "validate_mc_number": "pass",
    "validate_phone_number": "pass",
    "verify_license_valid": "human_review",
    "cross_reference_us_dot": "fail"
}
```

### Application State Visibility & Anti-Gaming Strategy

**Critical Rule**: Applicants must never know WHY their application was rejected unless an approver explicitly chooses to share that information. This prevents applicants from gaming the system through trial-and-error re-submissions.

#### What Applicants CAN See
- **Approved**: Full approval message
- **Returned**: Only the `correction_feedback` message from the approver
- **Processing**: General status ("Your application is being reviewed")
- **Draft**: Their own unsent application

#### What Applicants CANNOT See
- **Rejected**: Application status shows "rejected" but NO reason provided
- Check results and names (even passed checks)
- Approver internal notes (`approver_notes` field)
- Specific validation failures or OCR data
- Which checks failed or passed

#### What Approvers CAN See (Full Visibility)
- Complete application data
- All OCR results and extracted data
- All check results (passed, failed, human_review)
- Previous approver notes and decision history
- Full validation details for debugging

#### Workflow States from Applicant Perspective
- **draft** → (applicant submits) → **submitted** → **processing** → **approved/rejected/returned**
- "returned" shows feedback for corrections; applicant can resubmit
- "rejected" shows no details; applicant cannot resubmit unless approver explicitly allows
- **Unregistered token behavior**: each new unregistered session creates a fresh token. A token cannot access or enumerate previous sessions; clearing cookies removes access. This helps protect sensitive documents from snooping by others using the same browser.
---

## User Roles & Access Control

### Role Types
- **Unregistered Trucker** – No login required. Receives a transient `unreg_token` cookie to upload and view only the applications tied to that token. Cannot see previous tokens’ data. Token persists until browser clears it or is upgraded by registration.
- **Registered Trucker/Applicant** – Logs in normally. Can view full submission history, reuse documents, and update profile.
- **Approver/Manager** – Can review applications, approve/reject/return, manage users
- **Admin** – Full system access, user management, configuration

### Authentication
All endpoints except `/login` and `/health` require authentication via:
- Session token or JWT (implementation detail)
- Each authenticated endpoint validates user role

---

## Error Handling & Validation

### Field Validation
- `MC_num`: Regex pattern validation required
- `driver_license_num`: Numbers only
- `driver_cell_phone`: Valid phone number format
- `truck_num`, `trailer_num`: Numbers only
- All file uploads: Must be JPEG, PNG, or PDF (max size: [TBD])

### API Response Format (Recommended)
```json
{
    "success": true,
    "data": { ... },
    "error": null
}
```

Or on error:
```json
{
    "success": false,
    "data": null,
    "error": {
        "code": "VALIDATION_ERROR",
        "message": "Invalid MC number format"
    }
}
```

### HTTP Status Codes
- `200 OK` - Successful request
- `201 Created` - Resource created
- `400 Bad Request` - Invalid input or missing/invalid token for unregistered user
- `401 Unauthorized` - Missing/invalid authentication
- `403 Forbidden` - Insufficient permissions
- `404 Not Found` - Resource not found (including token mismatch)
- `500 Internal Server Error` - Server error

---

## Database Considerations

### Recommended MongoDB Indexes
- `_id` (automatic)
- `submitted_timestamp` (for sorting/filtering applications)
- `status` (for querying by application status)
- `driver_license_num` (for duplicate detection)
- `app_id` (if not using `_id` as primary key)

### Data Retention
- Applications should be archived after [TBD] days/months
- Document files may need separate storage (S3/filesystem)

---

## Security Considerations

- **Token-based User Creation**: `/createUser` requires a valid onboarding token to prevent unauthorized user creation
- **Role-based Access**: All protected endpoints must verify user role before granting access
- **Password Storage**: Passwords should be hashed (bcrypt/Argon2 recommended)
- **File Uploads**: Validate file types and scan for malware
- **OCR Accuracy**: Limit OCR confidence thresholds; flag low-confidence results for human review

---

- Email notifications when application status changes
- Application history/audit trail
- Batch processing for legacy applications
- Admin dashboard with analytics and reporting
- Integration with external validation services (FMCSA, state registries)
- Multi-language support
- Mobile application
- Webhook integrations for downstream systems