411 lines
17 KiB
Plaintext
411 lines
17 KiB
Plaintext
# LogJensticks - Application Management System
|
||
|
||
## Project Overview
|
||
LogJensticks is a trucking application management system that automates the processing of job applications through document verification, OCR analysis, and automated checks.
|
||
|
||
## Architecture
|
||
|
||
### Infrastructure
|
||
- **3 Docker Containers**: Caddy (reverse proxy + static frontend) + Go server + MongoDB, orchestrated with Docker Compose
|
||
- **Build System**: Makefile for building, running, and testing
|
||
- `make build` - Compiles Go binary
|
||
- `make run` - Builds Docker images and starts all containers
|
||
- `make run-server` - Starts only the Go server
|
||
- `make run-db` - Starts only MongoDB
|
||
- `make run-caddy` - Starts only Caddy
|
||
- `make run-frontend` - Starts Vite dev server locally (no Docker)
|
||
- `make run-test` - Runs unit tests
|
||
|
||
---
|
||
|
||
## API Endpoints
|
||
|
||
### Authentication
|
||
| Method | Endpoint | Access | Description |
|
||
|--------|----------|--------|-------------|
|
||
| POST | `/login` | Public | User login with credentials |
|
||
| POST | `/logout` | Authenticated | User logout |
|
||
| GET | `/health` | Public | Health check endpoint |
|
||
|
||
> **Unregistered Flow**: Carriers may use the system without registering. Upon first upload/submit the server issues a random `unreg_token` stored in a **secure, HTTP‑only** cookie. "Secure" means the browser will only send it over HTTPS connections, preventing exposure on unencrypted networks. "HTTP‑only" means JavaScript running in the browser cannot read the cookie; this reduces the risk of token theft via XSS. All subsequent requests (save, submit, get, status) require that cookie; the token is a single-use key that only allows access to applications created with the same token. Tokens cannot access others' applications. Registration upgrades the session to a normal authenticated user and allows reuse of prior documents.
|
||
|
||
### Application Management (Carrier)
|
||
| Method | Endpoint | Access | Description |
|
||
|--------|----------|--------|-------------|
|
||
| POST | `/saveApplication` | Public or cookie | Save application as draft (auto-generates `app_id`). Sets/reads `anon_token` cookie when user is not registered. |
|
||
| POST | `/submitApplication` | Public or cookie | Submit application for review (sets `submitted_timestamp`); requires valid token if unregistered. |
|
||
| GET | `/getApplication/:applicationID` | Authenticated or cookie | Retrieve application details. Registered users may view their whole history; unregistered users may only view apps tied to their current `unreg_token`. |
|
||
| GET | `/applicationStatus/:applicationID` | Authenticated or cookie | Check status and results with same access rules as above. |
|
||
|
||
### Application Review (Broker)
|
||
| Method | Endpoint | Access | Brokers Only | Description |
|
||
|--------|----------|--------|---|-------------|
|
||
| GET | `/approvalQueue` | Yes | - | Get list of pending applications |
|
||
| GET | `/getApplication/:id` | Yes | - | Retrieve application for review (shows all details including check results) |
|
||
| POST | `/approve/:applicationID` | Yes | - | Approve application with optional internal comments |
|
||
| POST | `/reject/:applicationID` | Yes | - | Permanently reject application with internal reason (carrier will not see reason) |
|
||
| POST | `/return/:applicationID` | Yes | - | Return application to carrier for corrections with required feedback message |
|
||
|
||
### Lane Management
|
||
| Method | Endpoint | Access | Description |
|
||
|--------|----------|--------|-------------|
|
||
| POST | `/lanes` | Broker | Create a new lane; returns the lane ID and shareable link |
|
||
| GET | `/lanes` | Broker | List all lanes created by the authenticated broker |
|
||
| GET | `/lanes/:laneID` | Public (via link) | View lane details |
|
||
| POST | `/lanes/:laneID/bid` | Authenticated Carrier | Place a bid on a lane; adds carrier's ID to `bidding_carriers` |
|
||
|
||
### User Management (Admin/Manager)
|
||
| Method | Endpoint | Access | Description |
|
||
|--------|----------|--------|-------------|
|
||
| POST | `/createUser/:username/:token` | Token Required | Create new user (password in request body, token must match DB record) |
|
||
| GET | `/users` | Admin Only | List all users and their roles |
|
||
| PUT | `/updateUser/:username` | Admin Only | Update user details or role |
|
||
|
||
### Dashboard (Optional)
|
||
| Method | Endpoint | Access | Description |
|
||
|--------|----------|--------|-------------|
|
||
| GET | `/dashboard/stats` | Admin Only | System statistics and metrics |
|
||
|
||
---
|
||
|
||
## Frontend Pages
|
||
|
||
### Public Pages (HTML/CSS/Vue.js)
|
||
- `/login` - User authentication form
|
||
|
||
### Carrier Pages (Authenticated or Unregistered)
|
||
- `/apply` - Application submission form (multi-step form with file uploads). Works without login; unregistered users are issued a token cookie. Registered users can save and reuse documents.
|
||
- `/applicationConfirmation` - Post-submission confirmation with application ID and notice about token cookie for unregistered users.
|
||
- `/applicationStatus` - Track application progress and view check results. Unregistered users may only access apps created with current token.
|
||
- `/register` - Optional sign-up page for users who want an account to retain history and reuse files.
|
||
|
||
### Broker Pages (Authenticated + Authorized)
|
||
- `/approvalQueue` - List of pending applications for review
|
||
- `/reviewApplication/:applicationID` - Detailed application review interface with approve/reject actions
|
||
- `/dashboard` - Manager dashboard with application statistics
|
||
- `/lanes` - List of broker's lanes with bid counts
|
||
- `/lanes/create` - Form to create a new lane
|
||
- `/lanes/:laneID` - Lane detail view showing all bids
|
||
|
||
### Carrier Pages (Lane Bidding)
|
||
- `/lanes/:laneID` - Public lane view; authenticated carriers can place a bid
|
||
|
||
---
|
||
|
||
## Data Schema
|
||
|
||
### Application Document Fields
|
||
|
||
#### User-Provided Fields
|
||
- `load_num` (string) - Load number
|
||
- `carrier_name` (string) - Carrier/company name
|
||
- `MC_num` (string) - Motor Carrier number (validated with regex)
|
||
- `driver_name` (string) - Full name of driver
|
||
- `driver_license_num` (string) - License number (numbers only)
|
||
- `truck_plate_num` (string) - License plate number
|
||
- `driver_cell_phone` (string) - Phone number (must be valid format)
|
||
- `truck_num` (string) - Truck identifier
|
||
- `trailer_num` (string) - Trailer identifier
|
||
|
||
#### Document Uploads (JPEG, PNG, PDF)
|
||
- `driver_with_truck_num` - Picture showing driver, truck, and visible US DOT/MC/KYU numbers
|
||
- `license_front` - Front of driver's license
|
||
- `license_back` - Back of driver's license
|
||
- `apportioned_cab_card` - Apportioned cab card
|
||
|
||
#### System-Generated Fields (on submission)
|
||
- `_id` (UUID) - MongoDB document ID, same as `app_id`
|
||
- `app_id` (UUID) - Application ID
|
||
- `submitted_timestamp` (UTC timestamp) - When application was submitted
|
||
- `status` (string) - Current status: "draft", "submitted", "processing", "approved", "rejected", "returned", "human_review"
|
||
- `broker_notes` (string, optional) - Internal notes from broker (not visible to carrier)
|
||
- `correction_feedback` (string, optional) - Feedback message sent to carrier when application is returned
|
||
- `unreg_token` (string, optional) – Random token assigned to unregistered users; stored in cookie and used to authorize access to this application
|
||
|
||
#### Other Metadata
|
||
- `created_by_user` (string, optional) – Username or ID if submitted by a registered user (otherwise absent)
|
||
- `last_modified` (UTC timestamp) – When the record was last updated
|
||
|
||
#### Check Result Fields (added during processing)
|
||
- `[CheckName]` (string) - One field per check with value: "processing", "pass", "fail", or "human_review"
|
||
|
||
### Lane Document Fields
|
||
|
||
- `_id` (UUID) - MongoDB auto-generated document ID; used in URLs and shareable links
|
||
- `lane_id` (string, nullable) - Brokerage's own internal reference number for the lane; provided by the broker, not generated by the system. May be null until the broker assigns one, but must be filled in eventually.
|
||
- `pickup_state` (string, nullable) - Two-letter state code for pickup location (e.g. `"CA"`); may be derived from `pickup_address`
|
||
- `dropoff_state` (string, nullable) - Two-letter state code for dropoff location; may be derived from `dropoff_address`
|
||
- `pickup_address` (string, nullable) - Full pickup address
|
||
- `dropoff_address` (string, nullable) - Full dropoff address
|
||
- `bidding_carriers` (array of strings) - `_id`s of carriers who have placed a bid
|
||
- `created_by` (string) - Username of the broker who created the lane
|
||
- `created_by_id` (ObjectID) - MongoDB `_id` of the broker user document
|
||
|
||
> **Validation rule**: At least one of the following must be provided at creation time: `lane_id`, both state codes, or both addresses. A lane cannot be created with none of these.
|
||
|
||
#### Example Lane Document
|
||
```json
|
||
{
|
||
"_id": "<<mongo uuid>>",
|
||
"lane_id": "LN-2094",
|
||
"pickup_state": "CA",
|
||
"dropoff_state": "TX",
|
||
"pickup_address": "123 Warehouse Blvd, Los Angeles, CA 90001",
|
||
"dropoff_address": "456 Distribution Dr, Houston, TX 77001",
|
||
"bidding_carriers": ["<<carrier _id>>", "<<carrier _id>>"],
|
||
"created_by": "broker_jane",
|
||
"created_by_id": "<<ObjectID>>"
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Application Processing Pipeline
|
||
|
||
### Submission Flow
|
||
When an application is submitted, it goes through three stages:
|
||
|
||
1. **Storage** - Application stored in MongoDB with all user-provided fields and documents
|
||
2. **OCR Processing** - Optical Character Recognition extracts data from uploaded document images
|
||
3. **Automated Checks** - Validation checks run to verify application completeness and accuracy
|
||
|
||
### Document File Storage Structure
|
||
File uploads are stored with metadata:
|
||
```json
|
||
{
|
||
"file_uploaded": "<<utc timestamp>>",
|
||
"file_location": "/path/to/file"
|
||
}
|
||
```
|
||
|
||
For example, during initial submission:
|
||
```json
|
||
{
|
||
"_id": "<<uuid>>",
|
||
"app_id": "<<uuid>>",
|
||
"load_num": "5395746",
|
||
"carrier_name": "1 Fab Express",
|
||
"MC_num": "1206193",
|
||
"driver_name": "Fabiano Caruana",
|
||
"driver_license_num": 060414478,
|
||
"truck_plate_num": "C180XZ",
|
||
"driver_cell_phone": 3859827777,
|
||
"truck_num": 1234,
|
||
"trailer_num": 53302,
|
||
"submitted_timestamp": "<<utc timestamp>>",
|
||
"status": "processing",
|
||
"driver_with_truck_num": {
|
||
"file_uploaded": "<<utc timestamp>>",
|
||
"file_location": "/path/to/file"
|
||
},
|
||
"license_front": {
|
||
"file_uploaded": "<<utc timestamp>>",
|
||
"file_location": "/path/to/file"
|
||
},
|
||
"license_back": {
|
||
"file_uploaded": "<<utc timestamp>>",
|
||
"file_location": "/path/to/file"
|
||
},
|
||
"apportioned_cab_card": {
|
||
"file_uploaded": "<<utc timestamp>>",
|
||
"file_location": "/path/to/file"
|
||
}
|
||
}
|
||
```
|
||
|
||
### OCR Processing
|
||
|
||
OCR data extracted from images is stored within the document field's `ocr_results` object:
|
||
|
||
```json
|
||
{
|
||
"driver_with_truck_num": {
|
||
"file_uploaded": "<<utc timestamp>>",
|
||
"file_location": "/path/to/file",
|
||
"ocr_results": {
|
||
"US_DOT": 3418975,
|
||
"MC_num": "1206193",
|
||
"KYU": "639224"
|
||
}
|
||
},
|
||
"license_front": {
|
||
"file_uploaded": "<<utc timestamp>>",
|
||
"file_location": "/path/to/file"
|
||
},
|
||
...
|
||
}
|
||
```
|
||
|
||
#### OCR Job Configuration
|
||
Each OCR job is defined with a JSON schema specifying which fields to extract. Fields without a `maps_to` property are stored using their `field_name` as the key:
|
||
|
||
```json
|
||
{
|
||
"document_type": "driver_with_truck_num",
|
||
"fields": [
|
||
{ "field_name": "US DOT", "type": "String", "maps_to": "US_DOT" },
|
||
{ "field_name": "MC", "type": "String", "maps_to": "MC_num" },
|
||
{ "field_name": "KYU", "type": "String" }
|
||
]
|
||
}
|
||
```
|
||
|
||
### Automated Checks System
|
||
|
||
#### Check Architecture
|
||
- **Check**: Abstract base class with a `run(document)` function
|
||
- **Primary Checks**: Checks with no dependencies; run immediately upon submission in separate Go routines
|
||
- **Dependent Checks**: Checks that depend on other checks completing first; launched as Go routines upon parent check completion
|
||
|
||
#### Check Lifecycle
|
||
1. Check begins → Set field in document to `"processing"`
|
||
2. Check executes logic
|
||
3. Check completes → Update field to one of: `"pass"`, `"fail"`, or `"human_review"`
|
||
4. Dependent checks are launched if check passed
|
||
5. Human review queue updated if `"human_review"` or `"fail"` result
|
||
|
||
#### Check Result Format
|
||
Each check adds a field to the document with status:
|
||
```json
|
||
{
|
||
"app_id": "<<uuid>>",
|
||
"validate_mc_number": "pass",
|
||
"validate_phone_number": "pass",
|
||
"verify_license_valid": "human_review",
|
||
"cross_reference_us_dot": "fail"
|
||
}
|
||
```
|
||
|
||
### Application State Visibility & Anti-Gaming Strategy
|
||
|
||
**Critical Rule**: Carriers must never know WHY their application was rejected unless a broker explicitly chooses to share that information. This prevents carriers from gaming the system through trial-and-error re-submissions.
|
||
|
||
#### What Carriers CAN See
|
||
- **Approved**: Full approval message
|
||
- **Returned**: Only the `correction_feedback` message from the broker
|
||
- **Processing**: General status ("Your application is being reviewed")
|
||
- **Draft**: Their own unsent application
|
||
|
||
#### What Carriers CANNOT See
|
||
- **Rejected**: Application status shows "rejected" but NO reason provided
|
||
- Check results and names (even passed checks)
|
||
- Broker internal notes (`broker_notes` field)
|
||
- Specific validation failures or OCR data
|
||
- Which checks failed or passed
|
||
|
||
#### What Brokers CAN See (Full Visibility)
|
||
- Complete application data
|
||
- All OCR results and extracted data
|
||
- All check results (passed, failed, human_review)
|
||
- Previous broker notes and decision history
|
||
- Full validation details for debugging
|
||
|
||
#### Workflow States from Carrier Perspective
|
||
- **draft** → (carrier submits) → **submitted** → **processing** → **approved/rejected/returned**
|
||
- "returned" shows feedback for corrections; carrier can resubmit
|
||
- "rejected" shows no details; carrier cannot resubmit unless broker explicitly allows
|
||
- **Unregistered token behavior**: each new unregistered session creates a fresh token. A token cannot access or enumerate previous sessions; clearing cookies removes access. This helps protect sensitive documents from snooping by others using the same browser.
|
||
---
|
||
|
||
## User Roles & Access Control
|
||
|
||
### Role Types
|
||
- **Unregistered Carrier** – No login required. Receives a transient `unreg_token` cookie to upload and view only the applications tied to that token. Cannot see previous tokens’ data. Token persists until browser clears it or is upgraded by registration.
|
||
- **Registered Carrier** – Logs in normally. Can view full submission history, reuse documents, and update profile.
|
||
- **Broker** – Can review applications, approve/reject/return, manage users
|
||
- **Admin** – Full system access, user management, configuration
|
||
|
||
### Authentication
|
||
All endpoints except `/login` and `/health` require authentication via:
|
||
- Session token or JWT (implementation detail)
|
||
- Each authenticated endpoint validates user role
|
||
|
||
---
|
||
|
||
## Error Handling & Validation
|
||
|
||
### Field Validation
|
||
- `MC_num`: Regex pattern validation required
|
||
- `driver_license_num`: Numbers only
|
||
- `driver_cell_phone`: Valid phone number format
|
||
- `truck_num`, `trailer_num`: Numbers only
|
||
- All file uploads: Must be JPEG, PNG, or PDF (max size: [TBD])
|
||
|
||
### API Response Format (Recommended)
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": { ... },
|
||
"error": null
|
||
}
|
||
```
|
||
|
||
Or on error:
|
||
```json
|
||
{
|
||
"success": false,
|
||
"data": null,
|
||
"error": {
|
||
"code": "VALIDATION_ERROR",
|
||
"message": "Invalid MC number format"
|
||
}
|
||
}
|
||
```
|
||
|
||
### HTTP Status Codes
|
||
- `200 OK` - Successful request
|
||
- `201 Created` - Resource created
|
||
- `400 Bad Request` - Invalid input or missing/invalid token for unregistered user
|
||
- `401 Unauthorized` - Missing/invalid authentication
|
||
- `403 Forbidden` - Insufficient permissions
|
||
- `404 Not Found` - Resource not found (including token mismatch)
|
||
- `500 Internal Server Error` - Server error
|
||
|
||
---
|
||
|
||
## Database Considerations
|
||
|
||
### Recommended MongoDB Indexes
|
||
|
||
**applications collection**
|
||
- `_id` (automatic)
|
||
- `submitted_timestamp` (for sorting/filtering)
|
||
- `status` (for querying by status)
|
||
- `driver_license_num` (for duplicate detection)
|
||
|
||
**lanes collection**
|
||
- `_id` (automatic)
|
||
- `created_by` (for listing a broker's lanes)
|
||
|
||
**users collection**
|
||
- `username` (unique)
|
||
|
||
**sessions collection**
|
||
- `token` (unique)
|
||
- `expires_at` (TTL)
|
||
|
||
### Data Retention
|
||
- Applications should be archived after [TBD] days/months
|
||
- Document files may need separate storage (S3/filesystem)
|
||
|
||
---
|
||
|
||
## Security Considerations
|
||
|
||
- **Token-based User Creation**: `/createUser` requires a valid onboarding token to prevent unauthorized user creation
|
||
- **Role-based Access**: All protected endpoints must verify user role before granting access
|
||
- **Password Storage**: Passwords should be hashed (bcrypt/Argon2 recommended)
|
||
- **File Uploads**: Validate file types and scan for malware
|
||
- **OCR Accuracy**: Limit OCR confidence thresholds; flag low-confidence results for human review
|
||
|
||
---
|
||
|
||
- Email notifications when application status changes
|
||
- Application history/audit trail
|
||
- Batch processing for legacy applications
|
||
- Admin dashboard with analytics and reporting
|
||
- Integration with external validation services (FMCSA, state registries)
|
||
- Multi-language support
|
||
- Mobile application
|
||
- Webhook integrations for downstream systems
|