Faculytics Docs

Core Components

Technology stack, module architecture, login strategies, cron jobs, and analysis pipeline components.

This document describes the high-level components, technology stack, and module architecture of the api.faculytics project.

1. System Overview

api.faculytics serves as an intermediary layer between Moodle and local institutional data. Its primary responsibilities include:

  • Authentication: Authenticating users via Moodle tokens and issuing local JWTs.
  • Data Synchronization: Mirroring Moodle's institutional hierarchy (Campuses, Semesters, Departments, Programs) and course enrollments.
  • Entity Management: Maintaining a normalized local database for analytics and extended features.
  • Questionnaire Management: Managing weighted questionnaires for student and faculty feedback. See Questionnaire Management for detailed architecture.

2. Technology Stack

  • Backend Framework: NestJS (v11)
  • Database ORM: MikroORM with PostgreSQL
  • Authentication: Passport.js (JWT and Refresh Token strategies)
  • External API: Moodle Web Services (REST)
  • Task Scheduling: NestJS Schedule (Cron)
  • Caching: @nestjs/cache-manager with Redis (@keyv/redis)
  • Job Queue: BullMQ (@nestjs/bullmq) on Redis
  • Health Checks: @nestjs/terminus with custom indicators
  • PDF Generation: Puppeteer (headless Chrome) + Handlebars templates
  • Object Storage: Cloudflare R2 via @aws-sdk/client-s3 with presigned URL downloads
  • Validation: Zod (Environment variables), class-validator (DTOs)

3. Module Architecture

The application is structured into Infrastructure and Application layers, coordinated by the AppModule.

4. Login Strategy Pattern

Authentication uses a priority-based strategy pattern (src/modules/auth/strategies/). Each strategy implements the LoginStrategy interface:

  • CanHandle(localUser, body): Determines if this strategy applies to the login request.
  • Execute(em, localUser, body): Performs authentication and returns the user + optional Moodle token.
  • priority: Numeric ordering (lower = higher precedence).
StrategyPriorityWhen it handles
LocalLoginStrategy10User exists and has a local password
MoodleLoginStrategy100User has no local password or doesn't exist yet

Priority ranges: 0-99 core auth, 100-199 external providers, 200+ fallbacks. To add a new provider, implement LoginStrategy and register it under the LOGIN_STRATEGIES injection token.

5. Moodle Sync Pipeline

Institutional sync (categories, courses, enrollments) uses a BullMQ-based composite job instead of individual cron jobs. See Institutional Sync Workflow for the full flow.

ComponentPurpose
MoodleStartupServiceBlocking startup orchestrator — categories always, courses/enrollments gated by env
MoodleSyncProcessorBullMQ processor — runs categories → courses → enrollments; creates/updates SyncLog with per-phase metrics
MoodleSyncSchedulerDynamic SchedulerRegistry-based cron (env-aware, admin-configurable via SystemConfig)
MoodleSyncControllerSync trigger, status, paginated history, schedule get/update — all SUPER_ADMIN only

Phase dependency: if categories fail, courses and enrollments are skipped. If courses fail, enrollments are skipped.

Cron Jobs

Cron jobs using the BaseJob pattern:

JobScheduleModulePurpose
RefreshTokenCleanupJobEvery 12 hoursAllCronJobsPurges refresh tokens older than 7 days
ReportCleanupJobDaily at 3 AMReportsModulePurges expired report jobs + R2 objects, orphaned waiting jobs

6. Moodle Connectivity & Error Handling

The MoodleClient enforces a 10-second timeout (MOODLE_REQUEST_TIMEOUT_MS) on all Moodle API calls via AbortSignal.timeout(). Network failures are wrapped in MoodleConnectivityError:

  • Timeout: "Moodle request timed out during {operation}"
  • Connection failure: "Failed to connect to Moodle service during {operation}"
  • General network error: "Network error during Moodle {operation}"

The MoodleLoginStrategy catches MoodleConnectivityError and translates it to a 401 Unauthorized with a user-friendly message.

7. Analysis Pipeline

The AnalysisModule provides a multi-stage analysis pipeline that orchestrates AI processing of qualitative feedback. See AI Inference Pipeline for the full architecture and Analysis Pipeline Workflow for the stage-by-stage flow.

Pipeline Orchestrator

The PipelineOrchestratorService manages the full analysis lifecycle through a confirm-before-execute pattern:

  1. Create — Computes coverage stats (response rate, submission/comment counts) and generates warnings
  2. Confirm — Validates configuration and dispatches the first stage
  3. Stage progression — Each processor calls back into the orchestrator to advance to the next stage
  4. Terminal statesCOMPLETED, FAILED, or CANCELLED

Components

ComponentPurpose
PipelineOrchestratorServiceCreates pipelines, manages stage transitions, dispatches batch jobs
AnalysisServiceLow-level entry point — EnqueueJob() and EnqueueBatch() for ad-hoc jobs
AnalysisControllerREST API for pipeline CRUD (POST/GET /analysis/pipelines)
BaseBatchProcessorAbstract base — HTTP dispatch, Zod validation, retry, stall detection
RunPodBatchProcessorRunPod-specific subclass — auth headers, { input/output } envelope handling
SentimentProcessorBatch sentiment analysis, triggers sentiment gate on completion
TopicModelProcessorBatch topic modeling via RunPod, chunked assignment persistence
TopicLabelServiceLLM-based labeling of BERTopic topics (gpt-4o-mini, inline before recommendations)
RecommendationGenerationServiceBuilds LLM prompts from DB data, calls OpenAI, computes confidence and evidence
RecommendationsProcessorBullMQ processor — delegates to RecommendationGenerationService, persists results
EmbeddingProcessorPer-submission embedding generation (upsert, extends BaseAnalysisProcessor)

Pipeline Stages

AWAITING_CONFIRMATION → SENTIMENT_ANALYSIS → SENTIMENT_GATE → TOPIC_MODELING → TOPIC_LABELING → GENERATING_RECOMMENDATIONS → COMPLETED

Each stage has a corresponding RunStatus (PENDINGPROCESSINGCOMPLETED / FAILED).

Queue Architecture

Eight BullMQ queues with independent concurrency. Queue names are centralized in src/configurations/common/queue-names.ts.

QueueProcessorConcurrency DefaultModule
moodle-syncMoodleSyncProcessor1MoodleModule
sentimentSentimentProcessor3AnalysisModule
embeddingEmbeddingProcessor3AnalysisModule
topic-modelTopicModelProcessor1AnalysisModule
recommendationsRecommendationsProcessor1AnalysisModule
analytics-refreshAnalyticsRefreshProcessor1AnalyticsModule
auditAuditProcessor1AuditModule
report-generationReportGenerationProcessor2ReportsModule

REST Endpoints

MethodPathDescription
POST/analysis/pipelinesCreate a pipeline (returns coverage stats + warnings)
POST/analysis/pipelines/:id/confirmConfirm and start execution
POST/analysis/pipelines/:id/cancelCancel a non-terminal pipeline
GET/analysis/pipelines/:id/statusGet pipeline status with stage details
GET/analysis/pipelines/:id/recommendationsGet recommendations for a completed pipeline

Analytics Endpoints

See Analytics Module for full architecture.

MethodPathDescription
GET/analytics/overviewDepartment overview with per-faculty stats
GET/analytics/attentionFaculty flagged for review with attention flags
GET/analytics/trendsFaculty trend data with linear regression results

Report Generation Endpoints

See Report Generation Workflow for the full async flow.

MethodPathDescription
POST/reports/generateQueue a single faculty evaluation PDF
POST/reports/generate/batchQueue batch PDF generation (scope-filtered, throttled)
GET/reports/status/:jobIdPoll single report job status + download URL
GET/reports/batch/:batchIdPoll batch progress with paginated per-job details

Resilience: Exponential backoff retries, stall detection, graceful degradation when Redis is unavailable (ServiceUnavailableException), HTTP timeout via AbortController.

Local development: docker compose up starts Redis and a mock worker (Hono HTTP server on port 3001) that simulates worker responses.

8. Health Checks

The HealthModule uses @nestjs/terminus to provide structured health checks at GET /health:

IndicatorChecks
databaseSELECT 1 via MikroORM EntityManager
redisRead/write test via cache manager

Returns HTTP 200 with status: 'ok' when healthy, HTTP 503 with status: 'error' and per-indicator details when unhealthy.

9. Scoped Query Pattern (Faculty & Curriculum)

The FacultyModule and CurriculumModule use a shared role-based scoping pattern for administrative queries. This ensures deans only see data within their assigned departments while super admins see everything.

Scope Resolution Chain

Request → JwtAuthGuard → RolesGuard → CurrentUserInterceptor → CLS Store → ScopeResolverService
  1. @UseJwtGuard(SUPER_ADMIN, DEAN, CHAIRPERSON) validates JWT and checks role membership via RolesGuard
  2. CurrentUserInterceptor loads the full User entity via UserLoader and stores it in CLS (CurrentUserService.set())
  3. ScopeResolverService.ResolveDepartmentIds(semesterId) reads the user from CLS and returns:
    • null — unrestricted (super admin)
    • string[] — department UUIDs the dean is assigned to for that semester
    • string[] — department UUIDs derived from the chairperson's program assignments
  4. ScopeResolverService.ResolveProgramIds(semesterId) provides program-level granularity:
    • null — unrestricted (super admin, dean)
    • string[] — program UUIDs the chairperson is assigned to

Filter Validation Cascade

When explicit filter params (departmentId, programId) are provided, they are validated against the resolved scope:

ScenarioResult
departmentId outside scope403 Forbidden
programId not found404 Not Found
programId department outside scope403 Forbidden
departmentId + programId mismatch400 Bad Request

Modules

ModuleEndpointsPurpose
FacultyModuleGET /facultyPaginated faculty list with course assignments
FacultyModuleGET /faculty/:facultyId/submission-countSubmission count for a faculty member per semester
CurriculumModuleGET /curriculum/departments, /programs, /coursesInstitutional hierarchy for filter dropdowns

Both modules import CommonModule (for ScopeResolverService) and DataLoaderModule (for CurrentUserInterceptorUserLoader).

The CurriculumModule endpoints return flat arrays (no pagination) since result sets are small within a dean's scope. All three endpoints require semesterId; the courses endpoint additionally requires at least one of programId or departmentId to prevent unbounded queries.

10. Startup & Initialization Flow

The application enforces a strict initialization sequence in InitializeDatabase before it begins accepting traffic. This ensures that the database schema and required infrastructure state are always synchronized with the code.

  1. Migration (orm.migrator.up()): Automatically applies any pending database migrations.
  2. Infrastructure Seeding (orm.seeder.seed(DatabaseSeeder)): Executes idempotent seeders (e.g., DimensionSeeder) to populate required reference data.
  3. Application Bootstrap: Only after both steps succeed does app.listen() execute. If any step fails, the process exits with code 1.