# Anonymization & Pseudonymization Pipelines

**Document Version**: 1.0 **Last Updated**: 2025-10-16 **Owners**: Privacy
Engineering & Data Governance Team

This document describes the anonymization and pseudonymization pipelines,
privacy budget tracking, and integration with data governance lifecycle
management.

## Overview

The anonymization service provides comprehensive privacy-preserving data
transformation using industry-standard algorithms:

- **K-Anonymity**: Generalizes quasi-identifiers to ensure at least k
  indistinguishable records
- **L-Diversity**: Ensures at least l distinct sensitive attribute values in
  each equivalence class
- **Differential Privacy**: Adds calibrated noise to protect individual privacy
  in aggregated queries
- **Pseudonymization**: Reversible/irreversible identifier transformation

## Architecture

```
┌──────────────────┐         ┌────────────────────────┐
│ Data Governance  │────────▶│  Anonymization Service │
│   Service        │         │                        │
│                  │         │  - K-Anonymity         │
│  Lifecycle:      │         │  - L-Diversity         │
│  ANONYMIZE action│         │  - Differential Privacy│
│                  │         │  - Pseudonymization    │
└──────────────────┘         │  - Privacy Budgets     │
                             └────────────────────────┘
                                         │
                                         ▼
                             ┌────────────────────────┐
                             │   Anonymized Datasets  │
                             │   Privacy Budget Ledger│
                             └────────────────────────┘
```

## Anonymization Methods

### 1. K-Anonymity

**Endpoint**: `POST /v1/anonymization/k-anonymity`

Ensures that each record is indistinguishable from at least k-1 other records
based on quasi-identifiers.

**Example Request**:

```json
{
  "dataset": [
    { "age": 25, "city": "New York", "salary": 75000 },
    { "age": 26, "city": "New York", "salary": 78000 },
    { "age": 27, "city": "New York", "salary": 72000 }
  ],
  "quasi_identifiers": ["age", "city"],
  "sensitive_attributes": ["salary"],
  "k": 3
}
```

**Response**:

```json
{
  "group_id": "k-anon-1",
  "anonymized_dataset": [
    { "age": "20-29", "city": "New***", "salary": 75000 },
    { "age": "20-29", "city": "New***", "salary": 78000 },
    { "age": "20-29", "city": "New***", "salary": 72000 }
  ],
  "k_value": 3,
  "anonymization_method": "k-anonymity",
  "groups_created": 1,
  "privacy_loss": 0.67,
  "applied_at": "2025-10-16T10:00:00Z"
}
```

### 2. L-Diversity

**Endpoint**: `POST /v1/anonymization/l-diversity`

Ensures each equivalence class has at least l well-represented sensitive
attribute values.

**Example Request**:

```json
{
  "dataset": [
    { "zip": "10001", "age": 25, "disease": "flu" },
    { "zip": "10001", "age": 26, "disease": "covid" },
    { "zip": "10001", "age": 27, "disease": "cold" }
  ],
  "quasi_identifiers": ["zip", "age"],
  "sensitive_attribute": "disease",
  "l": 3
}
```

### 3. Differential Privacy

**Endpoint**: `POST /v1/anonymization/differential-privacy`

Adds calibrated Laplace or Gaussian noise to protect individual privacy in
aggregated statistics.

**Example Request**:

```json
{
  "query": "SELECT COUNT(*) FROM users WHERE age > 25",
  "result": 1523,
  "sensitivity": 1,
  "epsilon": 1.0,
  "noise_type": "laplace"
}
```

**Response**:

```json
{
  "query_id": "dp-query-1",
  "query": "SELECT COUNT(*) FROM users WHERE age > 25",
  "original_result": 1523,
  "noisy_result": 1528,
  "epsilon": 1.0,
  "sensitivity": 1,
  "noise_added": 5,
  "privacy_guarantee": "(1.0, 0)-differential privacy",
  "noise_type": "laplace",
  "applied_at": "2025-10-16T10:00:00Z"
}
```

### 4. Pseudonymization

**Endpoint**: `POST /v1/anonymization/pseudonymize`

Creates reversible pseudonyms or irreversible anonymous identifiers.

**Reversible Pseudonymization**:

```json
{
  "identifiers": ["user-12345", "user-67890"],
  "context": "analytics",
  "reversible": true
}
```

**Response**:

```json
{
  "mappings": [
    {
      "original_id": "user-12345",
      "pseudonym": "pseudo_analytics_a3f8e9b..._1a2b3c",
      "context": "analytics",
      "reversible": true,
      "algorithm": "aes-256-cbc",
      "created_at": "2025-10-16T10:00:00Z"
    }
  ],
  "mapping_type": "pseudonymization"
}
```

**Irreversible Anonymization**:

```json
{
  "identifiers": ["user-12345"],
  "context": "research",
  "reversible": false
}
```

## Privacy Budget Management

Privacy budgets track the cumulative privacy loss from repeated queries,
essential for differential privacy guarantees.

### Creating a Privacy Budget

**Endpoint**: `POST /v1/privacy-budgets`

```json
{
  "user_id": "user-12345",
  "total_budget": 100,
  "period_days": 30,
  "auto_reset": true,
  "budget_type": "differential_privacy"
}
```

**Response**:

```json
{
  "id": "budget-1",
  "user_id": "user-12345",
  "total_budget": 100,
  "consumed_budget": 0,
  "remaining_budget": 100,
  "period_days": 30,
  "auto_reset": true,
  "period_start": "2025-10-16T00:00:00Z",
  "period_end": "2025-11-15T00:00:00Z",
  "consumption_log": []
}
```

### Consuming Budget

**Endpoint**: `POST /v1/privacy-budgets/:user_id/consume`

```json
{
  "operation": "differential_privacy_query",
  "cost": 5,
  "metadata": {
    "query_id": "dp-query-1",
    "epsilon": 1.0
  }
}
```

**Budget Exceeded (429)**:

```json
{
  "code": "anonymization.budget_exceeded",
  "message": "Insufficient privacy budget",
  "remaining_budget": 2,
  "required_budget": 5,
  "period_end": "2025-11-15T00:00:00Z"
}
```

### Privacy Budget for Memories

User memories and personalization vectors consume privacy budget when:

1. **Memory retrieval**: Queries accessing user memories cost budget
   proportional to epsilon
2. **Personalization inference**: RAG queries over user data consume budget
3. **Analytics aggregation**: Aggregated statistics about user behavior

**Budget Allocation**:

- **Standard tier**: 100 epsilon per 30 days
- **Premium tier**: 200 epsilon per 30 days
- **Research tier**: 50 epsilon per 30 days (stricter privacy)

**Budget Costs**:

- Simple memory retrieval: 1 epsilon
- Complex RAG query: 2-5 epsilon
- Aggregated analytics: 5-10 epsilon

## Transformation Pipelines

Pipelines orchestrate multiple anonymization steps in sequence.

### Creating a Pipeline

**Endpoint**: `POST /v1/transformation-pipelines`

```json
{
  "pipeline_name": "User Data Anonymization",
  "source_service": "conversation",
  "transformation_steps": [
    {
      "type": "k-anonymity",
      "parameters": {
        "k": 5,
        "quasi_identifiers": ["age", "location"],
        "sensitive_attributes": ["message_content"]
      }
    },
    {
      "type": "pseudonymization",
      "parameters": {
        "context": "analytics",
        "reversible": false
      }
    }
  ],
  "schedule": "0 2 * * *",
  "enabled": true,
  "target_anonymization_level": "high"
}
```

### Executing a Pipeline

**Endpoint**: `POST /v1/transformation-pipelines/:id/execute`

```json
{
  "dry_run": false
}
```

## Integration with Data Governance

The anonymization service integrates with data governance lifecycle management:

### Lifecycle Action: ANONYMIZE

When data-governance executes a lifecycle rule with the `ANONYMIZE` action:

1. **Data-governance service** identifies records requiring anonymization
2. Creates an **anonymization job** via `POST /v1/anonymization/jobs`
3. **Anonymization service** executes the job using configured method
   (k-anonymity, l-diversity, etc.)
4. **Anonymized data** is stored in anonymized_datasets
5. **Original data** lifecycle_stage updated to `ANONYMIZING` then `ANONYMIZED`
6. **Audit trail** created in data-governance service

**Example Lifecycle Rule**:

```json
{
  "name": "Anonymize old PII after 2 years",
  "trigger_conditions": {
    "age_days": 730,
    "access_count_threshold": 0
  },
  "actions": ["ANONYMIZE", "DELETE"],
  "data_classifications": ["PII_HIGH", "PII_MEDIUM"]
}
```

### Integration Flow

```
Data Governance Service                    Anonymization Service
        │                                           │
        │  1. Lifecycle rule triggers               │
        │     ANONYMIZE action                      │
        │                                           │
        │  2. POST /v1/anonymization/jobs          │
        ├──────────────────────────────────────────▶│
        │                                           │
        │                                           │  3. Create job
        │                                           │     (PENDING)
        │                                           │
        │  4. POST /v1/anonymization/jobs/:id/execute│
        ├──────────────────────────────────────────▶│
        │                                           │
        │                                           │  5. Execute K-anonymity/
        │                                           │     L-diversity/DP
        │                                           │
        │  6. Return anonymized result             │
        │◀────────────────────────────────────────┤
        │                                           │
        │  7. Update lifecycle_stage              │
        │     ANONYMIZED                          │
        │                                           │
        │  8. Create audit trail                  │
        │                                           │
```

## Anonymization Policies

Define organization-wide anonymization requirements.

**Endpoint**: `POST /v1/anonymization/policies`

```json
{
  "policy_name": "User Memory Anonymization Policy",
  "description": "Anonymize user memories after 1 year of inactivity",
  "data_types": ["personalization_vectors", "agent_memory", "chat_history"],
  "anonymization_requirements": {
    "method": "k-anonymity",
    "k": 5,
    "quasi_identifiers": ["user_demographic", "usage_pattern"]
  },
  "retention_after_anonymization": "3 years",
  "compliance_frameworks": ["GDPR", "CCPA"],
  "enabled": true
}
```

## Privacy Budget Best Practices

### 1. Budget Allocation Strategy

- **Conservative**: Start with low budgets (50-100 epsilon/month)
- **Monitor consumption**: Track usage patterns via
  `GET /v1/privacy-budgets/:user_id`
- **Alert on depletion**: Notify users at 80% consumption

### 2. Query Optimization

- **Batch queries**: Combine multiple queries to reduce epsilon cost
- **Caching**: Cache noisy results to avoid re-querying
- **Higher epsilon for less sensitive data**: Adjust epsilon based on data
  sensitivity

### 3. Budget Reset Policies

- **Auto-reset**: Enable for continuous access (suitable for analytics)
- **Manual reset**: Require approval for sensitive research applications

## API Reference

### Anonymization Jobs

| Endpoint                             | Method | Description              |
| ------------------------------------ | ------ | ------------------------ |
| `/v1/anonymization/jobs`             | POST   | Create anonymization job |
| `/v1/anonymization/jobs/:id`         | GET    | Get job status           |
| `/v1/anonymization/jobs/:id/execute` | POST   | Execute job              |

### Privacy Budgets

| Endpoint                               | Method | Description           |
| -------------------------------------- | ------ | --------------------- |
| `/v1/privacy-budgets`                  | POST   | Create privacy budget |
| `/v1/privacy-budgets/:user_id`         | GET    | Get current budget    |
| `/v1/privacy-budgets/:user_id/consume` | POST   | Consume budget        |

### Transformation Pipelines

| Endpoint                                   | Method | Description      |
| ------------------------------------------ | ------ | ---------------- |
| `/v1/transformation-pipelines`             | POST   | Create pipeline  |
| `/v1/transformation-pipelines/:id/execute` | POST   | Execute pipeline |

### Anonymization Policies

| Endpoint                     | Method | Description   |
| ---------------------------- | ------ | ------------- |
| `/v1/anonymization/policies` | POST   | Create policy |
| `/v1/anonymization/policies` | GET    | List policies |

## Testing

The anonymization service has comprehensive test coverage (87/87 tests passing,
86% coverage):

- **Unit tests**: Individual algorithm correctness
- **Integration tests**: End-to-end anonymization workflows
- **Budget tests**: Privacy budget consumption and enforcement
- **Pipeline tests**: Multi-step transformation validation

**Test Locations**:

- `/services/anonymization/__tests__/`
- Test coverage: `cd services/anonymization && npm test`

## Monitoring & Metrics

Key metrics to monitor:

1. **Budget consumption rate**: Track epsilon consumption per user
2. **Anonymization job success rate**: Monitor failed jobs
3. **Privacy loss metrics**: Track information loss from k-anonymity
4. **Pipeline execution time**: Optimize transformation performance

## Compliance

This implementation supports:

- **GDPR Article 25**: Privacy by Design and Default
- **CCPA Section 1798.100**: Right to anonymized data
- **NIST Privacy Framework**: Data processing principles
- **ISO/IEC 29100**: Privacy framework standards

## References

- Implementation: `/services/anonymization/src/app.ts`
- Tests: `/services/anonymization/__tests__/`
- Data Governance Integration: `/docs/security/DATA_GOVERNANCE.md`
- Privacy Budget Algorithms: Dwork & Roth, "The Algorithmic Foundations of
  Differential Privacy"
