Building Scalable REST APIs with Pagination: From Concept to Production

[Lomanu4]

This guide provides a comprehensive overview of implementing pagination in REST APIs using Node.js, Express, and TypeScript. It covers various pagination strategies, their advantages and disadvantages, and how to implement them effectively.

Table of Contents

1. Why Pagination Matters in Modern APIs
2. Project Setup & Architecture
3. Core Components Implementation
   • Data Modeling with Zod
   • Realistic Mock Data Generation
   • Pagination Strategies Implementation
   • Offset Pagination
   • Cursor-Based Pagination
   • Keyset Pagination
4. Validation & Error Handling
   • Central Validation Middleware
   • Unified Pagination Response
   • Error Handling Middleware
5. Testing & Validation
6. Production-Ready Considerations
   • Performance Optimization
   • Security Practices
   • Monitoring
7. Choosing Your Strategy
8. Next Steps & Improvements
9. Conclusion
10. References

1. Why Pagination Matters in Modern APIs


Key Challenges in Data Handling:

• ? Performance degradation with large datasets
• ? Mobile users needing smaller payloads
• ? Bandwidth cost reduction
• ? Predictable data navigation

Pagination Strategy Selection Guide:
| Strategy | Best For | Advantages | Limitations |
|----------|----------|------------|-------------|
| Offset | Simple apps, small datasets | Easy implementation | Performance issues at scale |
| Cursor | Social feeds, infinite scroll | Stable performance | Complex client implementation |
| Keyset | Ordered data with unique IDs | No duplicates/skips | Requires sequential access |

2. Project Setup & Architecture

Tech Stack Rationale

• Express.js: Minimalist web framework
• Zod: Type-safe schema validation
• Faker.js: Realistic mock data
• TypeScript: Enhanced code quality

Initialize Project


npm init -y
npm install express zod @faker-js/faker
npm install --save-dev typescript ts-node-dev @types/express @types/node
Folder Structure


src/
├── data/ # Mock data generation
├── schemas/ # Validation blueprints
├── routes/ # API endpoints
├── utils/ # Reusable utilities
└── index.ts # Server entry
3. Core Components Implementation

3.1 Data Modeling with Zod


src/schemas/user.ts


import { z } from "zod";

export const userSchema = z.object({
id: z.number().int().positive(),
name: z.string().min(2).max(100),
email: z.string().email().max(320),
createdAt: z.date(),
});

export type User = z.infer<typeof userSchema>;
3.2 Realistic Mock Data Generation


src/data/users.ts


import { faker } from "@faker-js/faker";
import { User } from "../schemas/user";

export const generateUsers = (count: number): User[] => {
const baseDate = new Date();
return Array.from({ length: count }, (_, i) => ({
id: i + 1,
name: faker.person.fullName(),
email: faker.internet.email(),
createdAt: faker.date.past({ refDate: baseDate, years: 1 }),
}));
};

export const users = generateUsers(1000); // Generate 1k test users
4. Pagination Strategies Implementation

4.1 Offset Pagination


Implementation Flow:


Client Request → Validate Input → Slice Array → Return Results

Route Implementation:


const offsetSchema = z.object({
limit: z.string().regex(/^\d+$/).transform(Number).default("10"),
offset: z.string().regex(/^\d+$/).transform(Number).default("0"),
});

router.get("/users/offset", (req, res) => {
const { limit, offset } = validateRequest(offsetSchema, req);
const data = users.slice(offset, offset + limit);

res.json(paginateResponse(data, users.length, limit, offset));
});
4.2 Cursor-Based Pagination


Implementation Flow:


Client Request → Validate Cursor → Filter & Sort → Calculate NextCursor

Route Implementation:


const cursorSchema = z.object({
limit: z.string().regex(/^\d+$/).transform(Number).default("10"),
cursor: z.string().datetime().optional(),
});

router.get("/users/cursor", (req, res) => {
const { limit, cursor } = validateRequest(cursorSchema, req);
const filtered = cursor
? users.filter((u) => u.createdAt < new Date(cursor))
: users;

const data = filtered.slice(0, limit);
const nextCursor = data[data.length - 1]?.createdAt.toISOString();

res.json(paginateResponse(data, users.length, limit, 0, nextCursor));
});
4.3 Keyset Pagination


Implementation Flow:


Client Request → Validate LastID → Find Position → Return Next Set

Route Implementation:


const keysetSchema = z.object({
limit: z.string().regex(/^\d+$/).transform(Number).default("10"),
lastId: z.string().regex(/^\d+$/).transform(Number).optional(),
});

router.get("/users/keyset", (req, res) => {
const { limit, lastId } = validateRequest(keysetSchema, req);
const startIdx = lastId ? users.findIndex((u) => u.id === lastId) + 1 : 0;

const data = users.slice(startIdx, startIdx + limit);
const nextId = data[data.length - 1]?.id;

res.json(paginateResponse(data, users.length, limit, startIdx, null, nextId));
});
5. Validation & Error Handling

5.1 Central Validation Middleware


src/utils/validation.ts


import { Request } from "express";
import { z, ZodSchema } from "zod";

export function validateRequest<T>(schema: ZodSchema<T>, req: Request): T {
const result = schema.safeParse({
...req.query,
...req.params,
...req.body,
});

if (!result.success) {
throw new Error(
JSON.stringify({
code: 400,
errors: result.error.errors,
})
);
}

return result.data;
}
5.2 Unified Pagination Response


src/schemas/pagination.ts


import { z } from "zod";

export const createPaginationSchema = <T extends z.ZodTypeAny>(schema: T) =>
z
.object({
total: z.number().min(0),
limit: z.number().min(1).max(100),
offset: z.number().min(0).optional(),
data: z.array(schema),
nextCursor: z.string().nullable().optional(),
nextId: z.number().nullable().optional(),
})
.strict();
5.3 Error Handling Middleware


src/index.ts


app.use((err: Error, req: Request, res: Response) => {
try {
const errorData = JSON.parse(err.message);
res.status(errorData.code).json({
success: false,
error: errorData.errors,
});
} catch {
res.status(500).json({
success: false,
error: "Internal server error",
});
}
});
6. Testing & Validation

Start Development Server


npx ts-node-dev src/index.ts
Test Endpoints


Offset Pagination


curl "

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

"

Cursor-Based Pagination


curl "

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

"
# Subsequent request using last item's cursor
curl "

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

"

Keyset Pagination


curl "

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

"
# Subsequent request using last item's ID
curl "

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

"
7. Production-Ready Considerations

Performance Optimization

• Redis caching for pagination results
• Database-level pagination (WHERE/OFFSET in SQL)
• Indexed sorting columns

Security Practices

• Rate limiting (express-rate-limit)
• Maximum page size enforcement
• Cursor encryption

Monitoring

• Track pagination usage patterns
• Monitor response sizes
• Alert on abnormal page requests

8. Choosing Your Strategy


Decision Flowchart:

1. Need simple navigation? → Offset
2. Handling infinite scroll? → Cursor
3. Ordered data with unique IDs? → Keyset
4. All else equal? → Benchmark with real data

Performance Characteristics (10k records):
| Operation | Offset | Cursor | Keyset |
|-----------------|--------|--------|--------|
| Page 1 | ~15ms | ~8ms | ~5ms |
| Page 100 | ~120ms | ~10ms | ~7ms |
| Page 1000 | ~950ms | ~12ms | ~9ms |

9. Next Steps & Improvements

1. Add Filtering

const filtered = users.filter((u) => u.name.includes(searchTerm));

1. Implement HATEOAS Include navigation links in responses:

{
"links": {
"next": "/users?cursor=2023-07-20T12:34:56Z",
"prev": "/users?cursor=2023-07-19T08:12:34Z"
}
}

This comprehensive guide connects each component through:

? Data Flow: Schema → Data → Pagination → Validation → Response

? Error Handling: Validation → Middleware → Client Feedback

? Performance: Strategy Choice → Implementation → Optimization

Conclusion


Building scalable REST APIs with pagination is essential for modern applications. By understanding the different strategies and their implications, you can create efficient, user-friendly APIs that handle large datasets gracefully. This guide provides a solid foundation for implementing pagination in your projects, ensuring both performance and usability.

References

• 
  

  Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

  
  
• 
  

  Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

  
  
• 
  

  Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

  
  

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.