How to Handle Large Data Sets Without Killing Performance

Accepted answer · Answered by Dipa Saha · February 3, 2026

What is the N+1 Query Issue?

It happens when:

• You run 1 query to get parent records (N),
• Then N separate queries to get each child (hence N+1).

🧱 Example:

// Fetching users and their posts (bad)
const users = await db.users.findMany();
for (const user of users) {
  user.posts = await db.posts.findMany({ where: { userId: user.id } }); // N queries
}

❌ Terrible for performance when user count is large.

Use Eager Loading or include

Fetch related data in a single query.

Prisma:

const users = await db.user.findMany({
  include: {
    posts: true,
  },
});

Sequelize:

User.findAll({
  include: [Post],
});

✅ Result: 1 query instead of N+1.

Answered by Mynul Islam · February 3, 2026

Batch Data Processing with DataLoader

Use batching libraries like Facebook’s DataLoader to combine multiple related requests.

📦 npm install dataloader

Example:

const postLoader = new DataLoader(async (userIds) => {
  const posts = await db.posts.findMany({ where: { userId: { in: userIds } } });
  return userIds.map(id => posts.filter(p => p.userId === id));
});

Used by GraphQL resolvers to solve N+1 elegantly.

Answered by Shaykot Hossain Selim · February 3, 2026

Paginate Results (Offset or Cursor)

Never fetch large datasets at once.

✅ Offset Pagination:

GET /posts?limit=20&offset=40

✅ Cursor-Based Pagination:

GET /posts?after=post_123&limit=20

Answered by Sahin Khan · February 3, 2026

Use Server-Side Filtering and Sorting

Never filter or sort large data sets on the frontend.

❌ Bad:

const allPosts = await db.posts.findMany();
const filtered = allPosts.filter(p => p.status === 'published');

✅ Good:

const filtered = await db.posts.findMany({
  where: { status: 'published' },
});

Answered by Fozle Rabbi · February 3, 2026

Stream Large Data (Node.js)

Use streams to process large data without loading all into memory.

✅ Example:

const stream = dbClient.query('SELECT * FROM big_table');
stream.on('data', row => processRow(row));

Streams reduce memory usage and improve throughput.

Answered by Md Sharifuzzaman Shajib · February 3, 2026

Use IN or JOIN Instead of Multiple Queries

Use IN to batch fetch related data.

✅ Example (SQL):

SELECT * FROM posts WHERE userId IN (1,2,3,4);

Now you can map posts to users in-memory using a dictionary/map.

Answered by Ratul Sikder · February 3, 2026

Denormalization and Materialized Views

If you're frequently joining large tables, consider precomputing relationships.

✅ Example:

user_id: 1

total_posts: 53

latest_post_date: 2024-12-01

Create this with a cron job or a database view.

This trades storage for speed — great for dashboards, reports, analytics.

Answered by Jhon ericson · February 3, 2026