I'm making static portfolio website in Next.js. List of projects has 2 types of elements: fullscreen image and small squared image (fits 3 in a row). The picture below explain an list design.

For now all items fetches at first time and I'm using grid-auto-flow: dense; for filling empty spaces (this works fine, but fetching everything at once kills performance). So I want to implement infinite scrolling (or pagination, if I may say so) to solve all issues.

I've made several attempts. The first is simple infinite scroll, but with grid-auto-flow: dense; this approach skips some projects (when we scroll down projects are moving to top in empty spaces) and we can see them when scrolling up. Secondly I've tried to make "smart buffer" which could hoard small items and return them and returns them when the number is a multiple of three. But my solution lost items...

May be anybody knows the right way to make a working and concise solution for static app?